COMMISSION STAFF WORKING DOCUMENT EVALUATION Final Evaluation of the Third Health Programme 2014-2020
Date: 2023-11-15
Institutions: Directorate-General for Health and Food Safety, European Commission
Subjects: EU programme, disease prevention, health care, health care system, health service, illness, public health

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 15.11.2023  SWD(2023) 369 final  COMMISSION STAFF WORKING DOCUMENT  EVALUATION  Final Evaluation of the Third Health Programme 2014-2020 {SWD(2023) 370 final}  
 
 
Table of contents 

1. 

INTRODUCTION ................................................................................................................................ 1 

Purpose and scope of the evaluation .......................................................................................... 1 

1.1. Purpose of the evaluation ................................................................................................ 1 

1.2. Scope of the evaluation .................................................................................................... 1 

2.  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION ? ...................................... 4 

2.1 

2.2 

Description of the intervention and its objectives .......................................................... 4 

 Point(s) of comparison .................................................................................................. 10 

3.  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? ....................... 14 

Specific objective 1: promote health, prevent diseases, and foster supportive 

3.1. 
environments for healthy lifestyles ........................................................................................... 15 

3.1.1. Indicators .................................................................................................................... 15 

3.1.2. Examples of actions implemented under Specific objective 1: promote health, 
prevent diseases, and foster supportive environments for healthy lifestyles: .................. 16 

3.2. 

Specific objective 2: protect EU citizens from serious cross-border health threats ....... 17 

3.2.1. Indicator ...................................................................................................................... 17 

3.2.2. Examples of actions implemented under specific objective 2: protect EU citizens 
from serious cross-border health threats: ........................................................................... 17 

Specific objective 3: support public health capacity building and contribute to 

3.3. 
innovative, efficient and sustainable health systems: ............................................................... 18 

3.3.1. Indicators .................................................................................................................... 18 

3.3.2. Examples of actions under specific objective 3: support public health capacity 
building and contribute to innovative, efficient, and sustainable health systems ............ 19 

3.4. 

Specific objective 4: facilitate access to better and safer healthcare for EU citizens .... 20 

3.4.1. Indicators .................................................................................................................... 20 

3.4.2. Actions under specific objective 4: facilitate access to better and safer healthcare 
for EU citizens........................................................................................................................ 20 

Examples of actions addressing horizontal/cross-cutting issues – health inequalities, 

3.5. 
legislation, the sustainable development goals - SDGs, dissemination of programme results . 21 

3.6. 

Current state of play: ..................................................................................................... 22 

4. 

EVALUATION FINDINGS (ANALYTICAL PART) ....................................................................... 23 

4.1.  TO WHAT EXTENT WAS THE INTERVENTION SUCCESSFUL AND WHY? .......................... 23 

4.1.1. Effectiveness ................................................................................................................ 23 

4.1.2 Efficiency ...................................................................................................................... 36 

 
 
 
 
4.1.3 Coherence ..................................................................................................................... 51 

4.2.  HOW DID THE EU INTERVENTION MAKE A DIFFERENCE AND TO WHOM? .................... 54 

4.3.  IS THE INTERVENTION STILL RELEVANT? .............................................................................. 58 

5.  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? ................................................. 62 

5.1.  CONCLUSIONS ................................................................................................................................. 62 

Effectiveness ............................................................................................................................. 62 

Efficiency .................................................................................................................................. 64 

Coherence .................................................................................................................................. 66 

EU-added value ......................................................................................................................... 66 

Relevance .................................................................................................................................. 67 

5.2.  LESSONS LEARNED ........................................................................................................................ 68 

Conclusions and recommendations ........................................................................................... 68 

Further building on the midterm evaluation recommendations............................................. 68 

Design of Programme and funding frameworks ................................................................... 68 

Facilitating and strengthening participation of all countries ................................................. 69 

Ensuring sustainability .......................................................................................................... 69 

ANNEX I:   PROCEDURAL INFORMATION ........................................................................................... 70 

ANNEX II. METHODOLOGY AND ANALYTICAL MODELS USED ................................................... 72 

1. 

2. 

3. 

Building an understanding of the Programme .............................................................. 72 

Consulting with stakeholders......................................................................................... 73 

Analysis and synthesis ................................................................................................... 74 

ANNEX III. EVALUATION MATRIX ....................................................................................................... 77 

ANNEX IV. OVERVIEW OF BENEFITS AND COSTS ........................................................................... 99 

ANNEX V. STAKEHOLDERS CONSULTATION - SYNOPSIS REPORT ........................................... 110 

1 

2 

INTRODUCTION ............................................................................................................................ 110 

APPROACH TO THE CONSULTATION ....................................................................................... 110 

2.1 

2.2 

Stakeholder selection .................................................................................................. 110 

Consultation activities.................................................................................................. 111 

2.2.1 Advertising the consultations ..................................................................................... 111 

2.2.2 Targeted survey .......................................................................................................... 113 

Limitations .......................................................................................................................... 117 

3 

OVERVIEW OF CONTRIBUTIONS .............................................................................................. 118 

Targeted Survey .................................................................................................................. 118 

Public consultation .............................................................................................................. 119 

Interviews ............................................................................................................................ 119 

Focus Groups ....................................................................................................................... 120 

4 

ANALYSIS OF THE REPLIES ....................................................................................................... 121 

 
 
Relevance 121 

Effectiveness ....................................................................................................................... 125 

Efficiency 129 

EU added value ................................................................................................................... 134 

Coherence ............................................................................................................................ 134 

5 

6 

FEEDBACK ON THE CONSULTATION PROCESS .................................................................... 136 

USE OF THE INFORMATION GATHERED ................................................................................. 137 

ANNEX VI. THEMATIC PRIORITIES OF THE 3RD HEALTH PROGRAMME ................................... 138 

ANNEX VII. KEY MONITORING INDICATORS OF THE 3RD HEALTH PROGRAMME ................. 139 

ANNEX  VIII.  EXAMPLES  OF  FUNDED  ACTIONS  IMPLEMENTED  UNDER  THE  THIRD 

HEALTH PROGRAMME 2014-2020 ................................................................................................ 144 

 
 
 
 
 
Glossary 

Term or acronym 

Meaning or definition 

1HP 

2HP 

Programme 

AHEAD 

AIR 

AMR 

AWP 

Best-ReMaP 

CARE 

CBA 

CEA 

CHAFEA 

CHRODIS+ 

COMPASS 

CPMS 

CVD 

DG 

First health programme 

Second health programme 

Third health programme  

Action for Health Equity Addressing Medical Deserts 

Annual implementation report 

Antimicrobial resistance  

Annual work plan 

Joint Action on the implementation of validated best practices in 
nutrition 

Common approach for refugees and other migrants' health 
(project) 

Cost-benefit analysis 

Cost-effectiveness analysis 

Consumers, Health, Agriculture and Food Executive Agency 

Joint Action on chronic diseases 

European Commission’s IT-based workflow system for the 
Horizon 2020 Framework Programme for Research and 
Innovation and for other EU Programmes of the research family, 
including the Third Health Programme 2014-2020 

Clinical patient management system for European Reference 
Networks 

Cardio vascular disease 

Directorate-General of the European Commission 

DG SANTE 

Directorate-General for Health and Food Safety  

DGA 

EC 

ECDC 

Direct grant agreement 

European Commission 

European Centre for Disease Prevention and Control 

EHaction 

Joint Action supporting the eHealth network 

 
 
 
 
EHDS 

ERDF 

ERN PaedCan 

ERNs 

ERWS 

ESF 

ESIF 

EU-JAV 

EUDAMED 

EURIPID 

EUnetHTA 

FPA 

GAPP 

GDP 

GNI 

HaDEA 

HCAIs 

HTA 

IHR 

European Health Data Space 

European Regional Development Fund 

European Reference Network on Paediatric Cancer paediatric 
cancer 1 

European Reference Networks for rare and complex diseases 

Early warning and response system 

European Social Fund 

European Structural and Investment Funds  

Joint Action on vaccination 

European Database on Medical Devices 

European medicine price database 

Joint Action European network for health technology assessment 

Framework Partnership Agreement 

Joint Action Good practice guidelines on the authorisation and 
preparation process for blood, tissues and cells 

Gross domestic product   

Gross national income 

The European Health and Digital Executive Agency 

Healthcare-associated infections 

Health technology assessment 

International Health Regulations 2005 

ImmuHubs 

Innovative immunisation hubs (project) 

IOM 

iPAAC 

JAHEE 

JAMRAI 

JANPA 

International Organisation for Migration  

Innovative partnership for action against cancer 

Joint Action Health Equity Europe 

Joint Action on antimicrobial resistance and healthcare-associated 
infections 

Joint Action on nutrition and physical activity 

1 The list of the other 23 established ERNs is provided through the following link: Networks (europa.eu)  

 
 
 
 
 
MFF 

Multiannual financial framework 

Mig-HealthCare 

Strengthen community-based care to minimise health inequalities 
and improve the integration of vulnerable migrants and refugees 
into local communities (project) 

MS 

MyHealth 

NCA 

NFP 

NGO 

PC 

ORAMMA 

RAHRA 

SCIROCCO 

SDGs 

SGA 

SH-CAPAC 

SGPP 

SYGMA 

TFEU 

VISTART 

WHO 

WP 

Member State 

Models to engage vulnerable migrants and refugees in their 
health, through community empowerment and learning alliance 
(project) 

National competent authority 

National Focal Point 

Non-governmental organisation  

Public consultation  

Operational refugee and migrant maternal approach (project) 

Joint Action on reducing alcohol-related harm 

Scaling integrated care in context (project) 

Sustainable development goals 

Specific grant agreement 

Supporting health coordination, assessments, planning, access to 
healthcare and capacity building in Member States under 
particular migratory pressure (project) 

Steering Group on Health promotion and disease prevention and 
management of non-communicable diseases 

European Commission’ IT-based grant management system for 
the Horizon 2020 Framework Programme for Research and 
Innovation and for other EU Programmes of the research family, 
including the Third Health Programme 2014-2020 

Treaty on the Functioning of the European Union  

Joint Action on vigilance and inspection for the safety of 
transfusion, assisted reproduction and transplantation 

World Health Organisation 

Work package 

 
 
1. 

INTRODUCTION 

Purpose and scope of the evaluation 

1.1. Purpose of the evaluation  
This staff working document sets out the final evaluation of the third programme for EU 
action in the field of health (2014-2020) (the Programme)2. It is supported by an external 
study  carried  out  by  a  contractor  of  the  Commission  between  July  2021  and  October 
2022.  Its  purpose  is  to  assess  the  management  and  implementation  of  the  Programme, 
including the follow-up to recommendations of past health programmes’ evaluations. The 
evaluation complies with the financial rules3 applicable to the general budget of the EU, 
which call on the Commission to monitor, evaluate and to report on the implementation 
of  actions  of  spending  programmes.  It  contributes  to  a  better  understanding  of  the 
strengths and weaknesses of Programme implementation and management and provides 
conclusions that can be used as a basis for improving subsequent EU health programmes. 

1.2. Scope of the evaluation  
The  evaluation  assesses  the  performance  of  the  Programme,  its  main  outcomes  and 
achieved  results  and  identifies  the  main  problems  and  solutions  with  regard  to  its 
implementation, including recommendations from previous evaluations.  

The  evaluation  covers  the  Programme's  entire  implementation  period  (2014-2020).  It 
covers all EU Member States4, EFTA countries and non-EU countries participating in the 
Programme (Moldova, Serbia and Bosnia and Herzegovina). It focuses on the five main 
evaluation criteria, namely: relevance, effectiveness, efficiency, coherence and EU-added 
value. 

While assessing the functioning of the entire Programme,  the external study supporting 
this  evaluation  concentrated  on  issues  that  were  insufficiently  explored  in  past 
evaluations,  and  provided  conclusions  that  can  be  used  as  a  basis  to  improve  the 
implementation of future EU health programmes. 

Following the outbreak, in the first quarter 2020, of the COVID-19 pandemic in the EU 
territory, relevant actions funded by the Programme (2014-2020) were switched to their 
emergency mode and geared towards combatting the pandemic.  

2 The Programme was established by  (EU) No 282/2014 of the European Parliament and of the Council of 11 March 
2014 on the establishment of a third Programme for the Union's action in the field of health (2014-2020) and repealing 
Decision No 1350/2007/EC, OJ L 86, 21.3.2014, p. 1–13 
3 Articles 34(1) and 34(3) of the Regulation (EU, Euratom) 2018/1046 of the European Parliament and of the Council 
of 18 July 2018 on the financial rules applicable to the general budget of the Union (Financial Regulation), OJ L 193, 
30.7.2018, p. 1–222 
4 27 Member States plus the UK before Brexit.  
For ease of reference the Commission will refer in the rest of the document  to Member States. However, 
where relevant, this shall be deemed to also include EFTA countries and other countries which participated 
in the Programme (i.e. Bosnia and Herzegovina Moldova, Serbia). 

1 

 
 
 
These  actions  were  not  included  in  the  scope  of  the  external  study  supporting  the 
evaluation, since some of them were in early stage or in the middle of implementation at 
the time when the external  study was launched. Nevertheless,  the SWD mentions these 
actions, which were funded by the Programme in its last implementation year, as part of 
the  EU  response  to  the  COVID-19.  These  emergency  actions  were  mostly  launched  in 
the  early  stages  of  the  pandemic,  before  the  adoption  of  the  EU4Health  Programme 
(2021-2027)5, which succeeded the Third Health Programme 2014-2020. 

The  comprehensive  EU  response  to  COVID-19  was  evaluated  through  a  continuous 
process  of assessment of actions and measures  and lessons learnt, which have been  the 
subject  of  several  Commission  communications,  including  the  Communication  on 
drawing the early lessons from the COVID-19 pandemic6, the Communication on short-
term  EU  health  preparedness  for  COVID-19  outbreaks7,  the  Communication  on  EU 
Strategy  for  COVID-19  vaccines8,  the  Communication  on  Preparedness  for  COVID-19 
vaccination  strategies  and vaccine deployment9, the Communication on EU Strategy on 
COVID-19 therapeutics10, and the Communication on EU Global Health Strategy11.  
Lessons  learnt  from  the  pandemic  led,  notably,  to  the  Commission  proposals  for 
establishing a Health Emergency and Response Authority12 (HERA) and for building the 
European Health Union13, a set of key actions and legal instruments which help14: 

•  better protecting the health of citizens, 

5 Regulation (EU) 2021/522 of the European Parliament and of the Council of 24 March 2021 establishing 
a Programme for the Union’s action in the field of health (‘EU4Health Programme’) for the period 2021-
2027, and repealing Regulation (EU) No 282/2014 (OJ L 107, 26.3.2021, p. 1–29) 
EUR-Lex - 32021R0522 - EN - EUR-Lex (europa.eu). 
6  European  Commission,  (2021),  Communication  on  drawing  the  early  lessons  from  the  COVID-19 
pandemic. 
7  European  Commission,  (2020),  Communication  on  short-term  EU  health  preparedness  for  COVID-19 
outbreaks. 
8  COMMUNICATION  FROM  THE  COMMISSION  TO  THE  EUROPEAN  PARLIAMENT,  THE 
EUROPEAN COUNCIL, THE COUNCIL AND THE EUROPEAN INVESTMENT BANK EU Strategy 
for 
https://eur-lex.europa.eu/legal-
content/EN/TXT/PDF/?uri=CELEX:52020DC0245 

COVID-19 

vaccines, 

(2020) 

COM 

final, 

245 

9  COMMUNICATION  FROM  THE  COMMISSION  TO  THE  EUROPEAN  PARLIAMENT  AND  THE 
COUNCIL Preparedness for COVID-19 vaccination strategies and vaccine deployment, COM (2020) 680 
final https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52020DC0680  
10  COMMUNICATION  FROM  THE  COMMISSION  TO  THE  EUROPEAN  PARLIAMENT,  THE 
COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF 
THE REGIONS EU STRATEGY ON COVID-19 THERAPEUTICS COM/2021/355/final 
11  COMMUNICATION  FROM  THE  COMMISSION  TO  THE  EUROPEAN  PARLIAMENT,  THE 
COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF 
THE REGIONS on EU Global Health Strategy COM/2022/675 final. 
12 Proposal for a Regulation of the European Parliament and of the Council establishing a European Health 
Emergency Response Authority (HERA) 
13  COMMUNICATION  FROM  THE  COMMISSION  TO  THE  EUROPEAN  PARLIAMENT,  THE 
COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF 
THE  REGIONS:  Building  a  European  Health  Union:  Reinforcing  the  EU’s  resilience  for  cross-border 
health threats 
14 The European Health Union comprises a set of key actions, notably in the areas of: Crisis preparedness, 
reform  of  the  EU  pharmaceutical  legislation,  Europe’s  Beating  Cancer  Plan,  A  comprehensive  approach 
mental health, the European Health Data Space, Global Health Security, together with accompanying legal 
acts. See European Health Union (europa.eu) 

2 

 
 
 
• 

• 

equipping the EU and its Member States to better prevent and address future 
pandemics, 
improving the resilience of Europe’s health systems. 

In April 2020, the Emergency Support Instrument (ESI) was activated15 to enable direct 
support  to  Member  States  through  targeted  measures,  deployed  strategically  and  in  a 
coordinated  manner,  in  order  to  provide  a  comprehensive  and  flexible  response  to  the 
urgent, evolving and diverse needs emerging during the COVID-19 pandemic. 

By  its  decision  C(2020)279416,  the  Commission  authorised  the  financing  of  emergency 
support actions under the ESI Regulation and cooperated closely with the Member States 
in the implementation of the instrument.  

Measures implemented through the ESI have been the subject of a separate reporting and 
assessment by the Commission17. 

Methodology 

The  methodology18  of  the  evaluation  followed  the  five  evaluation  criteria  of 
effectiveness, efficiency, coherence, EU added value and relevance and comprised three 
main aspects: (1) an assessment of the publicly-available health programme database19 as 
well as other documents related to the Programme, in order to build an understanding of 
the  functioning  of  the  Programme;  (2)  consultations  with  stakeholders  through 
interviews,  focus  groups,  a  targeted  survey,  a  public  consultation    and  the  monitoring 
social media to understand their views on the Programme implementation; and (3) an in-
depth  analysis  of  a  subset  of  funded  actions  on  six  areas  of  the  Programme  (nutrition, 
alcohol,  health  inequalities,  antimicrobial  resistance,  health  technology  assessment  and 
vaccination).  The  areas  for  in-depth  analysis  were  selected  through  a  comprehensive 
consultation  of  health  policy  units  of  DG  SANTE,  with  a  view  to  highlighting  more 
prominent  achievements  and  draw  lessons  from  health  policy  themes  that  are  the  most 
relevant  from  health  policy  point  of  view  and  the  most  representative  of  actions 
implemented by the Programme.  

The  method  of  contribution  analysis  has  been  used  in  the  case  studies  to  identify  links 
between inputs, outcomes, and impacts of specific actions of the Programme. 

15  Council  Regulation  (EU)  2020/521  activating  emergency  support  under  Council  Regulation  (EU) 
2016/369  and  amending  its  provisions  to  finance  expenditure  necessary  to  address  the  COVID‐19 
pandemic. 
16  Commission  Decision  C(2020)2794  of  24  April  2020  on  the  financing  of  Emergency  Support  under 
Council Regulation (EU) 2016/369. 
17 REPORT FROM THE COMMISSION TO THE COUNCIL on the implementation of Regulation (EU) 
2020/521 on activating emergency support to finance necessary expenditure to address the COVID‐19 
pandemic 
18 Methodological information on how the evaluation was conducted is detailed in the Annex II to the staff 
working document. 
19 Health Programme DataBase - European Commission (europa.eu). 

3 

 
 
evaluation 

study  gathered 

The 
seven 
feedback 
groups/categories20. The participation of experts involved in the management and design 
of  the  Programme  in  interviews  and  focus  groups  was  particularly  important.  This 
approach  enabled  insights  from  relevant  stakeholder  groups  to  be  corroborated  with 
feedback from the  various  consultations to  ensure that the assessment underpinning the 
evaluation was based on reliable evidence and data.  

stakeholders 

from 

from 

An  extensive  document  review21  was  conducted  prior  to  the  stakeholder  consultations, 
providing  a  solid  basis  for  the  evaluation  and  a  steer  on  useful  lines  of  enquiry.  The 
document  review  and  desk  research  provided  a  sound  basis  for  triangulation  of 
information and confirming the findings of the subsequent consultation activities. 

Limitations to the evaluation study include the fact that it was developed mainly by using 
the  publicly-available  database  for  the  Programme,  which  only  includes  actions  funded 
through  grant  agreements  and  does  not  include  comprehensive  information,  such  as 
detailed financial data and outputs of the actions. 

The related data gaps were mitigated through the provision of additional documentation 
and  specific  data,  such  as  a  list  of  all  procurement  contracts  concluded  under  the 
Programme. For the operating grants awarded to NGOs, the final technical reports were 
made available to the contractor, as were the final reports and detailed technical results of 
a sample of actions funded by the Programme.  

These measures, combined with the triangulation of data from different sources, enabled 
evidence-based conclusions, confirmed by more than one information source. 

2.  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION ? 

2.1 

  Description of the intervention and its objectives 

According to the Treaty on the Functioning of the European Union (TFEU), a high level 
of  health  protection  must  be  ensured  in  the  definition  and  implementation  of  all  EU 
policies  (Article  168  (1)  TFEU).  EU  action,  complementing  national  policies,  must  be 
directed  towards  improving  public  health,  preventing  physical  and  mental  illness  and 
diseases, and preventing sources of danger to physical and mental health.  

20  Public  authorities  (central  government/ministries  of  health,  and  public  health  authorities  or  agencies); 
academic/research organisations; NGOs; EU citizens; patients and service users and organisations representing them; 
Consumer  organisations;  company/business  organisations;  other  (international  organisations  e.g.  WHO,  OECD; 
Healthcare  service  providers;  Organisations  presenting  healthcare  service  providers;  Healthcare  professionals’ 
associations; Independent experts). 
21  Such  as  annual  Work  Programmes,  Annual  Implementation  Reports,  work  programmes,  annual  implementation 
reports,  final  reports  of  selected  funded  actions,  final  reports  of  relevant  studies  carried  out  under  the  Programme, 
guidance  to  call  for  proposals  applicants,  funding  opportunities,  programme  dissemination  strategy,  reviews  or 
assessments  of  the  Programme  by  other  organisations  or  institutions,  DG  SANTE  and  wider  Commission  policy 
documents. 

4 

 
 
 
 
The  2014-2020  health  programme  is  the  third  programme  of  EU  action  in  the  field  of 
health,  established  by  Regulation  (EU)  No.  282/2014.  With  a  budget  of  EUR  449.422 
million  over  7  years,  it  was  the  Commission's  main  tool  to  underpin  and  support  EU 
policy coordination in the area of health during that timeframe.  

Designed to help Member States improve the health of their population, the Programme 
contributed  to  the  Europe  2020  objective  of  smart,  sustainable  and  inclusive  growth. 
While  fully  respecting  Member  States'  responsibility  for  drawing  up  their  own  health 
policies and organising the delivery of health services and medical care, the Programme 
aimed to complement, support and add value to national policies, in order to improve the 
health of people across the EU and reduce health inequalities.  

The  Programme  aimed  to  serve  Member  States’  needs  to  meet  the  Commission's 
overarching priorities:  

– 

the link between the health status of the population and its contribution to growth 
and jobs through labour market participation and labour productivity; 

– 

investment in health as a source of economic prosperity and social cohesion; 

–  societal challenges (such as demographic ageing, inequalities, burden of chronic 

diseases, effectiveness, sustainability and resilience of health systems). 

In line with the intervention logic diagram in Figure 1 below, the EU was faced, during 
the design of the Programme, with the following health challenges, (first column of the 
intervention logic):  

– 

– 

– 

– 

increasingly challenging demographic context (ageing population) threatening the 
sustainability of health systems; 

fragile  economic  recovery  limiting  the  resources  available  for  investment  in 
healthcare; 

increase of health inequalities between/within Member States; 

increase in non-communicable disease prevalence. 

To  address  these  challenges  and  Commission’s  overarching  priorities,  the  general 
objective of the Programme was (column 2 of the intervention logic): 

'to  complement,  support  and  add  value  to  the  policies  of  Member  States  to 
improve  the  health  of  EU  citizens  and  reduce  health  inequalities  by  promoting 
health, encouraging innovation in  health, increasing  the sustainability of  health 
systems  and  protecting  Union  citizens  from  serious  cross-border  health 
threats.'23. 

As shown in column 3 of the intervention logic, the Programme's four specific objectives 
with related indicators covered by Article 3 of Regulation (EU) No 282/2014, were to:  

(i)  promote  health,  prevent  diseases  and  foster  supportive  environments  for  healthy 
lifestyles;  

22 Budget expressed in 2014 prices; Programme budget in current prices is equal to EUR 452.3 million. 
23 Article 2 of Regulation (EU) No 282/2014). 

5 

 
 
(ii) protect EU citizens from serious cross-border health threats;  
(iii) contribute to innovative, efficient and sustainable health systems; and  
(iv) facilitate access to better and safer healthcare for EU citizens.  

The  4  specific  objectives  are  translated  into  4  operational  objectives  that  describe 
concretely and enable to identify the typology of operational actions to be considered for 
implementation. 

The specific objectives  of the Programme are achieved through actions in line with the 
23  thematic  priorities  listed  in  Annex  I  and  implemented  via  the  annual  work 
programmes referred to in Article 11 of the Regulation (EU) 282/201424 (column 5 of the 
intervention logic). See also Annex VI to this document.  

The  majority  of  actions  relating  to  the  specific  objective  1  (promote  health,  prevent 
diseases  and  foster  supportive  environments  for  healthy  lifestyles)  consist  of  the 
exchange  and  implementation  of  evidence-based  best  practices  geared  at  health 
promotion  and  disease  prevention.  They  are  complemented  by  the  establishment  of 
guidelines  funded  by  the  Programme  on  specific  topics,  such  as  cancer  screening, 
nutrition,  healthy  lifestyles.  These  actions,  including  studies,  are  mainly  directed  to 
Member States health authorities and policymakers. Therefore, attainment of the specific 
objective 1 is measured by the increased use of evidence-based practices at national and 
sub-national levels.  In-line with the Regulation (EU) No 282/14, relevant indicators for 
this specific objective should therefore relate to the number of Member States involved in 
health promoting and disease prevention and adopting the evidence-based practices and 
guidelines developed under the Programme. 

Activities relating  to  the  specific  objective 2  (protection of  citizens  from serious  cross-
border health threats) evolve around: risk assessment, capacity building (notably through 
the  establishment,  regular  assessment  and  strengthening  of  preparedness  plans)  against 
health threats in Member States and where appropriate cooperation with third countries, 
implementation  of  EU  legislation  on  communicable  diseases  and  other  health  threats, 
strengthening of health information and knowledge system for evidence-based decision-
making.  The  natural  stakeholders  for  actions  undertaken  by  the  Programme  under  the 
specific  objective  2  are  Member  States  health  authorities  at  national  and  sub-national 
levels.  The  implementation  of  actions  under  the  specific  objective  2  should  notably 
enable the integration of coherent approaches25 in Member States’ preparedness plans, as 
reflected by the indicator associated with this objective in the regulation establishing the 
Programme. 
As  regards  the  specific  objective  3  (contribute  to  innovative,  efficient  and  sustainable 
health systems) actions funded by the Programme relate, inter alia, to the support to EU 
cooperation  in  health  technology  assessment,  to  innovation  and  digital  health  and  to 

24  Specific  and  operational  objective  1  mapped  with  thematic  priorities  1.1-1.6;  Specific  and  operational 
objective  2,  mapped  with  thematic  priorities  2.1.-2.4;  specific  and  operational  objective  3  mapped  with 
thematic priorities 3.1-3.7; specific and operational objectives 4 mapped with thematic priorities 4.1-4.6. 
25 ie. based on agreed guidelines and practices and compatible/compliant with relevant EU legislation and 
International Health Regulations 

6 

 
 
health workforce forecasting  and planning. Other actions consisted of pooling expertise 
at EU level, supporting  healthy and active ageing, implementing EU legislation (e.g. in 
medical  devices,  medicinal  products  and  cross-border  healthcare),  developing  a  health 
information and knowledge system. The expected common result of these actions is the 
production of EU-level advice (in particular in the field of health technology assessment), 
tools (including the adoption and implementation of legislation) and mechanisms for use 
by Member States in the reform of their health systems.  
Six  types  of  actions  are  co-funded  under  the  specific  objective  4  (facilitate  access  to 
better  and  safer  healthcare  for  EU  citizens),  relating  to:  the  European  Reference 
Networks;  rare  diseases;  patient  safety  and  quality  of  healthcare;  measures  to  prevent 
antimicrobial  resistance  and control of  healthcare-associated infections; implementation 
of Union legislation in  the fields  of tissues and cells,  blood,  organs;  health information 
and knowledge system to contribute to evidence-based decision-making.  
Although  the  six  types  of  funded  actions  are  important  for  achieving  the  specific 
objective  4,  the  creation  of  European  Reference  Networks  (ERNs),  including  for  rare 
diseases, and the increase in the number of healthcare providers and centres of expertise 
joining  the  ERNs  are  considered  as  the  main  and  most  representative  results  of  this 
specific objective.  

One of the over-arching rationales of actions implemented under the 4 specific objectives 
is  to  enable  the  use  by  Member  States  of  the  knowledge  and  results  produced  by  the 
Programme, thereby helping to improve health and healthcare policies across the EU. 
The  indicators  associated  in  the  basic  act  establishing  the  Programme,  the  Regulation 
(EU)  No  282/2014,  therefore  relate  at  least  partly  to  the  ‘number  of  Member  States’ 
using the tools and knowledge produced under each of the 4 objectives into their policy-
making process.  
For  the  specific  objective  3  (contribute  to  innovative,  efficient  and  sustainable  health 
systems), the basic act included an additional indicator concerning the advice produced, 
in  particular  the  number  of  Health  Technology  Assessments  (HTA)  produced  per  year, 
taking into account the important part of HTA-related activities in this specific objective. 
Likewise,  taking  into  account  the  important  role  played  by  the  European  Reference 
Networks (ERNs), the basic act included 2 additional indicators relating to the ‘number 
of  European  reference  networks  established  in  accordance  with  Directive  2011/24/EU’ 
and  to  ‘number  of  healthcare  providers  and  centres  of  expertise  joining  European 
reference networks’. 
These  indicators26  are  relevant  and  instrumental  for  assessing  the  extent  or  degree  of 
attainment of the expected outcomes described in the intervention logic. 
The  Programme  was  managed  by  the  Commission  and  implemented  through  annual 
work  programmes  adopted  following  the  positive  opinion  of  a  Programme  Committee 
consisting  of  Member  States’  representatives.  The  implementation  was  entrusted  to  the 
Consumers, Health, Agriculture and Food Executive Agency (CHAFEA). 

26  see  sections  2.2  and  3  and  annex  VII  for  more  information  on  the  indicators  used  to  assess  the 
performance of the Programme 

7 

 
 
The executive agency  CHAFEA was closed with  effect  on 31 march 202127. Further to 
this  winding  up  of  the  agency,  the  ‘legacy  activities  (on-going  activities,  running 
projects)’ of the Programme were transferred to the newly created European Health and 
Digital  Executive  Agency  –  HaDEA,  which  succeeded  CHAFEA,  with  regard  to  the 
implementation  of  legacy/remaining  activities  of  the  Programme  and  which  was  also 
entrusted  with  the  implementation  of  the  successor  health  programme,  the  EU4Health 
Programme on the period 2021-2027. 
Table 1 below shows the financial mechanisms used by the Programme. 

Table 1: Financial mechanisms of the health programme 

Financial 
mechanism 

Project grants 

Operating grants 

Direct  grants 
to international 
organisations 

Joint actions 

Procurement 
contracts 

Presidency 
conferences 

Others 

Description 

Used to fund collaborative efforts between organisations in various EU MS that join forces to perform 
tasks to achieve a common set of objectives for a defined period of time28 29 
Contribute to the core activities of non-governmental bodies or networks, over a period equivalent to 
their accounting year30 

Awarded to international organisations, such as the WHO, to equip them with the capacity needed to 
tackle relevant health priorities 

Actions with a clear EU added value, co-financed either by competent authorities responsible for health 
in the Member States or the participating non-EU countries, or by public sector bodies and non-
governmental bodies mandated by those competent authorities31 
Also called service contracts or tenders) over specific needs related to the support of EU health policies 
(e.g. studies, development of IT tools, etc.)32  

Thematic conferences on health topics (such as personalised medicine) to mark the Presidencies of 
Council of the EU 

For example: 'Payment of membership fee and reimbursement of expert mission costs', 'Reimbursement 
of auditor mission costs", "Cross sub-delegation to EUROSTAT'. 

Figure 1: Intervention logic of the Programme  

27  Article  21  of  COMMISSION  IMPLEMENTING  DECISION  (EU)  2021/173  of  12  February  2021 
establishing  the  European  Climate,  Infrastructure  and  Environment  Executive  Agency,  the  European 
Health and Digital Executive Agency, the European Research Executive Agency, the European Innovation 
Council  and  SMEs  Executive  Agency,  the  European  Research  Council  Executive  Agency,  and  the 
European Education and Culture Executive Agency and repealing Implementing Decisions 2013/801/EU, 
2013/771/EU, 2013/778/EU, 2013/779/EU, 2013/776/EU and 2013/770/EU (OJ  OJ L 50, 15.2.2021, p. 9–
28). 
28  http://ec.europa.eu/chafea/health/index.html 
29  http://ec.europa.eu/chafea/documents/health/hp-factsheets/project-grants/factsheets-hp-pg_en.pdf 
30 http://ec.europa.eu/chafea/documents/health/hp-factsheets/operating-grants/factsheets-hp-og_en.pdf 
31  http://ec.europa.eu/chafea/documents/health/hp-factsheets/joint-actions/factsheets-hp-ja_en.pdf 
32  http://ec.europa.eu/chafea/health/tenders.html 

8 

 
 
 
 
THE 
THE 
THE 
CHALLENGES 
CHALLENGES 
CHALLENGES 
THE 
CHALLENGES 

- increasingly 
- increasingly 
- increasingly 
challenging 
challenging 
challenging 
- increasingly 
demographic 
demographic 
demographic 
challenging 
context 
context 
context 
demographic 
threating the 
threating the 
threating the 
context 
sustainability of 
sustainability of 
sustainability of 
threating the 
health systems 
health systems 
health systems 
sustainability 
of health 
systems 
- fragile 
- fragile 
- fragile 
economic 
economic 
economic 
recovery limiting 
recovery limiting 
recovery limiting 
the resources 
the resources 
the resources 
- fragile 
available for 
available for 
available for 
economic 
investment in 
investment in 
investment in 
recovery 
healthcare 
healthcare 
healthcare 
limiting the 
resources 
available for 
- increase of 
- increase of 
- increase of 
investment in 
health 
health 
health 
healthcare 
inequalities 
inequalities 
inequalities 
between/within 
between/within 
between/within 
Member States 
Member States 
Member States 

- increase of 
health 
- increase in 
- increase in 
- increase in 
inequalities 
chronic diseases 
chronic diseases 
chronic diseases 
between/withi
prevalence  
prevalence  
prevalence  
n Member 
States 

- increase in 
chronic 
diseases 
prevalence  

GENERAL 
GENERAL 
GENERAL 
GENERAL 
GENERAL 
GENERAL 
OBJECTIVES 
OBJECTIVES 
OBJECTIVES 
OBJECTIVES 
OBJECTIVES 
OBJECTIVES 

Complement, 
Complement, 
Complement, -
Complement, 
Complement, -
Complement, -
support and 
support and 
support and 
support and 
support and 
support and add 
add value to the 
add value to the 
add value to the 
add value to the 
add value to the 
value to the 
policies of 
policies of 
policies of 
policies of 
policies of 
policies of 
Member States  
Member States  
Member States  
Member States  
Member States  
Member States  

- to improve the 
- to improve the 
- to improve the 
- to improve the 
- to improve the 
- to improve the 
health of EU 
health of EU 
health of EU 
health of EU 
health of EU 
health of EU 
citizens and  
citizens and  
citizens and  
citizens and  
citizens and  
citizens and  

-reduce health 
-reduce health 
- reduce health 
-reduce health 
-reduce health 
-reduce health 
inequalities  
inequalities  
inequalities  
inequalities  
inequalities  
inequalities  

Budget: €449.4 million 
Budget: €449.4 
Budget: €449.4 
Budget: €449.4 
Budget: €449.4 
(2014-2020) 
million (2014-2020) 
million (2014-2020) 
million (2014-2020) 
million (2014-2020) 

Management mode: 
Centralised  direct and 
Management 
indirect management  
Management 
Management 
Management 
mode : Centralised  
mode : Centralised  
mode : Centralised  
mode : Centralised  
Programming and 
direct and indirect 
direct and indirect 
direct and indirect 
direct and indirect 
implementation on 
Budget: €449.4 
management  
management  
management  
management  
the basis of adoption 
million (2014-2020) 
of Annual Work 
Programmes through 
Programming and 
Programming and 
Programming and 
Programming and 
implementing acts 
Management 
implementation 
implementation 
implementation 
implementation 
Monitoring and 
mode : Centralised  
on the basis of 
on the basis of 
on the basis of 
on the basis of 
reporting Mid-term 
direct and indirect 
adoption of Annual 
adoption of Annual 
adoption of Annual 
adoption of aAnnual 
management  
review in 2017 
Work Programmes 
Work Programmes 
wWork 
Work Programmes 
through 
Annual 
through 
through 
pProgrammes 
implementing acts 
implementation report 
implementing acts 
implementing acts 
through 
Programming and 
sent to EP& Council 
implementing acts 
implementation 

SPECIFIC 
SPECIFIC 
SPECIFIC 
SPECIFIC 
SPECIFIC 
SPECIFIC 
OBJECTIVES 
OBJECTIVES 
OBJECTIVES 
OBJECTIVES 
OBJECTIVES 
OBJECTIVES 

1) Promote health, 
1) Promote health, 
1) Promote health, 
1) Promote health, 
1) Promote health, 
1) Promote health, 
prevent disease and 
prevent disease and 
prevent disease and 
prevent disease and 
prevent disease and 
prevent disease and 
foster supportive 
foster supportive 
foster supportive 
foster supportive 
foster supportive 
foster supportive 
environments for 
environments for 
environments for 
environments for 
environments for 
environments for 
healthy lifestyles 
healthy lifestyles 
healthy lifestyles 
healthy lifestyles 
healthy lifestyles 
healthy lifestyles 

2) Protect citizens 
2) Protect citizens 
2) Protect citizens 
from serious cross-
from serious cross-
from serious cross-
border health 
border health 
border health 
threats 
threats 
threats 

2) Protect citizens 
2) Protect citizens 
2) Protect citizens 
from serious cross-
from serious cross-
from serious cross-
border health 
border health 
border health 
3) Contribute to 
3) Contribute to 
3) Contribute to 
threats 
threats 
threats 
innovative, efficient 
innovative, efficient 
innovative, efficient 
and sustainable 
and sustainable 
and sustainable 
health systems 
health systems 
health systems 

3) Contribute to 
3) Contribute to 
3) Contribute to 
innovative, efficient 
innovative, efficient 
innovative, efficient 
and sustainable 
and sustainable 
and sustainable 
health systems 
health systems 
health systems 

4) Facilitate access 
4) Facilitate access 
4) Facilitate access 
to better and safer 
to better and safer 
to better and safer 
healthcare for 
healthcare for 
healthcare for 
Union citizens 
Union citizens 
EUUnion citizens 

4) Facilitate access 
4) Facilitate access 
4) Facilitate access 
to better and safer 
to better and safer 
to better and safer 
healthcare for 
healthcare for 
healthcare for 

OPERATIONAL OBJECTIVES 
OPERATIONAL OBJECTIVES 
OPERATIONAL OBJECTIVES 
OPERATIONAL 
OPERATIONAL 
OPERATIONAL 
OBJECTIVES 
OBJECTIVES 
OBJECTIVES 
1) Identify, disseminate and promote the 
1) Identify, disseminate and promote the 
1) Identify, disseminate and 
up-take of evidence-based and good 
up-take of evidence-based and good 
promote the up-take of evidence-
practices for cost-effective disease 
practices for cost-effective disease 
1) Identify, disseminate and 
1) Identify, disseminate and 
1) Identify, disseminate and 
prevention and health promotion 
prevention and health promotion 
based and good practices for 
promote the up-take of evidence-
promote the up-take of evidence-
promote the up-take of evidence-
measures by addressing in particular the 
measures by addressing in particular the 
cost-effective disease prevention 
based and good practices for 
based and good practices for 
based and good practices for 
key lifestyle related risk factors with a 
key lifestyle related risk factors with a 
and health promotion measures 
focus on the Union added value.  
focus on EUthe Union added value.  
cost-effective disease prevention 
cost-effective disease prevention 
cost-effective disease prevention 
by addressing in particular the 
and health promotion measures 
and health promotion measures 
and health promotion measures 
key lifestyle related risk factors 
by addressing in particular the 
by addressing in particular the 
by addressing in particular the 
with a focus on the Union added 
key lifestyle related risk factors 
key lifestyle related risk factors 
key lifestyle related risk factors 
value.  
with a focus on the Union added 
with a focus on the Union added 
with a focus on the Union added 
value.  
value.  
value.  

2) Identify and develop coherent 
2) Identify and develop coherent 
2) Identify and develop coherent 
approaches and promote their 
approaches and promote their 
approaches and promote their 
implementation for better preparedness 
implementation for better preparedness 
implementation for better 
and coordination in health emergencies. 
and coordination in health emergencies. 
preparedness and coordination 
2) Identify and develop coherent 
2) Identify and develop coherent 
2) Identify and develop coherent 
in health emergencies. 
approaches and promote their 
approaches and promote their 
approaches and promote their 
implementation for better 
implementation for better 
implementation for better 
preparedness and coordination 
preparedness and coordination 
preparedness and coordination 
in health emergencies. 
in health emergencies. 
in health emergencies. 
3) Identify and develop tools and 
3) Identify and develop tools and 
3) Identify and develop tools and 
mechanisms at Union level to 
mechanisms at Union EU level to address 
mechanisms at Union level to address 
address shortages of resources, 
shortages of resources, both human and 
shortages of resources, both human and 
both human and financial, and 
financial, and facilitate the voluntary up-
financial, and facilitate the voluntary up-
take of innovation in public health 
take of innovation in public health 
facilitate the voluntary up-take 
intervention and prevention strategies. 
intervention and prevention strategies. 
of innovation in public health 
intervention and prevention 
strategies. 

3) Identify and develop tools and 
3) Identify and develop tools and 
3) Identify and develop tools and 
mechanisms at Union level to 
mechanisms at Union level to 
mechanisms at Union level to 
4) Increase access to medical 
address shortages of resources, 
address shortages of resources, 
address shortages of resources, 
expertise and information for 
both human and financial, and 
both human and financial, and 
both human and financial, and 
specific conditions also beyond 
facilitate the voluntary up-take 
facilitate the voluntary up-take 
facilitate the voluntary up-take 
national borders, facilitate the 
of innovation in public health 
of innovation in public health 
of innovation in public health 
application of the results of 
intervention and prevention 
intervention and prevention 
intervention and prevention 
4) Increase access to medical expertise and 
4) Increase access to medical expertise and 
research and develop tools for 
strategies. 
strategies. 
strategies. 
information for specific conditions also 
information for specific conditions also 
the improvement of healthcare 
beyond national borders, facilitate the 
beyond national borders, facilitate the 
application of the results of research and 
application of the results of research and 
quality and patient safety 
develop tools for the improvement of 
develop tools for the improvement of 
through, inter alia, actions 
healthcare quality and patient safety 
healthcare quality and patient 
4) Increase access to medical 
4) Increase access to medical 
4) Increase access to medical 
contributing to improve health 
through, inter alia, actions contributing to 
safety, including through, inter 
expertise and information for 
expertise and information for 
expertise and information for 
literacy. 
improve health literacy. 
9 
alia,  actions contributing to 
specific conditions also beyond 
specific conditions also beyond 
specific conditions also beyond 
improve health literacy. 
national borders, facilitate the 
national borders, facilitate the 
national borders, facilitate the 
application of the results of 
application of the results of 
application of the results of 
research and develop tools for 
research and develop tools for 
research and develop tools for 

ACTIONS (see annex I of the 
ACTIONS (see annex I of the 
ACTIONS (see annex I of the 
ACTIONS (see annex I of the 
ACTIONS (see annex I of the 
ACTIONS (see Aannex I of the 
Programme Regulation) 
Programme Regulation) 
Programme Regulation) 
Programme Regulation) 
Programme Regulation) 
Programme Regulation) 

1.1 Risk factors such as use of tobacco and 
1.1 Risk factors such as use of tobacco and 
1.1 Risk factors such as use of tobacco and 
1.1 Risk factors such as use of tobacco and passive 
1.1 Risk factors such as use of tobacco and passive 
1.1 Risk factors such as use of tobacco and passive 
smoking, harmful use of alcohol, unhealthy dietary 
smoking, harmful use of alcohol, unhealthy dietary 
smoking, harmful use of alcohol, unhealthy dietary 
passive smoking, harmful use of alcohol, 
passive smoking, harmful use of alcohol, 
passive smoking, harmful use of alcohol, 
habits and physical inactivity. 
habits and physical inactivity. 
habits and physical inactivity. 
unhealthy dietary habits and physical 
unhealthy dietary habits and physical 
unhealthy dietary habits and physical 
1.2. Drugs-related health damage, including 
1.2. Drugs-related health damage, including 
1.2. Drugs-related health damage, including 
inactivity. 
inactivity. 
inactivity. 
information and prevention. 
information and prevention. 
information and prevention. 
1.2. Drugs-related health damage, including 
1.2. Drugs-related health damage, including 
1.2. Drugs-related health damage, including 
1.3. HIV/AIDS, tuberculosis and hepatitis  
1.3. HIV/AIDS, tuberculosis and hepatitis  
1.3. HIV/AIDS, tuberculosis and hepatitis  
information and prevention. 
information and prevention. 
information and prevention. 
1.4. Chronic diseases including cancer, age-related 
1.4. Chronic diseases including cancer, age-related 
1.4. Chronic diseases including cancer, age-related 
1.3. HIV/AIDS, tuberculosis and hepatitis  
1.3. HIV/AIDS, tuberculosis and hepatitis  
1.3. HIV/AIDS, tuberculosis and hepatitis  
diseases and neurodegenerative diseases 
diseases and neurodegenerative diseases 
diseases and neurodegenerative diseases 
1.4. Chronic diseases including cancer, age-
1.4. Chronic diseases including cancer, age-
1.4. Chronic diseases including cancer, age-
1.5. Tobacco legislation 
1.5. Tobacco legislation 
1.5. Tobacco legislation 
1.6. Health information and knowledge system to 
1.6. Health information and knowledge system to 
1.6. Health information and knowledge system to 
related diseases and neurodegenerative 
related diseases and neurodegenerative 
related diseases and neurodegenerative 
contribute to evidence-based decision making 
contribute to evidence-based decision making 
contribute to evidence-based decision making 
diseases 
diseases 
diseases 
2.1 Risk assessment additional capacities for 
2.1 Risk assessment additional capacities for 
2.1 Risk assessment additional capacities for 
1.5. Tobacco legislation 
1.5. Tobacco legislation 
1.5. Tobacco legislation 
scientific expertise 
scientific expertise 
scientific expertise 
1.6. Health information and knowledge 
1.6. Health information and knowledge 
1.6. Health information and knowledge 
2.2. Capacity building against health threats in 
2.2. Capacity building against health threats in 
2.2. Capacity building against health threats in 
system to contribute to evidence-based 
system to contribute to evidence-based 
system to contribute to evidence-based 
Member States, including, where appropriate, 
Member States, including, where appropriate, 
Member States, including, where appropriate, 
decision making 
decision making 
decision making 
cooperation with neighbouring countries 
cooperation with neighbouring countries 
cooperation with neighbouring countries 
2.1 Risk assessment additional capacities for 
2.1 Risk assessment additional capacities for 
2.1 Risk assessment additional capacities for 
2.3. Implementation of Union legislation on 
2.3. Implementation of Union legislation on 
2.3. Implementation of Union EU legislation on 
communicable diseases and other health threats, 
communicable diseases and other health threats, 
communicable diseases and other health threats, 
scientific expertise 
scientific expertise 
scientific expertise 
including those caused by biological, and chemical 
including those caused by biological, and chemical 
including those caused by biological, and chemical 
2.2. Capacity building against health threats 
2.2. Capacity building against health threats 
2.2. Capacity building against health threats 
incidents, environment and climate change 
incidents, environment and climate change 
incidents, environment and climate change 
in Member States, including, where 
in Member States, including, where 
in Member States, including, where 
2.4 Health information and knowledge system to 
2.4 Health information and knowledge system to 
2.4 Health information and knowledge system to 
appropriate, cooperation with neighbouring 
appropriate, cooperation with neighbouring 
appropriate, cooperation with neighbouring 
contribute to evidence-based decision making 
contribute to evidence-based decision making 
contribute to evidence-based decision making 
countries 
countries 
countries 
3.1 HTA 
3.1 HTA 
3.1 HTA 
2.3. Implementation of Union legislation on 
2.3. Implementation of Union legislation on 
2.3. Implementation of Union legislation on 
3.2 Innovation and e-health 
3.2 Innovation and e-health 
3.2 Innovation and e-health 
communicable diseases and other health 
communicable diseases and other health 
communicable diseases and other health 
3.3 Health workforce forecasting and planning 
3.3 Health workforce forecasting and planning 
3.3 Health workforce forecasting and planning 
3.4 Setting up a mechanism for pooling expertise at 
3.4 Setting up a mechanism for pooling expertise at 
3.4 Setting up a mechanism for pooling expertise at 
threats, including those caused by biological, 
threats, including those caused by biological, 
threats, including those caused by biological, 
Union level 
Union level 
Union EU level 
and chemical incidents, environment and 
and chemical incidents, environment and 
and chemical incidents, environment and 
3.5 European Innovation Partnership on Active and 
3.5 European Innovation Partnership on Active and 
3.5 European Innovation Partnership on Active and 
climate change 
climate change 
climate change 
Healthy Ageing 
Healthy Ageing 
Healthy Ageing 
2.4 Health information and knowledge 
2.4 Health information and knowledge 
2.4 Health information and knowledge 
3.6 Implementation of Union legislation in the field 
3.6 Implementation of Union legislation in the field 
3.6 Implementation of Union EU legislation in the 
system to contribute to evidence-based 
system to contribute to evidence-based 
system to contribute to evidence-based 
of medical devices, medicinal products and cross-
of medical devices, medicinal products and cross-
field of medical devices, medicinal products and 
decision making 
decision making 
decision making 
border healthcare 
border healthcare 
cross-border healthcare 
3.1 HTA 
3.1 HTA 
3.1 HTA 
3.7 Health information and knowledge system 
3.7 Health information and knowledge system 
3.7 Health information and knowledge system 
including support to the Scientific Committees set 
including support to the Scientific Committees set 
including support to the Scientific Committees set 
3.2 Innovation and e-health 
3.2 Innovation and e-health 
3.2 Innovation and e-health 
up in accordance with Commission Decision 
up in accordance with Commission Decision 
up in accordance with Commission Decision 
3.3 Health workforce forecasting and 
3.3 Health workforce forecasting and 
3.3 Health workforce forecasting and 
2008/721/EC 
2008/721/EC 
2008/721/EC 
planning 
planning 
planning 
3.4 Setting up a mechanism for pooling 
3.4 Setting up a mechanism for pooling 
3.4 Setting up a mechanism for pooling 
expertise at Union level 
expertise at Union level 
expertise at Union level 
3.5 European Innovation Partnership on 
3.5 European Innovation Partnership on 
3.5 European Innovation Partnership on 
Active and Healthy Ageing 
Active and Healthy Ageing 
Active and Healthy Ageing 
4.1 European Reference Networks 
4.2 Rare diseases 
3.6 Implementation of Union legislation in 
3.6 Implementation of Union legislation in 
3.6 Implementation of Union legislation in 
4.1 European Reference Networks 
4.1 European Reference Networks 
4.3 Patient safety and quality of healthcare 
the field of medical devices, medicinal 
the field of medical devices, medicinal 
the field of medical devices, medicinal 
4.2 Rare diseases 
4.2 Rare diseases 
4.4 Measures to prevent antimicrobial resistance 
products and cross-border healthcare 
products and cross-border healthcare 
products and cross-border healthcare 
4.3 Patient safety and quality of healthcare 
4.3 Patient safety and quality of healthcare 
and control healthcare-associated infections 
4.4 Measures to prevent Antimicrobial resistance 
4.4 Measures to prevent aAntimicrobial resistance 
3.7 Health information and knowledge 
3.7 Health information and knowledge 
3.7 Health information and knowledge 
4.5 Implementation of Union legislation in 
and control healthcare-associated infections 
and control healthcare-associated infections 
system including support to the Scientific 
system including support to the Scientific 
system including support to the Scientific 
the fields of tissues and cells, blood, organs 
4.5 Implementation of Union legislation in 
4.5 Implementation of Union EU legislation 
Committees set up in accordance with 
Committees set up in accordance with 
Committees set up in accordance with 
4.6 Health information and knowledge 
the fields of tissues and cells, blood, organs 
in the fields of tissues and cells, blood, organs 
Commission Decision 2008/721/EC 
Commission Decision 2008/721/EC 
Commission Decision 2008/721/EC 
system to contribute to evidence-based 
4.6 Health information and knowledge 
4.6 Health information and knowledge 
4.1 European Reference Networks 
4.1 European Reference Networks 
4.1 European Reference Networks 
decision making 
system to contribute to evidence-based 
system to contribute to evidence-based 
4.2 Rare diseases 
4.2 Rare diseases 
4.2 Rare diseases 
decision making 
decision making 
4.3 Patient safety and quality of healthcare 
4.3 Patient safety and quality of healthcare 
4.3 Patient safety and quality of healthcare 
4.4 Measures to prevent Antimicrobial 
4.4 Measures to prevent Antimicrobial 
4.4 Measures to prevent Antimicrobial 
resistance and control healthcare-associated 
resistance and control healthcare-associated 
resistance and control healthcare-associated 

RESULTS 
RESULTS 
RESULTS 
RESULTS 
RESULTS 
RESULTS 

1. Increased use of 
1. Increased use of 
1. Increased use of 
1. Increased use of 
1. Increased use of 
1. Increased use of 
evidence-based 
evidence-based 
evidence-based 
evidence-based 
evidence-based 
evidence-based 
practices at the 
practices at the 
practices at the 
practices at the 
practices at the 
practices at the 
appropriate level in MS 
appropriate level in MS 
appropriate level in MS 
appropriate level in MS 
appropriate level in MS 
appropriate level in 
MS 

2. Coherent 
2. Coherent approaches 
2. Coherent approaches 
integrated into the MS 
integrated in the MS 
approaches 
preparedness plans 
preparedness plans 
integrated in the MS 
preparedness plans 
2. Coherent approaches 
2. Coherent approaches 
2. Coherent approaches 
integrated in the MS 
integrated in the MS 
integrated in the MS 
preparedness plans 
preparedness plans 
preparedness plans 

3. Increased 
3. Increased production 
3. Increased production 
production of advice 
of advice and use of 
of advice and use of 
and use of developed 
developed tools and 
developed tools and 
mechanisms by MS in 
mechanisms by MS in 
tools and 
the reform of their 
the reform of their 
mechanisms by MS 
health systems 
health systems 
in the reform of their 
health systems 
3. Increased production 
3. Increased production 
3. Increased production 
of advice and use of 
of advice and use of 
of advice and use of 
developed tools and 
developed tools and 
developed tools and 
mechanisms by MS in 
mechanisms by MS in 
mechanisms by MS in 
the reform of their 
the reform of their 
the reform of their 
health systems 
health systems 
health systems 
4. Creation of the 
European Reference 
Networks including 
4. Creation of the 
4. Creation of the 
for rare diseases, and 
European Reference 
European Reference 
increasing number 
Networks including for 
Networks including for 
of healthcare 
rare diseases, and 
rare diseases, and 
providers and 
increasing number of 
increasing number of 
centres of expertise 
4. Creation of the 
4. Creation of the 
4. Creation of the 
healthcare providers 
healthcare providers 
joining the ERNs 
European Reference 
European Reference 
European Reference 
and centres of 
and centres of 
Networks including for 
Networks including for 
Networks including for 
expertise joining the 
expertise joining the 
rare diseases, and 
rare diseases, and 
rare diseases, and 
ERNs 
ERNs 
increasing number of 
increasing number of 
increasing number of 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
2.2  Point(s) of comparison  

The final  evaluation  of the  third health programme (the  Programme) uses as  a point of 
comparison,  the  results  of  the  final  evaluation  of  the  second  health  programme  (2008-
2013)33, combined with the findings of the Programme’s mid-term evaluation34.  
According  to  the  results  of  its  final  evaluation,  the  second  health  programme  had  very 
broad objectives and a large scope. This implied that the themes addressed by the funded 
actions were all relevant. This large scope and lack of focus meant that the programme’s 
limited resources  were  thinly  spread among  a  high number of funded actions, affecting 
the ability to reach a critical mass and to have a major impact. 
To  address  this  issue,  the  Programme  focused  on  certain  key  objectives  and  on  fewer 
actions with proven EU added value and with the following characteristics:  

- 

- 

- 

- 

a focus on fostering the  exchange of best  practices between Member States and 
supporting  networks  for  knowledge  sharing  or  mutual  learning.  (specific 
objectives 1, 3 and 4); 
addressing cross-border health threats to reduce risks and mitigate consequences 
(specific objective 2); 
addressing  issues  relating  to  the  internal  market  where  the  EU  has  substantial 
legitimacy to  ensure high-quality  solutions  across Member States and actions  to 
unlock the potential of innovation in health (specific objectives 3 and 4);  
actions that could lead to a system of benchmarking or improving economies of 
scale  by  avoiding  waste  due  to  duplication  and  optimising  the  use  of  financial 
resources. (specific objectives 1, 3 and 4). 

The mid-term evaluation of the Programme concluded that implementation was on track, 
striving to maximise synergies with other EU policies and financial instruments such as 
Horizon 2020. All thematic priorities remained  valid, and most actions delivered useful 
outcomes  with  high  EU-added  value,  in  particular  for  crisis  management  and  for  the 
safety and security in Europe. Compared to the second Health Programme, the 3rd Health 
programme  demonstrated  its  value  as  an  effective  and  flexible  instrument.  However, 
there was scope to further streamline the added-value criteria to focus on three key areas: 
addressing  serious  cross-border  health  threats,  improving  economies  of  scale,  and 
fostering  the  exchange  and  implementation  of  best  practices.  The  mid-term  evaluation, 
notably recommended to keep the Programme focused, to improve the arrangements for 
the monitoring of the programme’s implementation, to increase participation of low-GNI 
Member States and underrepresented organisations. 

33 Ex-post evaluation of the 2nd Health Programme 2008-2013 (europa.eu) 
34 Commission report - Mid-term evaluation of the 3rd Health Programme 2014-2020 (europa.eu) 

10 

 
 
 
 
 
 
2.2.1  Under  specific  objective  1  for  promoting  health  and  preventing  diseases,  the 
Programme  aimed  at  identifying,  disseminating  and  promoting  the  uptake  of  evidence-
based best practices, including measures to address health inequalities. 
According  to  Article  3(1)  of  Regulation  (EU)  No.  282/2014,  the  attainment  of  this 
specific  objective  is  measured,  in  particular,  through  'the  increase  in  the  number  of 
Member  States  involved  in  health  promotion  and  disease  prevention,  using  evidence-
based and good practices through measures and actions taken at the appropriate level in 
Member States'. 
At  implementation  level,  this  indicator  has  been  broken  down  into  the  two  (proxy) 
following components or sub-indicators:  

-  number  of  Member  States  with  a  national  initiative  on  reducing  saturated  fat  in 

food; 

-  number  of  Member  States  implementing  the  European  accreditation  scheme  for 

breast cancer services. 

The baselines and targets for the two sub-indicators, established at the beginning of the 
Programme implementation, are presented in the box below. 

Sub-indicator 1: Saturated fat 
Baseline: 12 Member States have had a national  initiative on the reduction of saturated 
fat in 2013. 
Target:  By  2020,  all  EU-28  should  have  had  a  national  initiative  on  the  reduction  of 
saturated fat35. 

Sub-indicator  2:  Implementation  of  the  European  accreditation  scheme  for  breast 
cancer services  
Baseline:  0  Member  States  implemented  the  European  accreditation  scheme  for  breast 
cancer services in 2013. 
Target: by 2020 all EU-28 should have implemented the accreditation scheme for breast 
cancer services. 

2.2.2  Under  specific  objective  2  on  protecting  citizens  from  cross-borderborder 
health threats, the Programme aimed at identifying and developing coherent approaches 
and  support  and  promoting  their  implementation  to  contribute  to  a  better  preparedness 
and coordination in health emergencies. 

Supported actions under this objective have identified gaps in Member States’ capacities 
and  helped  prioritise  actions  and  implement  capacity  building  activities  to  fill  in  those 
gaps.  

In line with Article 3(2) of Regulation  (EU) No. 282/2014, the achievement of specific 
objective  2  is  measured,  in  particular,  through  ‘the  increase  in  the  number  of  Member 
States integrating coherent approaches in the design of their preparedness plans”, 

The  integration  by  Member  States  of  coherent  approaches  in  the  design  of  their 
preparedness plans means the inclusion, the taking into account and implementation by 
Member  States,  when  designing  their  preparedness  plans,  of  guidelines,  protocols  and 
tools concerning :  

35 including through the exchange and implementation of best practices co-funded by the Programme. 

11 

 
 
 
- 
- 

- 

- 

the improvement of risk assessment capacities as regard health threats,  
capacity-building against health threats, including where appropriate, cooperation 
with  other  Member  States  and  neighbouring  countries;  taking  into  account,  and 
coordinating with global initiatives; public health response coordination (notably 
through  the  promotion  of  the  interoperability  of  preparedness  systems);  non-
binding  approaches  on  vaccination;  addressing  the  increasing  health  threats 
resulting  from  global  population  movements;  guidelines  development  on 
protective  measures  in  an  emergency  situation;  information  and  guides  to  good 
practice; the contribution to the framework for a voluntary mechanism, including 
the  introduction  of  optimal  vaccination  coverage  to  effectively  combat  the 
resurgence  in  infectious  diseases  and  for  the  joint  procurement  of  medical 
countermeasures; development of coherent communication strategies. 
actions required by, or contributing to the implementation of Union legislation in 
the  fields  of  communicable  diseases  and  other  health  threats,  including  those 
caused by biological and chemical incidents, environment and climate change.  
fostering  a  health  information  and  knowledge  system  to  contribute  to  evidence-
based decision-making, 

These guidelines, protocols and tools are tailored to the specific needs and gaps of each 
Member State, which are identified through regular training sessions, workshops, round 
tables, simulations and command post exercises aimed at testing established procedures, 
tools and systems for reporting, monitoring and assessing risks and threats to people and 
communicating  on  these  risks  and  threats.  The  command  post  exercises  serve  to  test 
multi-sector arrangements for responding to an outbreak. 
Through the identification of capacity gaps, these activities are aimed at contributing to a 
better preparedness  of  Member States  and to  their capacity to  respond to health threats 
and  at  implementing  EU  legislation  –  Decision  1082/2013/EU  -  and  international 
regulations (International Health Regulations - IHR36 2005) that seek to contribute to the 
development of an adequate management of serious cross-border health threats. 

The coherent approaches adopted by Member States are therefore in line with the above-
described  guidelines,  protocols  and  tools  resulting  from  actions  funded  by  the 
Programme  and  also  compliant  with  international  health  regulations  adopted  by  the 
WHO. 

The baseline and target for this indicator, established at the beginning of the Programme 
implementation, are presented in the box below: 

Baseline: in 2014, 0 Member States had integrated coherent approaches into the design 
of its preparedness plans. 
Target: by 2020, all EU-28 Member States should have integrated coherent approaches 
into the design of their preparedness plans. 

36 International Health Regulations (2005) (who.int) 

12 

 
 
 
 
 
 
 
 
2.2.3  Under  specific  objective  3  on  contributing  to  innovative,  efficient  and 
sustainable  health  systems,  the  Programme  aimed  to  identify  and  develop  tools  and 
mechanisms  at  EU  level  to  address  shortages  of  human  and  financial  resources,  and 
facilitate the voluntary uptake of innovation in public health intervention and prevention 
strategies.  The  main  ones  are  health  technology  assessment,  eHealth,  the  European 
Innovation Partnership on active and healthy ageing, the Expert Panel on effective ways 
of investing in health and the Commission’s scientific committees. 

In line with Article 3(3) of Regulation  (EU) No. 282/2014, the achievement of specific 
objective 3 is measured, in particular, through ‘the increase in the advice produced and 
the  number  of  Member  States  using  the  tools  and  mechanisms  identified  in  order  to 
contribute to effective results in their health systems’. 

The indicator relates to HTA advice produced by the joint assessment actions undertaken 
and  completed  by  the  joint  actions  carried  out  under  the  Programme.  The  EUnetHTA 
JA3  (European  Network  for  Health  Technology  Assessment  Joint  Action  3)  was 
launched with the participation of 61 partners and 16 affiliated partners from 26 Member 
States, NO and the UK. 

At implementation level, this indicator is broken down into the following sub-indicators:  

-  Advice produced by actions launched under the Programme (e.g., the EUnetHTA 
3 ), in particular the number of health technology assessments  produced per year. 
-  Number  of  Member  States  using  the  identified  tools  and  mechanisms  to 
contribute to effective results in their health systems. This indicator is measured 
by  the  number  of  Member  States  that  have  effectively  integrated  and 
implemented 
tools  and  mechanisms  produced  by  HTA  actions  (advice, 
assessment results and recommendations) in their health systems. 

The  baselines  and  targets  for  these  sub-indicators,  established  at  the  beginning  of  the 
Programme implementation, are presented in the box below. 

Number of Member States using the identified tools and mechanisms (i.e. the joint 
HTA  reports  and  recommendations)37to  contribute  to  effective  results  in  their 
health systems. 
Baseline: 0 in 2013 
Target in 2020: 18 Member States 

Health technology assessment (HTA) 
Baseline: 2 HTA per year in 2013 at EU level 
Target in 2020: 50 HTA annually involving all EU-28 at EU level 

2.2.4  Under specific objective 4 on facilitating access to better and safer healthcare, 
the  aim  was  to  increase  access  to  medical  expertise  and  information  for  specific 
conditions, also beyond national borders, to facilitate the application of research results38 

37 Advice, assessment and recommendations to support evidence-based, sustainable and equitable choices 
in  healthcare  and  health  technologies  (see  the  following  hyperlink  for  more  information  on  the  tools 
produced and the governance of EUnetHTA3 joint action: ev_20201027_co07_en_0.pdf (europa.eu) 
38 These refer to the results of health-related research activities carried out at EU and/or at national level 

13 

 
 
 
and to improve healthcare quality and patient safety through actions on health literacy. 

In line with Article 3(4) of Regulation (EU) No. 282/2014, the achievement of specific 
objective 4 is measured, in particular, through  ‘the increase in the number of European 
Reference  Networks  (ERNs)  established  under  Directive  2011/24/EU  of  the  European 
Parliament and of the Council39’, ‘the increase in the number of healthcare providers and 
centres  of  expertise  joining  European  Reference  Networks’,  and  ‘the  increase  in  the 
number of Member States using the tools developed’40. 

The baselines and targets for these indicators, established at the beginning of Programme 
implementation, are presented in the box below. 

Number  of  European  ReferenceReference  Networks  (ERNs)  established 
accordance with Directive 2011/24/EU 
Baseline: 0 ERNs in 2014 
Target in 2020: 30 ERNs 

in 

Number of healthcare providers and centres of expertise joining the ERNs 
Baseline: 0 in 2014 
Target in 2020: 1450 

Number of Member States using the tools developed: 
Baseline: 0 in 2014 
Target in 2020: 28 Member States or 27, taking account of Brexit41 

3.  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? 

At the end of 2020, 100% of the allocated budget (EUR 452.3 million42 for 2014-2020) 
was committed for projects and other actions pursuing the Programme’s objectives. 67% 
of the total budget (i.e. EUR 305.3 million) was paid to beneficiaries or used to procure 
services  necessary  for  the  implementation  of  the  Programme.  Outstanding  payments 
(33% of the budget or EUR 147 million) account for projects or actions launched in the 
last  2-3  years  which  have  not  yet  been  completed  and  therefore  have  not  led  to  final 
payments by the Commission. 

Directive 2011/24/EU of the European Parliament and of the Council of 9 March 2011 on the application of patients' 
rights in cross-border healthcare (OJ L 88, 4.4.2011, p. 45). 
40 Tools developed within ERNs for diagnosis, clinical trials and treatment, such as the IT-based Clinical 
Patient Management System (CPMS) 
41 The transition period for the Brexit ended on 31/12/2020. 
42 The budget amounts to EUR 452.3 million in current prices and EUR 449.4 million in 2014 prices. 

14 

 
 
 
 
 
 
 
 
Table 2: Overview of financial programming over the implementation period 

2014 

2015 

Financial Programming (EUR million) 
2016 

2019 

2018 

2017 

2020 

Administrative support 
Operational 
appropriations 
Executive Agency 
Total 
(*) Figure in 2014 prices 

1,5 
52,9 

4,2 
58,6 

1,5 
54,0 

4,2 
59,8 

1,5 
56,5 

4,2 
62,2 

1,5 
58,8 

4,2 
64,5 

1,5 
60,5 

4,4 
66,4 

1,5 
62,3 

4,6 
68,3 

Total 
Programme 
10,5 
408,5 

1,5 
63,6 

4,6 
69,7 

30,3 
449,4(*) 

The  Programme  addressed  four  specific  objectives  in  line  with  the  Commission’s 
priorities, namely: the implementation of best practices for health promotion and disease 
prevention;  crisis  preparedness  and  risk  management;  contribution  to  innovative, 
efficient  and  sustainable  health  systems;  and  facilitation  of  access  to  better  and  safer 
healthcare. 

Despite  a  relatively  small  budget43,  the  Programme  contributed  to  a  better  health 
protection through its policies and activities, in accordance with Article 168 TFEU.  

A  majority  of  indicators44  and  sub-indicators  (except  for  one  sub-indicator45)  for 
assessing  progress  towards  the  Programme’s  general  and  specific  objectives  improved 
over  the  implementation  period  (2014-2020),  with  many  of  them  meeting  their  2020 
targets. 

A comprehensive list of Programme indicators is set out in Annex VII to this document. 

3.1. 

Specific  objective  1:  promote  health,  prevent  diseases,  and  foster  supportive 
environments for healthy lifestyles 

3.1.1. Indicators 
28% (EUR 127 million) of the overall Programme budget was allocated to actions falling 
under specific objective 1. EUR 8 million was spent on cancer, including on supporting 
screening  programmes  in  the  MS,  improving  the  quality  of  patients’  lives,  addressing 
survivorship issues, assessing the impact of cancer research and facilitating the uptake of 
innovative treatments. EUR 11 million was allocated to major risk factors of cancer and 
other  chronic  diseases,  i.e.  alcohol  and  tobacco  control,  physical  activity  and  nutrition. 
Other  activities  aimed  at  preventing  chronic  diseases  included  promoting  the  EU-wide 
uptake of validated best practices in preventing cardiovascular disease and diabetes. The 
Programme also developed the EU-Compass for Action on mental health and wellbeing 

43 As compared to health-related expenditure under the European structural and investment funds or  with 
the budget of the ongoing EU4Health Programme (2021-2027) 
44 The full list of indicators is presented in annex VII to this document 
45  Indicator: number  of  Member  States  involved  in  health  promotion  and  disease  prevention,  using 
evidence-based and good practices through measures and actions taken at the appropriate level in Member 
States, Sub-indicator: Cancer 

15 

 
 
 
 
 
 
which is a web-based mechanism used to collect exchange of best practices and analyse 
information on policy and stakeholder activities in mental health. 

Indicator:  number  of  Member  States  involved  in  health  promotion  and  disease 
prevention,  using  evidence-based  and  good  practices  through  measures  and  actions 
taken at the appropriate level in Member States 

Saturated fat: The indicator on the number of Member States  with an initiative on the 
reduction  of  saturated  fat  improved  over  the  implementation  period,  reaching  24 
Member States in 2020. 

European  accreditation  scheme  for  breast  cancer:  The  indicator  on  the  number  of 
Member  States  implementing  the  EU  scheme  for  breast  cancer  services  improved  in 
2014-2018, reaching 10 Member States by the end of that period, before declining to 6 in 
2020.  

3.1.2. Examples of actions implemented under Specific objective 1: promote 
health, prevent diseases, and foster supportive environments for healthy 
lifestyles: 
(i)  The online ‘best practice portal’46 was launched in April 2018. Since then, it has had 
more than 6 650 visitors from all EU Member States as well as neighbouring countries. 
Portal visitors can access good practices collected by previous health programme actions. 
Stakeholders can also submit a practice for evaluation. 

As of 2020, more than 12 best  practices  selected by the Member States in  the Steering 
Group  on  health  promotion,  disease  prevention  and  management  of  non-communicable 
diseases  (SGPP)  have  been  taken  up  in  75%  of  EU  Member  States.  Areas  covered 
include  integrated  care,  mental  health  and  the  fight  against  depression,  nutrition  and 
physical activity, the prevention of alcohol abuse and the prevention and management of 
chronic diseases. 

(ii) The Programme developed in 2016, the ‘EU Compass for Action on mental health 
and  wellbeing’,  a  web-based  tool  to  collect  and  exchange  best  practices  and  analyse 
information on policy and stakeholder activities in mental health. 

 (iii) On cancer, the European Quality Assurance scheme has been developed (over the 
period 2015 to  2019),  in  a harmonised, evidence-based and flexible way  to  grant  equal 
and quality-benchmarked treatment  to  patients.  The activities of the European Network 
of  Cancer  Registries  coordinated  by  the  Joint  Research  Centre  (JRC)  provides  a  ‘data-
brokering’  service  to  ensure  the  integrity  of  a  single  European  dataset  for  different 
purposes.  128  Cancer  Registries  from  29  European  countries  regularly  provide  data  to 
JRC with more than 25 900 000 records so far in the database. 

46 BP Portal (europa.eu) 

16 

 
 
 
 
 
3.2. 

Specific  objective  2:  protect  EU  citizens  from  serious  cross-border  health 
threats 

3.2.1. Indicator 
12% (EUR 52 million) of the overall budget47 was allocated to this objective. Over EUR 
14  million  was  allocated  to  projects  aiming  to  contribute,  to  the  extent  possible,  to 
Member  States’  preparedness  to  respond  to  possible  major  health  threats.  Projects 
included  the  organisation  of  simulations  and  other  exercises  on  generic  preparedness, 
capacity  building  -  such  a  quality  assurance  for  diagnostic  capacity,  and  specific 
activities  addressing  the  air  transport  and  shipping  sectors.  Other  actions  aimed  to 
support Member States in addressing the challenges of the migratory crisis of 2015-2016. 

Indicator: number of Member States integrating coherent approaches into the design of 
their preparedness plans (see paragraph 2.2.2) 

This indicator showed progress, increasing from 0 Member States in 2014 to 16 in 2015, 
and 22 in 2020 

Despite  the  positive  trend  of  this  indicator  and,  while  the  Programme  continuously 
contributed to improve the EU and Member States’ preparedness and response to cross-
border health threats of usual magnitude (including outbreaks such as Zika and Ebola that 
occurred  in  third  countries),  the  COVID-19  outbreak  in  the  first  quarter  of  2020 
uncovered weaknesses and fragilities in the Preparedness and response of the EU and of 
the Member States with regard to a major health threat such as the COVID-19 pandemic 
(see section 4.1.1, paragraph a). 

3.2.2. Examples of actions implemented under specific objective 2: protect EU 
citizens from serious cross-border health threats: 

(i) As response to the COVID-19 outbreak in the first quarter of 2020, running actions on 
health  security,  funded  by  the  Programme,  were  steered  to  contribute  to  combating  the 
pandemic, in particular in its early stages. Key examples are: the Joint Action Healthy 
gateways which supported coordination among EU Member States to improve capacity 
to  combat  cross-border  health  threats  at  points  of  entry,  including  ports,  airports  and 
ground  crossings,  and  the  Joint  Action  on  Strengthened  International  Health 
Regulations  and  preparedness  (SHARP)48,;    which  involved  a  collaboration  with 
ECDC’s EVD-LabNet (Emerging Viral Diseases-Expert Laboratory Network) to ensure 
quality  control  and  capacity  building  for  precise  diagnostics  at  an  early  phase  of  the 

47 Note that apart from actions funded under the health programme, other EU-level actions, mechanisms, 
permanent  structures  and  organisations.  also  address  health  threats,  e.g.,  the  early  warning  and  response 
system  (EWRS),  the  Health  Security  Committee,  the  Health  Emergency  Operation  Facility  and  the 
European  Centre  for  Prevention  and  Disease  Control.  This  may  partly  explain  the  relatively  smaller 
proportion of the Programme budget devoted to health threats. 
48 The Joint Action SHARP supports coordination among EU reference laboratories to prevent, detect and 
respond to biological outbreaks, chemical contamination and environmental and unknown threats to human 
health. 

17 

 
 
 
pandemic  and  demonstrated  the  importance  of  using  laboratory  networks  as  a 
preparedness and response tool. 

Under the 3rd Health Programme, actions have been reoriented to specific needs related 
to  the  coronavirus  threat,  such  as  training  for  health  professionals  (including  practical 
advice  on  isolation,  waiting  rooms  and  reception  areas,  cleaning,  and  appropriate 
personal protective equipment), and the provision of real-time RT-PCR tests to detect the 
virus, including shipment costs if needed.49 

These  emergency  actions  implemented  by  the  Programme  were  complementary  with 
other emergency actions carried out outside the health programme by DG SANTE (e.g. 
measures  taken  by  the  Health  Security  Committee,  the  Joint  Procurement  to  purchase 
medical countermeasures, the Early Warning and Response System – EWRS, the Health 
Emergency Operation Facility  -  HEOF) and with  emergency measures  taken by a wide 
range  of  other  Commission  services  as  well  as  the  ECDC,  which  were  mobilised  to 
combat the pandemic. 

(ii)  During  the  Ebola  and  Zika  outbreaks,  part  of  the  Programme  funding  was  used  to 
support measures to limit the spread of these  viruses by strengthening the preparedness 
and  response  of  individual  Member  States  working  together  under  the  Health  Security 
Committee  (entry  screening,  medical  evacuations,  preventing  transmission  in  transport 
and hospital settings). The budget in 2014-2016 for strengthening EU response to health 
threats  amounted  to  EUR  11  million.  The  budget  for  the  remaining  implementation 
period (2017-2020) amounted to EUR 41 million. 

3.3. 

Specific objective 3: support public health capacity building and contribute to 
innovative, efficient and sustainable health systems:  

3.3.1. Indicators 
24%  (EUR  109  million)  of  the  overall  budget  was  allocated  to  objective  3. 
EUR 30 million was spent on collaboration on health technology assessment (HTA) to 
develop  commonly  agreed  tools  and  procedures  and  carrying  out  joint  assessments  or 
early  dialogues  taking  a  ‘life  cycle’  approach  to  health  technologies.  Other  actions 
supported the exchange of experience and best practices in addressing the ageing of the 
population,  promoted  integrated  care  models  and  practices  (e.g.  the  Joint  Action  on 
implementation  of  digitally  enabled  integrated  person-centred  care  –  JADECARE  that 
was  launched  in  autumn  2020  and  will  run  for  3  years)  and  supported  the  EU  eHealth 
network  in  promoting  the  uptake  of  digital  solutions.  The  Joint  Action  on  health 
information  helped  to  streamline  and  harmonise  health  information  activities  across 

49 EU response to the COVID-19 pandemic was carried out by a wide range of other Commission services, 
EU  programmes  and  funding  instruments,  as  referred  to  in  the  following  non-exhaustive  list:  the  Union 
Civil  Protection  Mechanism;  the  Emergency  Support  Instrument;  the  EU  Solidarity  Fund;  the  health 
programmes49,  the  Joint  Procurement  for  Medical  Countermeasures;  the  European  Centre  for  Disease 
Prevention  and  Control  -  ECDC,  the  Horizon 2020  Framework  Programme  for  Research  and  Innovation 
(2014-2020), the Single Market Programme 

18 

 
 
 
Europe  by  developing  a  sustainable  and  solid  infrastructure  on  EU  health  information, 
and to increase consistency and sustainability. 

Indicator:  advice  produced  and  the  number  of  Member  States  using  the  tools  and 
mechanisms identified to contribute to effective results in their health systems 

Number  of  Member  States  using  the  tools  and  mechanisms  identified  (i.e.,  advice, 
joint HTA reports and recommendations produced by the EUnetHTA Joint Action 
3) to contribute to effective results in their health systems: Progress was made on this 
indicator, with the number of Member States increasing from 0 in 2014, to 5 in 2015, to 9 
in 2017 and finally to 23 in 2020. 

Advice produced, in particular the number of health technology assessments (HTA) 
produced  per  year:  This  indicator,  improved,  increasing  from  2  HTA  at  EU  level  in 
2012, to 22 in 2018 and finally to 41 in 2020 

3.3.2. Examples of actions under specific objective 3: support public health 
capacity building and contribute to innovative, efficient, and sustainable health 
systems 
(i)  State  of  Health  in  the  EU  -  (1)  2019  country  health  profiles  and  (2)  Health  at  a 
Glance 2020: Europe 

On 28 November 2019, the European Commission published 30 country health profiles 
as part of the State of Health in the EU cycle. The accompanying ‘companion report’ (a 
Commission staff working document) and factsheet flagged five key challenges faced by 
EU health systems: 

-  Tackling the decline in vaccination confidence across the EU 
-  Harnessing the digital transformation of health promotion & disease prevention 
-  Strengthening the evidence base on access to healthcare 
-  Shifting tasks and changing the skill mix to explore new ways of providing care 
-  Breaking down silos for safe, effective, and affordable medicines. 

Several  of  these  are  closely  linked  to  objective  3  of  the  Programme  and  to  the  key 
priorities of the 2019-2024 Commission, as set out in the mission letter to Commissioner 
Kyriakides. 

(ii) Digital innovation – paving the way to a European health data space 

Building  on  past  actions  in  the  field  of  eHealth  (e.g.  inclusion  of  eHealth  into  health 
policies,  better  alignment  of  eHealth  investments  into  health  needs,  as  part  of  the 
implementation of the Digital Single Market Strategy), the Commission aims to support 
EU Member States in making the most of the potential of digital health to provide high-
quality healthcare and reduce inequalities. Key to achieving this aim is the creation of a 
‘European Health Data Space’ (EHDS), to promote health data exchange and support 
research,  innovation  and  policymaking.  The  European  Health  Data  Space  will  lead  to 
better  health  outcomes  for  patients  and  the  public,  reduced  costs,  increased  efficiency, 
more resilient health systems, new treatments and better policymaking. The Programme 
funded  the  Joint  Action  TEHDAS  (The  European  Health  Data  Space)  in  2020  and 

19 

 
 
 
started  in  2021,  with  25  participating  countries,  to  help  set  up  a  European  health  data 
space,  by  developing  principles  for  the  cross-border  secondary  use  of  health  data.  This 
joint action shows continued relevance, with the adoption of the Commission proposal on 
EHDS50 on 3 May 2022 and its potential impact for healthcare, research, innovation and 
policymaking. 

3.4. 

Specific  objective  4:  facilitate  access  to  better  and  safer  healthcare  for  EU 
citizens  

3.4.1. Indicators 
19%  (EUR  87  million)  of  the  overall  budget  was  allocated  to  objective  4.  EUR  26 
million  was  invested  for  setting  up  and  coordinating  of  the  European  Reference 
Networks (ERNs)51. 

Actions  included  assessments  of  the  networks  and  their  healthcare  provider  members. 
Key  priorities  of  Programme  spending  for  this  objective  included  the  effort  to  jointly 
address  the  effects  and  challenges  of  increased  antimicrobial  resistance  and  healthcare 
acquired infections, support for Member States’ collaboration on blood, tissues and cells, 
and rare diseases. 

Indicators: number of ERNs set up in accordance with Directive 2011/24/EU; number of 
healthcare  providers  and  centres  of  expertise  joining  the  ERNs;  number  of  Member 
States using the tools developed. 

Number  of  ERNs  set  up  in  accordance  with  Directive  2011/24/EU:  the  number 
increased from 0 in 2014 to 23 in 2016, and finally to 24 in 2017. 

Number  of  healthcare  providers  and  centres  of  expertise  joining  the  ERNs:  the 
number increased from 0 in 2014 to 936 in 2016, and finally to 1 185 in 2020. 

Number  of  Member  States  using  the  tools  developed  (i.e.,  diagnostic  and  treatment 
tools,  Clinical  Patient  Management  System  -  CPMS):  The  number  increased  from  0  in 
2014 to 25 in 2017 and to 27 in 2020. 

3.4.2. Actions under specific objective 4: facilitate access to better and safer 
healthcare for EU citizens 
(i)  The  European  Reference  Networks  (ERNs)  are  virtual  networks  involving 
healthcare providers across Europe. They aim to tackle complex or rare medical diseases 
or conditions that require highly specialised treatment and a concentration of knowledge 
and resources. 
The first ERNs were set up in 2017. Over the five subsequent years, as the ERNs reach 
higher  capacity,  thousands  of  EU  patients  suffering  from  a  rare  or  complex  condition 

50 Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL on the 
European Health Data Space COM/2022/197 final 
51  This  amount  does  not  include  cost  of  IT  tools  for  the  Clinical  Patient  management  System  Clinical 
(CPMS) which are estimated at EUR 6 million. 

20 

 
 
 
 
have  benefited  from  better  diagnosis,  better  advice  on  specialised  treatments,  new  or 
better  clinical  practice  guidelines,  while  healthcare  professionals  involving  in  rare 
diseases were offered better training and support. 

(ii) Rare diseases: the Joint Research Centre has developed in 2018 and is maintaining 
the  European  Platform  on  Rare  Diseases  Registration  receiving  specific  financial 
support from the Programme. The Platform is promoting the interoperability of existing 
registries and has helped in the creation of new ones, including those developed by the 
ERNs.  The  migration  of  the  two  databases  -  the  European  Surveillance  of  Congenital 
Anomalies (EUROCAT) and the Surveillance of Cerebral Palsy in Europe (SCPE) – has 
been successfully implemented.  

In  the  area  of  rare  diseases  codification  general  rules  for  routine  coding  with  Orpha 
codes  have  been  established  and  guidelines  are  provided  to  achieve  internationally 
standardised data collection. 

3.5.  Examples  of  actions  addressing  horizontal/cross-cutting  issues  –  health 
-  SDGs, 

the  sustainable  development  goals 

inequalities, 
dissemination of programme results 

legislation, 

Finally, 8% (EUR 38 million) of the overall budget was devoted to horizontal and cross-
cutting issues (health inequalities, dissemination of Programme results, SDGs, and 
legislation)52. 

(i)  The  Joint  Action  on  health  equity  Europe  (JAHEE)  (2018-2021)  enables  Member 
States to work jointly to address health inequalities and achieve greater equity in health 
outcomes  across  all  groups  in  society,  in  all  participating  countries  and  in  Europe  at 
large. The general objective of this initiative is to improve  the health and well-being of 
people  across  the  EU  and  across  all  societal  groups.  It  also  has  a  specific  focus  on 
vulnerable groups and migrants.  

(ii)  The  programme  provided  resources  for 
implementing  the  EU’s  political 
commitments  and  legal  obligations  in  health  (e.g.  implementation  of  the  tobacco 
control53  cross  borderand  health  threats54  legislation,  the  EU  regulatory  framework  for 

52 The percentages devoted to the 4 specific objective and to horizontal/cross-cutting issues do not add up 
to  100%  since  a  small  (residual)  part  of  the  overall  Programme  budget  is  allocated  to  administrative 
support and to the cost of functioning of the executive agency CHAFEA. 
53  Directive  2014/40/EU  of  the  European  Parliament  and  of  the  Council  of  3 April  2014  on  the 
approximation of the laws, regulations and administrative provisions of the Member States concerning the 
manufacture,  presentation  and  sale  of  tobacco  and  related  products  and  repealing  Directive  2001/37/EC 
(OJ L 127, 29.4.2014, p. 1–38) 
54  Decision  No 1082/2013/EU  of  the  European  Parliament  and  of  the  Council  of  22 October  2013  on 
serious cross-border threats to health and repealing Decision No 2119/98/EC (OJ L 293, 5.11.2013, p. 1–
15), repealed by REGULATION (EU) 2022/2371 OF THE EUROPEAN PARLIAMENT AND OF THE 
COUNCIL of 23 November 2022 

21 

 
 
 
 
medicinal products55 and medical  devices56, for  substances of human origin57 and cross-
border  healthcare58),  including  through  the  development  of  common  tools  (e.g.  the 
European  Database  on  Medical  Devices  -  EUDAMED59  and  European  Medicine  Price 
Database  -  EURIPID60  database)  which  are  necessary  for  the  smooth  operation  of  the 
internal market in the medical devices and medicines sectors. 

More examples of actions implemented by the Programme are presented in Annex VIII 
to this document. 

3.6.  Current state of play:  

Given that the majority of actions funded by the Programme through grant agreements 
have a typical duration of 3 years, the actions launched in 2020 and 202161 are still 
ongoing and should come to an end only in 2023 and 2024 respectively.  

This  is  the case for  example  for the joint  actions  on implementing best practices  in  the 
field of nutrition (Best-REMAP); to strengthen health preparedness and the response to 
biological  and  chemical  terror  attacks  (TERROR);  on  implementing  digitally  enabled, 
integrated, person-centred care (JADECARE), along with 20 projects that were launched 
in 2019. 

Ongoing  actions  launched  in  2020  include:  the  second  Joint  Action  on  tobacco  control 
(JATC2)62  to  facilitate  the  exchange  of  good  practices  between  the  Member  States  in 
order  to  improve  the  implementation  of  the  Tobacco  Products  Directive63  and  the 
delegated acts relating to e-cigarettes; the Joint Action on exchange and implementation 
of best practices in the field of mental health (ImplMENTAL), with particular focus on 
suicide  prevention  and  the  reform  of  mental  health  services;  and  the  Joint  Action 

55 Regulation (EC) No 726/2004 of the European Parliament and of the Council of 31 March 2004 laying 
down Community procedures for the authorisation and supervision of medicinal products for human and 
veterinary use and establishing a European Medicines Agency (OJ L 136, 30.4.2004, p. 1–33) 
56 Regulation (EU) 2017/746 of the European Parliament and of the Council of 5 April 2017 on in  vitro 
diagnostic medical devices and repealing Directive 98/79/EC and Commission Decision 2010/227/EU (OJ 
L 117, 5.5.2017) 
57  Directive  2002/98/EC  of  the  European  Parliament  and  of  the  Council  of  27  January  2003  setting 
standards  of  quality  and  safety  for  the  collection,  testing,  processing,  storage  and  distribution  of  human 
blood and blood components and amending Directive 2001/83/EC (OJ L 33, 8.2.2003, p. 30–40)  
and  Directive  2004/23/EC  of  the  European  Parliament  and  of  the  Council  of  31  March  2004  on  setting 
standards of quality and safety for the donation, procurement, testing, processing, preservation, storage and 
distribution of human tissues and cells (OJ L 102, 7.4.2004, p. 48–58) 
58 Directive 2011/24/EU of the European Parliament and of the Council of 9 March 2011 on the application 
of patients’ rights in cross-border healthcare (OJ L 88, 4.4.2011, p. 45–65) 
59 EUDAMED database - EUDAMED (europa.eu) 
60 European medicine price database (EURIPID) | WHOCC PPRI (goeg.at) 
61 A number of actions under the Programme were launched in 2021, through a carry-over of their budget 
from 2020 to 2021. 
62 JAOTC2 is a 3-year joint action with EC co-funding of EUR 2.5 million 
63 DIRECTIVE 2014/40/EU OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 3 April 
2014  on  the  approximation  of  the  laws,  regulations  and  administrative  provisions  of  the  Member  States 
concerning the manufacture, presentation and sale of tobacco and related products and repealing Directive 
2001/37/EC (OJ L 127, 29.4.2014, p. 1–38) 

22 

 
 
 
‘increasing  the  capacity  of  National  Focal  Points  to  provide  guidance,  information  and 
assistance  to  national  applicants  to  the  health  programmes64  and  to  other  health-related 
EU programmes and funding instruments (NFP4Health)’65. 

Three  projects  to  boost  vaccination  uptake  were  also  funded  in  2020,  namely  RISE-
Vac66, ImmuHubs67 and ActToVAx4NAM68 (‘increased access to vaccination for newly 
arrived migrants’). 

Furthermore,  due  to  the  disruptions  and  delays  caused  by  the  COVID-19  outbreak  in 
2020, certain actions initiated in 2018 incurred additional delays and have not come to an 
end as originally planned in 2022.  

Because of their ongoing/unfinished status, the results of the above actions could not be 
fully taken into account in this Programme evaluation. 

4.  EVALUATION FINDINGS (ANALYTICAL PART) 

4.1.  To what extent was the intervention successful and why?  

4.1.1. Effectiveness 
The  Programme  has  been  effective  in  meeting  its  own  implementation  objectives,  by 
following  up  on  the  recommendations  from  previous  evaluations,  and  applying  the 
exceptional  utility  criteria  to  encourage  participation  of  low-GNI  countries.  It  has  also 
made efforts to  ensure the sustainability  of  the outcomes and results  of its actions over 
time. 

a)  to what extent have the Programme objectives been met 

a.1. ANALYSIS OF THE PROGRAMME INDICATORS 

 The effectiveness of funded actions has been measured using the indicators set under the 
Programme (see Section 3 and Annex VII69). 

Although  they  do  not  cover  all  the  implemented  actions,  progress  on  these  indicators 
shows the extent to which the Programme has met its objectives.  

64 ie. The ending 3rd Health Programme (2014-2020) and its successor, the EU4Health programme (2021-
2027). 
65 Health Programme DataBase - European Commission (europa.eu). 
66 Duration: 3 years, EU co-funding: EUR 951 120. 
67 Duration 3 years, EU co-funding: EUR 989 104. 
68 Duration: 3 years, EU co-funding: EUR 994 393. 
69 It was decided not to use the indicator ‘Number of Healthy Life Years at birth’ in the analysis conducted 
in this document, since this indicator is shown to be influenced by several other factors beyond the health 
programme and since it was found difficult and technically challenging to separate or isolate the influence 
of the Programme implementation on the indicator. 

23 

 
 
 
 
(i) The specific objective 1 aimed at promoting health, preventing diseases and fostering 
supportive environments for healthy lifestyles. The implemented actions should result in 
an increased use of evidence-based practices at national level. 
Associated  with  this  specific  objective,  the  indicator  on  the  number  of  Member  States 
having a national initiative on reduction of saturated fat showed a moderate increasing 
trend  over  the  implementation  period  2014-2020,  reaching  or  exceeding  the  annual 
targets  set  out.  The  Programme  contributed  to  increased  initiatives  put  in  place  by 
Member  States  on  reduction  of  saturated  fat,  which,  in  line  with  scientific  evidence 
promotes health by reducing in the long-term the occurrence of diseases. 
The  second  indicator  (‘number  of  Member  States  in  which  the  European  accreditation 
scheme for breast cancer services is implemented’) saw a decrease from 10 in 2018 to 6 
in  2020.  This  is  explained  by  the  fact  that  in  2019,  developers  of  guidelines  and/or 
national  authorities  of  (only)  six  Member  States  had  used,  implemented  or  adapted  in 
their  national  cancer  plans,  the  guidelines  or  methodology  developed  by  the  European 
Commission  initiative  on  breast  cancer,  coordinated  by  the  Joint  Research  Centre.  The 
reduction  in  the  number  of  national  implementations  of  the  European  accreditation 
scheme  for  breast  cancer  was  due  to  the  choice  made  by  certain  Member  States  to 
implement  similar  other  schemes,  taking  into  account  country-specific  factors  and 
building  on  national  know-how  and  practice.  This  reduction  in  the  number  of  national 
implementations does not significantly affect the conclusion on the overall ability of the 
Programme  to  meet  the  objectives  set  out  under  the  Specific  Objective  1.  Qualitative 
analysis  demonstrates  that  other  cancer-related  actions  funded  by  the  Programme  were 
taken  up  by  Member  States  in  their  health  policies.  This  was,  notably  the  case  for  the 
accreditation schemes for cervical and for colorectal cancers.  

Beyond  cancer  prevention  and  treatment,  the  stakeholder  consultations  showed  that 
funded actions relating to health promotion and disease prevention actions were overall 
effective  in  reaching  their  objectives,  particularly  in  the  field  of  nutrition,  alcohol70, 
tobacco  control,  and  more  generally  through  further  rationalised  and  more  systematic 
organisation of the exchange of best practices, as enabled by the dedicated portal set-up 
under the Programme in 2018. 

 (ii)  The  specific  objective  2  (protection  of  citizens  from  serious  cross-border  health 
threats)  evolves  around:  risk  assessment,  capacity  building  (notably  through  the 
establishment,  regular  assessment  and  strengthening  of  national  preparedness  plans) 
against  health  threats  in  Member  States  and  where  appropriate  cooperation  with  third 
countries, implementation of EU legislation on communicable diseases and other health 
threats,  strengthening  of  health  information  and  knowledge  system  for  evidence-based 
decision-making.  The  natural  stakeholders  for  actions  undertaken  by  the  Programme 
under the specific objective 2 are Member States health authorities at national and sub-

70 see section below on stakeholders consultations 

24 

 
 
 
 
national  levels.  The  implementation  of  actions  under  the  specific  objective  2  should 
notably  enable  the  integration  of  coherent  approaches1  in  Member  States’  preparedness 
plans,  as  reflected  by  the  indicator  associated  with  this  objective  in  the  regulation 
establishing the Programme.  
The indicator showed a positive trend, starting from 0 in 2014, nearly reaching the annual 
targets or exceeding them in all years except in 2017 and 2020.  
In 2020, 22 Member States integrated coherent approaches (as described in sections 2.1. 
and  2.2.2.)  in  their  preparedness  plans,  as  shown  by  the  assessment  of  the  reports  on 
national  preparedness  plans  transmitted  to  the  Commission.  The  coherent  approaches 
included, for instance, the implementation of International Health Regulations (IHR), the 
interoperability between the health sector and other sectors, business continuity plans to 
cope with outbreaks, setting up and testing standard operating procedures (SOP) for the 
coordination between the health sector and a number of other sectors, notification to the 
Commission of substantial revisions of national  preparedness planning,  testing, training 
and exercises to ensure that IHR core capacities are maintained and strengthened in the 
future. 

Although  actions  on  preparedness  and  response  to  cross-border  health  threats  are  also 
under  the  responsibility  of  other  services,  bodies  and  mechanisms71  outside  the  health 
programme, actions funded by the Programme contributed to strengthen the preparedness 
and  response  of  the  EU  and  of  Member  States  to  cross-border  health  threats  (see 
paragraph  3.2.2  Annex  VIII,  paragraph  2.).  The  funded  actions  contributed  to  improve 
the preparedness  plans at EU and Member States  levels  for  cross-border health threats, 
thereby enabling on the one hand to effectively respond to the frequent, moderate health 
threats that occurred during the Programme implementation period, and on the other hand 
to  quickly  gear  the  Programme-funded  actions  towards  combating  the  COVID-19 
outbreak from the first quarter of 2020 (see paragraph 3.2.2. (i))  

As  has  been  the  case  in  other  geographical  areas,  the  COVID-19  pandemic  revealed, 
however, weaknesses  and fragilities in the preparedness  and response of the EU and of 
Member States to health threats of big magnitude. As mentioned in the Communication 
on short-term EU health preparedness for COVID-19 outbreaks, the main weaknesses at 
the beginning of the pandemic were: 

- 

- 

 the shortage of tests and testing materials; insufficient contact tracing, public 
health  surveillance  and  rapid  response  to  avoid  further  spread  of  the  virus; 
insufficient  trained  laboratory  personnel  and  some  supplies  of  laboratory 
equipment; 
insufficient  availability  of  medical  countermeasures  (personal  protective 
equipment,  medicines  and  medical  devices)  enabling  to  cope  with  increased 
demand; 

71  The  Health  Security  Committee,  the  Early  Warning  and  Response  System  –  EWRS,  the  Joint 
Procurement  to  purchase  medical  countermeasures,  the  epidemiologic  surveillance  and  rapid  response 
activities carried out by the European Centre for Disease prevention and Control (ECDC) 

25 

 
 
 
-  healthcare  surge  capacity  (shortages  of  intensive  care  units,  insufficient 

availability of healthcare staff). 

According  to  the  Communication  on  ‘Drawing  the  early  lessons  from  the  COVID-19 
pandemic’,  preparedness  and  planning  have  been  exposed  as  being  under-funded  and 
under-developed and much of the EU response had to be ad hoc and temporary, implying 
that preparedness and response systems and cultures have to be strengthened. 

The  above  analyses  lead  to  the  conclusion  that,  despite  the  increase  of  the  number  of 
Member States integrating coherent approaches in the design of their preparedness plans, 
the  specific  objective  2  (protecting  EU  citizens  against  cross-border  health  threats)  has 
been met only partially by the Programme. 

(iii)  Under  the  specific  objective  3,  actions  implemented  by  the  Programme  enabled  to 
identify  and  develop  tools  and  mechanisms  at  EU  level  to  address  shortages  of  human 
and  financial  resources,  and  to  facilitate  the  voluntary  uptake  of  innovation  in  public 
health  intervention  and  prevention  strategies.  The  main  identified  areas  of  action  were: 
Health  Technology  Assessment  (HTA),  Digital  health,  Health  Information  and 
Knowledge system (the State of Health and the Health at a glance publications cycles), 
the  setting-up  and  operation  of  the  Health  Policy  Platform,  implementation  of  Union 
legislation  in  the  field  of  medical  devices,  medicinal  products  and  cross-border 
healthcare.  The  results  of  these  actions  should  be  reflected  in  an  increase  of  advice 
produced and of the use by Member States (in the reform of their health systems) of the 
tools and mechanisms developed by these actions. 

The two indicators associated (advice produced and the number of Member States using 
the  tools  and  mechanisms  identified  in  order  to  contribute  to  effective  results  in  their 
health  systems  and  the  Number  of  Member  States  using  the  tools  and  mechanisms 
identified  in  order  to  contribute  to  effective  results  in  their  health  systems)  showed 
positive  trends  falling  short  to  reach  their  annual  target  (except  in  2020)  for  the  first 
indicator and exceeding the annual targets for several years for the second indicator. 

The  first  indicator  (e.g.,  ‘number  of  health  technology  assessments’)  showed  major 
progress in 2020, evolving from 29 in 2019 to 50 in 2020. 

The  above  analysis  of  indicators,  combined  on  the  one  hand  with  an  assessment  of  a 
sample  of  actions  funded  under  the  specific  objective  3  (e.g.  the  EUnetHTA  series  of 
joint  actions,  TEHDAS  joint  action,  the  Commission  proposal  on  the  European  Health 
Data Space – EHDS, the State of Health in the EU publication cycle, the Health Policy 
Platform), and on the other hand with the results of stakeholders surveys enable to infer 
that the specific objective 3 has been met to a good extent by the Programme. 

(iv) Concrete actions under the specific objective 4 (facilitate access to  better and safer 
healthcare)  aim  at  increasing  access  to  medical  expertise  and  information  for  specific 
conditions  (such  as  rare  diseases),  also  beyond  national  border.  They  also  facilitate  the 

26 

 
 
 
application  of  research  results  and  develop  tools  for  the  improvement  of  healthcare 
quality and patient safety, through, inter alia actions to improve health literacy. Effective 
implementation of these actions should notably be reflected in the creation of European 
Reference  Networks  for  rare  and  complex  diseases  (ERNs)  and  in  an  increase  of 
healthcare providers and centres of expertise joining the ERNs.  

Three  indicators  were  defined  to  measure  the  attainment  by  the  Programme  of  the 
specific  objective  4:  the  number  of  European  reference  networks  established  in 
accordance with  Directive 2011/24/EU,  the number of healthcare providers and  centres 
of  expertise  joining  European  reference  networks,  and  the  number  of  Member  States 
using the tools developed. All the indicators exhibited overall increasing trends. 23 ERNs 
were  established  in  2016,  and  1  more  established  in  2017.  This  figure  continued  to 
increase and fell short of the set out target of 30 in 2022.  

As for the second indicator, (the number of healthcare providers and centres of expertise 
joining ERNs, it fell between 2018 and 2019, due to a cleaning of duplicate records; the 
number has increased again since then and reached 1185 in 2020. 

Finally,  the  third  indicator  showed  a  steady  increasing  trend  over  the  implementation 
period.  

In  addition,  by  the  end  of  the  Programme  (2020),  23  Member  States  had  deployed  or 
were  using  patient  summaries  data/e-prescription,  in  line  with  the  EU  guidelines  (the 
target was 18 Member States) and 24 ERNs had been established. 

The above quantitative analysis indicates that the specific objective 4 has been met by the 
Programme. This indication is confirmed by stakeholders views and by in-depth analysis 
of a sample of funded actions under this specific objective.  

(v)  Attainment  of  targets  set  out  for  the  indicators  at  the  beginning  of  Programme 
implementation 

Despite  displaying  significantly  increasing  trends  (except  the  indicator  on  cancer)  the 
analyses above show that certain indicators did not reach their intermediate targets and/or 
their final targets set out at the end of the Programme implementation period, in 202072.  
The targets were not revised to take sufficiently into account the dynamic and complex 
implementation as well as unforeseen obstacles encountered. 

72 For instance, this is the case for the indicator for: the specific objective 1 (‘saturated fat’) which was 24 
versus a target of 28 Member States set out in 2020, the indicator for the specific objective 2 (‘number of 
Member States integrating coherent approaches in the design of their preparedness plans’) which reached 
the  value  of 22, versus an initial target of 28 in 2020, the indicator for the specific objective 3 (‘Advice 
produced, in particular the number of Health Technology Assessments  – HTA produced per year’) which 
reach 41 versus a target value of 50 in 2020, and the indicators for the specific objective 4 the ‘number of 
European  reference  networks  established  in  accordance  with  Directive  2011/24/EU’  (which  reached  24 
versus  an  initial  target  of  30  ERNs  in  2020)  and  the  ‘number  of  healthcare  providers  and  centres  of 
expertise  joining  European  reference  networks’  (which  reached  1180  versus  a  target  value  of  1450  in 
2020).  

27 

 
 
 
a.2.  RESULTS  OF  STAKEHOLDER  SURVEYS  AND  QUALITATIVE  ANALYSIS  BY  TYPE  OF  FUNDING 
INSTRUMENT  

The available data from stakeholder surveys and qualitative analysis show that the most 
effective  funded  actions  were  joint  actions  and  projects  and,  to  a  lesser  extent,  direct 
grants to international organisation and service contracts. 

Over  the  implementation  period,  a  number  of  health  topics  have  been  considered  as 
particularly  important,  including  chronic  diseases,  lifestyle  risk  factors,  HTA,  rare 
diseases and vaccination.  

The  effectiveness  of  joint  actions  and  projects  is  supported  by  targeted  stakeholder 
surveys,  in  which  the  respondents  stated  that  two  main  actions  which  contributed  the 
most  to  achieving  Programme  objectives  were  joint  actions  (79%  of  responses)  and 
projects  (61%  of  responses).  Conversely,  only  43%  of  respondents  considered  that 
operating grants contributed to the achievement of the Programme’s objectives, with the 
corresponding  figure  being  even  lower  for  the  Presidency  conferences  (23%  of 
respondents). 

According to stakeholders, examples of successful actions were the European Reference 
Networks  for  rare  diseases,  actions  to  promote  vaccination,  to  address  antimicrobial 
resistance (AMR), nutrition and alcohol consumption. 

With regard to alcohol consumption, findings from the dedicated case study  on alcohol 
included  in  the  external  study  supporting  the  evaluation  shows  that  the  Programme 
contributed to  addressing the objectives  and priorities in  the area of alcohol  marketing. 
More specifically, it has addressed the sub-theme of reducing  alcohol-related harm and 
alcohol  marketing  by  supporting  the  Joint  Action  on  reducing  alcohol-related  harm 
(RAHRA) under the second health programme (2HP). The RAHRA Joint Action aimed 
to support Member States in their work on common priorities, in line with the 2006 EU 
alcohol  strategy,  and  to  strengthen  their  capacity  to  reduce  and  address  alcohol-related 
harm. RAHRA contributed to capacity building and strengthened the ability to deliver a 
survey methodology and monitoring instrument for alcohol-related-harm. 

Although  no  joint  action  on  alcohol  consumption  was  funded  under  the  Programme, 
outputs from the RAHRA Joint Action supported by 2HP were further developed using 
other funding mechanisms. These comprised the DEEP SEAS73 service contract and the 
Presidency conference on alcohol marketing. 

73 DEEP SEAS., 2014. About DEEP SEAS. Available at: About DEEP SEAS | Deep Seas (deep-seas.eu) 

28 

 
 
 
 
 
a.3. FACTORS HINDERING THE ACHIEVEMENT OF THE PROGRAMME OBJECTIVES  

The evaluation findings (desk research, corroborated by stakeholder views) identified the 
following  factors  at  national  level  that  may  have  hindered  the  achievement  of  the 
Programme objectives: insufficient resources, expertise and data, insufficient knowledge 
of  population  health  needs,  difficulties  engaging  with  stakeholders,  and  lack  of  (or 
insufficient) political will.  

EU  action  can  address  these  factors  through  strengthened/enhanced  cooperation  and 
engagement to support the Member States. 

Increased  resources  at  EU  level  dedicated  to  health  issues  (such  as  under  the  the 
EU4Health  programme)  would  help  address  national  difficulties  in  participating  in  the 
health programmes. Furthermore, an even stronger role of the Commission in brokering 
the  existing  knowledge  and  pooling  the  existing  data  and  resources  being  generated 
would  contribute  to  closing  the  knowledge  gaps  where  needed,  while  also  steering 
national action. 

a.4. CONCLUSION ON THE ACHIEVEMENT BY THE PROGRAMME OF ITS OBJECTIVES 

Taking  into  account  the  above  quantitative  analyses  and  the  results  of  the  stakeholders 
surveys,  it  can  be  concluded  that  the  specific  objective  1  (promote  health  and  prevent 
diseases) was met to a good extent. The decrease of the number of Member States using 
the  European  accreditation  scheme  for  breast  cancer  does  not  significantly  affect  this 
conclusion since other European guidelines (e.g., on colorectal and cervical cancers) had 
been developed and implemented by Member States. The qualitative analysis of funded 
actions under this specific objective (e.g., on alcohol, tobacco control, physical activity, 
exchange of best practices) demonstrate the effectiveness of the Programme in achieving 
its objectives. 

With  regard  to  the  specific  objective  2  (protect  EU  citizens  against  cross-border  health 
threats),  22  Member  States  integrated  coherent  approaches  in  their  Preparedness  plans, 
which  were  reviewed  by  the  Commission.  The  associated  indicator  exhibited  an 
increasing trend over the implementation period. Through the implementation of actions 
aimed  at  reinforcing  the  laboratory  capacities  to  respond  to  highly  dangerous  and 
emerging pathogens, and through regular tests and simulations exercises, the Programme 
contributed to protect the EU citizens against moderate health threats (e.g. ZIKA, Ebola 
outbreaks).  

As  response  to  the  COVID-19  outbreak,  the  Programme  steered  its  running  actions  to 
fight  the  pandemic,  notably  by  switching  these  actions  from  an  inter-epidemic  to  an 
emergency mode. Despite this response, the COVID-19 pandemic revealed fragilities and 
weaknesses in the preparedness plans at Member State and EU level.  

29 

 
 
 
It  can  therefore  be  inferred  that  the  Programme  only  partially  met  its  objectives  on 
protecting EU citizens against serious cross-border health threats. 

Under the specific objective 3 (support public health capacity building and contribute to 
innovative, efficient and sustainable health systems) the Programme met its objectives to 
a good extent. This is demonstrated by the increasing trends of the indicators associated 
with  health  technology  assessment  (HTA),  further  corroborated  by  in-depth  analysis  of 
the outcome of a sample of actions (such as the EUnetHTA Join Action 3; The proposal 
on the European Health Data Space as a result of the joint action on this issue) and by the 
results of the stakeholders surveys. 

All indicators associated with the specific objective 4 (facilitate access to better and safer 
healthcare),  and  mainly  relating  to  the  ERNs  showed  a  positive  trend  falling  short  or 
attaining  the  targets  set  out  for  2020.  The  quantitative  analysis  is  corroborated  by 
stakeholders views who rated the ERNs as one of the most important achievements of the 
Programme.  Other  actions  under  this  specific  objective,  such  as  the  ones  addressing 
AMR were positively assessed by stakeholders. 
It can therefore be concluded that, under the specific objective 4, the Programme met its 
objectives to a good extent. 

b)  contribution to improvements in health and healthcare at both EU and Member State 

level  

In addition to progress on the various indicators, the effectiveness of the Programme is 
demonstrated by the fact that knowledge produced by funded actions was used in 
policymaking, enabling the Programme to contribute to improvements in health and 
healthcare in the EU and at Member State level.  

This is particularly the case for actions in the area of cancer, HTA, and blood, tissue and 
cells which influenced national practices and helped create and strengthen national 
legislation. 

For instance, the iPAAC (Innovative Partnership for Action Against Cancer) Joint Action 
was  very  effective  in  terms  of  providing  ready-made  solutions  that  have  been 
implemented in the Polish National Oncology Strategy. 

The Programme  also  contributed to  improving health and healthcare in  the EU through 
the  transfer  and  implementation  of  best  practices.  For  example,  the  CHRODIS+  Joint 
Action  enabled  the  transfer  of  good  practices  related  to  nutrition  in  schools  between 
countries.  Furthermore,  following  the  successful  pilot  of  an  integrated  multi-morbidity 
care model as part of CHRODIS+, Member States’ health authorities decided to extend 
this project to other healthcare institutions, using resources from the European structural 
funds. 

30 

 
 
 
Additional  examples  include  a  Joint  Action  (RD-ACTION)  that  developed  a  toolset  to 
help European countries implement the Orphanet nomenclature of rare diseases (ORPHA 
codes,  standardised  coding  system);  and  a  Joint  Action  that  helped  extend  successful 
national initiatives on physical activity in primary schools (Active Schools Flag) to other 
Member States. 

c)  effectiveness  of  the  exceptional  utility  criteria  in  supporting  and  increasing 

participation of low-GNI countries in the Programme 

The exceptional utility criteria provided for a higher level of co-funding for actions that 
include a certain proportion of members from low-GNI countries. They applied to joint 
actions, project grants and operating grants. These criteria were introduced in the second 
Health Programme 2008-2013 (2HP) but have been further refined under the Programme.  

To qualify for the exceptional utility criteria, a country needed a GNI of less than 90% of 
the EU average. A total  of 16 countries met these requirements (BG, CY, CZ, EE, EL, 
ES, HR, HU, LT, LV, MT, PL, PT, RO, SI and SK).  

The  exceptional  utility  criteria  were  introduced  at  the  beginning  of  the  Programme  in 
2014, as they were included in the basic act, namely in Article 7(3) of Regulation (EU) 
No. 282/2014.  

The  exceptional  utility  criteria  were  used  relatively  often;  between  2  and  212  funded 
actions met these criteria per year. In the targeted stakeholder survey organised as part of 
this  evaluation,  respondents  often  did  not  know  whether  their  country  had  used  this 
mechanism.  Among  the  interviewed  stakeholders  who  applied  for  funding  using  the 
exceptional  utility  criteria,  a  majority  stated  that  their  country’s  participation  has  been 
incentivised to a small or moderate extent.  

Data from the public project database of HaDEA indicates that low-GNI countries were 
less  likely  on  average  to  participate  in  funded  actions  as  partners  or  coordinators  than 
high-GNI countries.  

A  number  of  factors  influenced  the  decision  not  to  apply  for  funding  under  the 
exceptional utility criterion. These included a lack of administrative capacity to manage 
actions  in  the  Member  State,  the  administrative  burden  (once  the  project  is  up  and 
running) and the complexity of the application processes. 

Programme  participation  by  low-GNI  countries  did  not  increase  over  time.  Low-GNI 
countries did not coordinate more than 11 funded actions in one year, and in 2014 did not 
coordinate any actions at all. In contrast, the high-GNI countries coordinated between 11 
and  66  actions  per  year.  Some  eligible  countries  (e.g.,  Czechia,  Latvia,  Lithuania  and 
Poland) did not coordinate any funded action.  

Figure 2- Number of funded actions coordinated per year 

31 

 
 
Source: HaDEA public project database of funded actions 

Finally, Programme participation by low-GNI countries has not increased as compared to 
the 2HP according to data analysis conducted as part of the evaluation.  

The reasons why the criteria did not greatly increase participation are not clear but may 
include administrative issues and costs. To improve participation of low-GNI countries, 
certain  stakeholders  suggested  to  raise  the  co-funding  percentages  from  60-80%  to  70-
90%. 

d)  publication  of  Programme’s  actions,  outcomes  and  results  by  the  Commission 
services  and  by  beneficiaries  and  stakeholders,  and  accessibility  to  the  wider 
scientific and health community and to the public.  

Beneficiaries  and  stakeholders  rated  the  Programme  results  as  ‘fair’  but  saw  space  for 
improvement in the dissemination of the publications resulting from it to a wider group 
of stakeholders and to the public.  

The  analysis  of  the  HaDEA  public  project  database  identified  4 866  outputs  related  to 
277  of  the  339  funded  actions  under  the  Programme.  In  HaDEA’s  public  database, 
outputs  were  classified  as  ‘layman’,  ‘newsletters’  and  ‘others’.  The  most  prevalent 
category was ‘others’, making up 79% of all outputs, compared with only 2% of outputs 
being  ‘newsletters’  (2%)  and  ‘layman’  (2%).  Some  publications  resulting  from  the 
Programme actions have been published in scientific journals.  

The findings from the stakeholder surveys point to difficulties faced by the Programme’s 
beneficiaries  in  developing  and  implementing  dissemination  activities  for  the  funded 
actions.  Furthermore,  there  is  no  systematic  method  in  place  to  monitor  the  extent  to 
which Programme beneficiaries disseminate findings after a project has ended.  

The evaluation therefore concludes that improvements to the dissemination of results are 
needed. These could be attained through supportive actions by the Commission, such as 

32 

 
 
 
 
 
organising knowledge transfer activities (e.g. communities of practice, policy dialogues 
and other events). Lastly, although Programme results have been used by stakeholders, 
this could be further strengthened if limitations to dissemination are addressed. 

For  instance,  the  support  provided  by  CHAFEA  (and  continued  by  HaDEA74)  in 
promoting  the  tool  developed  under  the  SCIROCCO  funded  action  was  considered  a 
concrete example of facilitating dissemination and thus the sustainability of Programme 
results.  However,  while  the  Commission  could  provide  further  support  on  the 
the 
dissemination  of  results,  as  suggested  above, 
dissemination  of  Programme  results  cannot  be  considered  a  shortcoming  of  the 
Programme  alone,  but  rather  a  shared  responsibility  with  Programme  beneficiaries,  in 
particular Member States competent authorities involved in funded actions. 

the  observed 

limitations 

to 

The  case  study  on  the  third  EUnetHTA  Joint  Action  (JA3)  showed  that  the  production 
and  use  of  pharmaceutical  assessments  (both  joint  assessments  and  collaborative 
assessments)  increased  under  JA3  as  compared  to  the  second  EUnetHTA  Joint  Action 
(JA2) funded under the second health programme. When considering other technologies, 
there  has  been  increased  production  of  joint  assessments  and  collaborative  assessments 
but a slightly decreased use, which can be partly explained by limited national capacity 
and increased outputs under Joint Action 3, and by the fact that other HTA processes are 
not fully established in some countries. For both pharmaceuticals and other technologies 
there  is  an  increased  number  of  countries  that  have  used  joint  assessments  and 
collaborative  assessments  (JA/CA)75  under  JA3  compared  to  JA276.  Other  EU  funded 
actions  that  produced  results  used  by  stakeholders  include  the  RARHA  Joint  Action77 
and the Oramma project78. 

According  to  stakeholder  surveys,  disseminating  results  has  helped  raised  awareness 
among patients and healthcare providers in the field of digital health, tackling scepticism 
and  helping  realise  a  European  health  data  space  (EHDS).  Likewise,  the  scientific 
publications  resulting  from  funded  actions  helped  prove  to  ministries  of  health  that  the 
interventions were effective. 

74  CHAFEA  was  closed  with  effect  on  31/3/2021  by  the  Commission  Decision  (EU)  2021/173  of 
12/2/2021 (OJ L 50, 15.2.2021, p. 9–28). Its legacy activities relating the Health Programme (2014-2021) 
were  transferred  to  the  newly  created  Health  and  Digital  Executive  Agency  –  HaDEA,  which  was  also 
entrusted with the implementation of the successor EU health programme (EU4Health Programme) on the 
period 2021-2027. 
75 ie assessments carried out jointly by multiple HTA agencies 
76 EUnetHTA Work Package 7, Deliverable 7.2 – Final report. Available at: https://www.eunethta.eu/wp-

content/uploads/2020/07/Final-Deliverable-7.2-report-after-consultation_FINAL.pdf?x69613  

77  The  Joint  Action  on  Reducing  Alcohol  Related  reducing  alcohol  related  harm  [RARHA]  aimed  to 
support  Member  State  cooperatecooperation  on  the  uptake,  exchange  and  development  of  common 
approaches relating to the priorities of the EU alcohol strategy. 

78 The ORAMMA project aimed to promote safe pregnancy and childbirth through efficient provision of, 

access to, and use of quality skilled care for all migrant and refugee women and their infants. 

33 

 
 
 
Despite  those  successes  and  considering  the  limitations  to  dissemination  discussed 
above, it can be concluded that there is room for improvement in the use of Programme 
results if these minor limitations are addressed. 

e)  to  what  extent  have 

the  recommendations 

from  previous  evaluations  been 

implemented? 

Commission’s  Directorate  General  SANTE  and  CHAFEA  have  taken  steps  to  address 
the 10 recommendations included in the Programme’s midterm evaluation. 

The  evaluation  shows  that  some  of  the  recommendations  set  out  in  the  midterm 
evaluation  have  been  addressed  successfully.  These  include  maintaining  a  focus  on 
thematic areas  of strong EU added value, strengthening and building  links between the 
Programme  and  the  wider  Commission  and  EU  policy  agenda  to  maximise  impact, 
developing  a  broader  strategy  to  increase  participation  from  poorer  Member  States  and 
underrepresented organisations, and improving dissemination of action results.  

Conversely,  some  recommendations  were  not  sufficiently  taken  up,  including  spelling 
out how actions targeting health promotion and health systems should generate EU added 
value  and  investing  in  the  resources  necessary  to  improve  systems  for  monitoring 
Programme  implementation.  The  latter  recommendations,  alongside  those  which  were 
only  partially  met,  should  be  followed  up  under  subsequent  EU  health  programmes. 
Similarly, despite progress in the dissemination of results and in the participation of low-
GNI countries, there are still limitations that affect full uptake. 

f)  how  are  the  results  and  effects  of  the  Programme  likely  to  last  at  the  end  of  its 

implementation if funding ceases to exist (self-sustainability)? 

According to the evaluation analyses the Programme was found sustainable overall. 

This  is  the  case  for  achievements  in  the  field  of  HTA.  The  work  developed  under  the 
EUnetHTA  joint  actions  strengthened  the  collaboration  of  national  HTA  agencies, 
promoting  coordination  and  increasing  production  of  HTA  joint  work.  EUnetHTA’s 
activities laid a strong foundation for sustainable cooperation, which has been reflected in 
the  permanent  framework  for  joint  work  established  by  the  HTA  Regulation79.  The 
Regulation  replaces  the  current  system  based  on  a  voluntary  network  of  national 
authorities  and  project-based  cooperation.  (See  the  case  study  on  HTA  for  more 
information). 

Another area where funded actions were deemed sustainable  is antimicrobial  resistance 
(AMR).  In the Joint Action on AMR  (EU-JAMRAI) which  ran from 2017 to  2021,  , a 
work  package  focused  on  sustainability,  thereby  enabling  the  exploitation  and  further 

79  Regulation  (EU)  2021/2282  of  the  European  Parliament  and  of  the  Council  of  15 December  2021  on 
health technology assessment and amending Directive 2011/24/EU (OJ L 458, 22.12.2021, p. 1–32) 
See  also  Judit  Erdös  et  al.  (2019),  ‘European  Collaboration  in  Health  Technology  Assessment  (HTA): 
goals, methods and outcomes with specific focus on medical devices”,’, Wien Med Wochenschr. 

34 

 
 
 
development of the joint action’s results beyond the end of the project. The joint action 
identified  two  main  ways  to  ensure  sustainability:  (i)  ensure  direct  follow-up  and 
cooperation  between  Member  States  (ii)  continue  action  at  EU  level,  and  if  necessary, 
making use of EU funding. 

The ERNs established through the Programme also had sustainable impacts. One of the 
tools  the  Commission  developed  for  the  ERNs  was  the  Clinical  Patient  Management 
System (CPMS), which allowed for cross-border virtual consultations. Another tool was 
the five first grants to support registry development for five ERNs started in 2018 and the 
19 following started in 2020. They collect data at EU level for patients with rare diseases. 
Thanks to the progressive development  and integration into the EU RD Platform,  these 
registries  linked  to  the  24  ERNs  should  progressively  become  more  and  more 
interoperable  (using  at  minima  the  ‘common  data  elements’  provided  by  the  EU  RD 
platform) and also visible.80. 

Finally,  the  SCIROCCO81  funded  action  has  created  sustainable  outputs,  as  the  tool  is 
used  in  35  countries  by  hundreds  of  thousands  of  users,  and  the  users  have  reportedly 
found it very useful. 

The  evaluation  found  that  there  were  common  elements  and  aspects  of  the  Programme 
itself which helped ensure that projects would be sustainable following their conclusion. 
For  example,  introducing  an  obligatory  work  package  (WP)82  exclusively  on  the 
sustainability  of  the  funded  action,  and  the  relationships  and  connections  built  through 
the funded actions.  

However, there were shortcomings in integrating the results of funded actions into policy 
making. Another challenge was related to the design of the Programme, with actions not 
lending  themselves  to  increasing  sustainability,  either  due  to  the  limited  duration  of 
funded actions, or to a slight mismatch between their results and the ability to implement 
them  directly  at  local  level.  Projects  were  found  by  certain  surveyed  stakeholders,  too 
limited  in  scale  and/or  in  ambition  to  be  sustainable.  These  stakeholders  gave 
recommendations  on  how  to  make  the  Programme  or  similar  programmes  more 
sustainable.  These  included  the  creation  of  an  EU-level  repository  of  outputs  and 
outcomes  of  the  funded  actions,  and  of  more  opportunities  or  funding  to  continue  and 
further develop existing projects or to disseminate their results to more Member States. 

According  to  the  various  consultations,  the  opportunity  to  continue  to  fund  critical 
actions  that have  proven  successful and  demonstrated strong EU added value would be 
welcome. For this, it would be useful to combine Health programme funding with other 

80 Academic and research organisation, in the focus group on project grants. 
81 SCIROCCO – Scaling Integrated Care in Context. The SCIROCCO project validated and tested a self-
assessment tool to identify the maturity of the health and social care systems for  the adoption and scaling 
up of integrated care solutions. 
82   This work package has been made mandatory for all grants 

35 

 
 
 
EU  financial  instruments  (e.g.  the  cohesion  policy  funds,  EU  Framework  Programmes 
for research and innovation, national funding programmes) that address health. 

The  sustainability  of  the  actions  funded  under  the  Programme  also  depends  on  efforts 
made  by  the  participating  countries  themselves  to  learn  from  and  take  up  the  results. 
Introducing  strategies  and  approaches  derived  from  the  Programme  into  their  national 
cancer plans is just one example. 

A  major  challenge  to  sustainability  is  the  limitation  to  Member  States’  or  other 
beneficiaries’  ability  to  take  over  the  funding  of  completed  projects.  Lack  of  resources 
and  reduced  political  will  or  interest  to  continue  specific  activities  at  national  level 
constitute additional barriers to sustainability. 

In  conclusion,  the  results  of  the  Programme  were  found  to  be  sustainable  overall. 
Challenges  and  barriers  to  sustainability  mainly  stem  from  insufficient  integration  of 
results into national policies, a lack of further resources and a lack of political will in the 
participating countries or by beneficiaries, and insufficient links with other programmes. 

4.1.2 Efficiency 

a)  to what extent has the Programme been cost effective? 
The evaluation found that the Programme achieved its objectives in a cost-effective way 
and  within  the  allocated  budget.  According  to  stakeholders,  the  Programme  produced 
high quality (and quantity of) outputs and work within the budget provided.  

The flexibility of the management of the budget and the adaptability of the Programme to 
changing circumstances was an important success factor in achieving those impacts in a 
cost-effective way. 

Moreover (as shown in Table 2, Section 3), the budget allocated to administrative support 
and to the functioning of the executive agency CHAFEA, amounted to EUR 10.5 million 
and  EUR  30.3  million  respectively,  which  makes  a  total  of  EUR  40.8  million  of 
operational  costs  or  9%  of  the  total  Programme  budget.  This  ratio  is  comparable  to  or 
lower  than  the  operational  cost  ratios  of  other  EU  programmes83  of  similar  size  and 
demonstrates the efficiency of the Programme. 

Several  consulted  stakeholders  (particularly  those  involved  in  project  grants  and  joint 
actions) expressed satisfaction with the fact that the budget allocated to actions could be 
changed  within  a  certain  limit,  without  having  to  make  formal  amendments  to  the 
underlying  grant  agreements.  This  avoided  long  administrative  procedures  linked  with 
the  preparation  of  amendments.  Funding  could  be  transferred  across  different  cost 

83 Eg. the Consumer Programme (2014-2020) [Regulation (EU) No 254/2014 of the European Parliament 
and of the Council of 26 February 2014 on a multiannual consumer programme for the years 2014-20 and 
repealing  Decision  No 1926/2006/EC  (OJ  L  84,  20.3.2014,  p.  42–56)]  e.g.,  the  Consumer  Programme 
(2014-2020). 

36 

 
 
 
 
categories  and  partners,  which  was  useful  to  many  stakeholders,  particularly  given  the 
uncertainty created by the COVID-19 outbreak. 

b)    to  what  extent  are  the  costs  associated  with  the  Programme  proportionate  to  the 

benefits it has generated? 

The  surveyed  stakeholders  found  the  management  and  operational  costs  of  the 
Programme to be generally reasonable, whereas the costs relating to the administration, 
preparation  and  coordination  of  proposals,  and  the  associated  personnel  costs,  were 
generally seen as disproportionate, especially for countries with a lower GNI and smaller 
organisations involved in funded actions.  

Although  improved  and  simplified,  administrative  costs  were  still  found  high  by  most 
categories of stakeholders, who also perceived the level of detail required for monitoring 
and reporting as ‘heavy’ and at times ‘bureaucratic’. 

Funded actions which were found to be the most effective were also found to entail the 
highest efficiency.   

Consulted stakeholders considered that the costs of specific actions were proportional to 
their benefits. These include the SCIROCCO Exchange Project84 which developed a self-
assessment tool for integrated care with a limited budget, which is now used by regional 
and national healthcare authorities in the EU and beyond85. ERNs were also mentioned as 
an area where the benefits were high, compared to related costs.  

Considering  the  specific  objective  1  (promote  health,  prevent  diseases  and  foster 
supportive environments for healthy lifestyles), the total costs (for the Commission, the 
Member States and Project participants) for implementing these actions are estimated at 
EUR 200 million. The qualitative direct and indirect benefits include 24 Member States 
using evidence-based good practices in health promotion and disease prevention notably 
on  nutrition  (in  particular  in  the  area  of  saturated  fat)  and  also  in  other  areas  such  as 
physical  activity,  alcohol  and  tobacco,  cardiovascular  diseases  and  diabetes.  As  regard 
cancer, the qualitative benefit includes the fact that 6 Member States are using evidence-
based  good  practices  on  EU  guidelines  and  accreditation  for  breast  cancer  screening. 
Moreover,  a  database  of  128  cancer  registries  from  29  European  Countries  has  been 
established  (including  on  breast,  colorectal  and  cervical  cancers)  and  provide  data  for 
more  than  25  900  000  records.  These  registries,  data  and  screening  guidelines  are 
accessible to researchers and specialised healthcare professionals, as tools for improving 

84  The  SCIROCCO  project  had  an  estimated  budget  of  EUR  2.2  million.  Taking  into  additional  cost 
(administrative and preparatory costs) incurred by beneficiaries and by the Commission, the total cost for 
implementing  the  Project  is  estimated  at  EUR  3.2  million.  Although  the  benefits  of  the  project  have  not 
been estimated in monetary terms, it is to be noted that has been positively evaluated, since it has generated 
short-term and longer-term results and tools that  used in 35 countries by hundreds of thousands of users, 
and the users have reportedly found it very useful. It can thus be inferred  that the qualitative results of the 
project exceed by several folds its total costs. 
85 This was mentioned by a government official/policymaker in the Project Grants focus group. 

37 

 
 
 
and  accelerating  diagnostics  and  for  providing  patients  with  more  tailored  treatments. 
The direct and short-term benefits from Programme are enhanced by longer term benefits 
due to positive changes of behaviour of the population (healthier eating habits, physical 
activity, reduced abusive consumption of alcohol and smoking) to which the Programme 
has contributed (see annex IV for more information). 

Concerning  specific  objective  2  (protect  EU  citizens  from  serious  cross-border  health 
threats)  the  estimated  total  costs  of  funded  actions  amounted  to  EUR  70  million. 
Qualitative  benefits  from  these  actions  include  the  production  of  new  knowledge  on 
health threats, the improvement of the diagnostics capacity of EU laboratory dealing with 
emerging  pathogens.  Funded  actions  contributed  to  the  response  to  frequent  (and 
moderate)  health  threats  that  occurred  during  the  implementation  period  of  the 
Programme,  including  outbreaks  in  third  countries  (Zika,  Ebola).  Twenty-two  Member 
States  integrated  coherent  approaches  in  the  design  of  their  prepared  plans,  even  if  the 
COVID-19 outbreak demonstrated weaknesses in preparedness and response to a major 
health threat of such magnitude at EU and Member States level. At the beginning of the 
COVID-19 outbreak, the Programme funded-actions were switched from inter-epidemic 
to  emergency  mode  and  geared  at  combatting  the  pandemic,  particularly  in  its  early 
stages,  before  adoption  of  the  EU4Health  Programme  (2021-2027)86.  Lessons  learned 
from  the  COVID-19  pandemic  enabled  to  design  the  EU4Health  Programme,  with 
actions aimed at  strengthening  the preparedness  and response of Member States and  of 
the EU to cross-border health threats87. 

As  regards  the  specific  objective  3  (contribute  to  innovative,  efficient  and  sustainable 
health systems) the total costs of implemented actions is estimated at EUR 160 million. 
Significant  qualitative  benefits  are  derived  from  funded  actions  by  Member  States, 
Stakeholders  and  EU  citizens  (see  annex  IV  for  more  information).  Funded  actions 
provided support to several EU policy developments:  

- 

the joint action on secondary use of health data (TEDHAS) paved the way for 
the  Commission  proposal  on  the  European  Health  Data  Space  (EHDS)  by 
developing  European  principles  for  the  secondary  use  of  health  data.  The 
EHDS  is  expected  to  benefit  not  only  citizens  and  health  professionals  by 
ensuring greater access and control by individuals to their health data collected 
(primary  use)  but  also  to  researchers,  regulators  and  policymakers  by 
supporting access to health data for secondary purposes that would benefit the 
society such as research and innovation, patient safety, personalised medicine, 
policy-making or regulatory activities (secondary use). 

86 REGULATION (EU) 2021/522 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 24 
March  2021  establishing  a  Programme  for  the  Union’s  action  in  the  field  of  health  (‘EU4Health 
Programme’) for the period 2021-2027, and repealing Regulation (EU) No 282/2014 
87 Although not in the scope of this document, other measures resulting from the lessons learned from the 
COVID-19  pandemic  are,  for  example,  the  setting-up  of  the  Health  and  Emergency  Response  Authority 
(HERA)  and  the  revision  and  strengthening  of  the  EU  legislation  on  cross-border  health  threats: 
REGULATION (EU) 2022/2371 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 23 
November 2022 on serious cross-border threats to health and repealing Decision No 1082/2013/EU  

38 

 
 
 
- 

the  adoption  of  a  legislation  on  permanent  EU  cooperation  in  health 
technology  assessment  (HTA)  which  resulted  from  the  series  of  EUnetHTA 
joint actions; improvement of country knowledge in the field of health through 
the  State  of  Health  in  the  EU  publication  cycle  and  the  related  Country 
Profiles. 

Concerning the specific objective 4 (facilitate access to better and safer healthcare for EU 
citizens) the total costs of funded actions are estimated at EUR 100 million.  Qualitative 
benefits  include,  inter  alia,  the  establishment  of  24  ERNs  which  enable  access  to 
healthcare concerning 6000-8000 rare diseases affecting the daily lives of around 30 million 
people in the EU, with 1185 healthcare providers, and 2100 virtual expert panels opened in 
the  clinical  patient  management  system  (CPMS).  1.7  million  patients  were  treated  by 
ERN members, and patient participated in 732 clinical trials within the ERNs. (See annex 
IV). The total amount spent should also be put in perspective with the less than EUR 60 
per  patient  targeted  expert  diagnosis  which  otherwise  would  not  be  accessible  to  those 
patients. 

Evaluating the above qualitative benefits of funded actions, against their  total costs and 
also  weighing  the  qualitative  benefits  stemming  from  cross-cutting  (horizontal)  actions 
relatively  to  their  total  costs  (estimated  at  EUR  20  million)  it  can  be  inferred  that  the 
benefits  generated  by  the  Programme  are  proportionate  with  its  total  costs  of 
implementation. These benefits can be further magnified by positive longer-term effects 
that the Programme may have contributed to create, such as the establishment of contacts 
and knowledge sharing and learning networks among health professionals, policymakers 
and  health  authorities  of  different  Member  States,  change  of  the  behaviour  of  the 
population towards healthier habits and lifestyles.  

c) 

to  what  extent  do  factors  linked  to  the  Programme  influence  the  efficiency  with 
which the observed achievements were attained? 

The  type  of  funding  mechanism,  and  the  availability  of  financial  and  human  resources 
are  the  main  factors  influencing  the  efficiency  with  which  results  were  attained.  The 
evaluation also analysed the influence of additional factors, such as the right balance of 
actors  involved  in  the  funded  actions.  The  limited  sustainability  also  affected  the 
efficiency  of  a  funded  action.  Finally,  outside  factors  affected  the  efficiency  of  the 
Programme,  for  example  internal  budgetary  and  accounting  rules  and  procedures  for 
hiring staff in participating organisations. 

d) 

to  what  extent  was  the  distribution  of  Programme  credits  among  the  four  specific 
objectives efficient? 

The  distribution  of  Programme  credits  among  the  four  specific  objectives  was  found 
efficient,  since  it  addressed  the  key  health  needs  identified  during  the  implementation 

39 

 
 
 
 
period. Analyses conducted showed that the Programme provided funding in a way that 
met the key health needs in the EU over the Programme implementation period.  

Surveyed  stakeholders  found  funding  critical  to  achieving  the  expected  results  for  both 
the four specific objectives and the 23 thematic priorities.  Two thirds of respondents to 
the  targeted  survey  considered  that  the  thematic  priority  structure  of  the  Programme 
fostered efficiency. 

e) 

if  there  are  significant  differences  in  costs  (or  benefits)  between  participating 
countries, what is causing them? How do these differences link to the Programme? 

Desk research undertaken as part of this evaluation indicates that the distribution of 
funding and actions has not been evenly spread across participating countries. 

An analysis of HaDEA’s database of funded actions revealed that across the Programme, 
there  were  25  coordinating  countries.  Overall,  countries  in  Western  Europe  were  much 
more likely to coordinate a funded action than countries in Northern or Eastern Europe. 

These  disparities  in  EU  funding  and  in  the  number  of  actions  coordinated  led  to  some 
differences in costs and benefits between participating countries. 

The  factors  identified  as  being  responsible  for  the  differences  in  costs  and  benefits 
between  the Programme’s  participating countries included different  financial resources, 
an  uneven  organisational  capacity  to  deliver  funded  actions,  and  the  administrative 
burden of applying for and receiving Programme funding. 

f) 

to  which  extent  did  the  simplification  measures  contribute  to  the  efficiency  of  the 
Programme?  Was  there  further  scope  for  simplification  to  make  the  Programme 
implementation more efficient? 

As  identified  in  the  midterm  evaluation,  a  wide  range  of  systems  and  processes  were 
simplified and digitised to streamline the Programme’s administration. The m ain 
simplifications to systems and processes were the following:  

  Application and grant management procedures were simplified and digitised. 
  Procedures for awarding joint actions and grants were simplified. 
  The rules of the Programme were made less complex, i.e., by harmonising the co-

financing rates to 60% (or up to 80% in cases of exceptional utility). 

  Operating  grants  were  allowed  to  be  funded  through  framework  agreements 

(which run for up to 3 years). 

  2016 ERN grants had been made longer-term (5 years) to ‘establish a partnership 
procedure for important actors at EU level, offer a clearer financial perspective for 
ERNs  and  provide  more  stability  and  efficiency  gains  for  all  involved’.  The 
procedure for this was the signing of an FPA and insurance of annual co-funding 
through  an  SGA.  It  was  acknowledged  that  having  these  two  consecutive  steps 
(and  repeating  annual  SGAs)  entailed  more  administration  for  applicants,  but  it 
was expected that the process would simplify awarding of ERNs in the future.  

  A quality assurance process was introduced for joint actions. 

40 

 
 
  The  requirements  for  amendment  procedures  were  simplified,  most  importantly 
by  allowing  beneficiaries  to  transfer  resources  between  different  cost  categories 
without the need for an amendment. 

  Electronic tools were introduced for the submission of proposals, management of 
grants  and  e-reporting  and  monitoring.  In  2015,  electronic  monitoring  and 
reporting  were  introduced  to  save  time;  beneficiaries  and  CHAFEA  became 
paperless88. 

  All electronic tools were centralised on the Participant Portal. 
  Simplification  measures  relating  specifically  to  the  exceptional  utility  criteria 

were introduced. 

  Conditions  were simplified and made less  restrictive, especially  for joint  actions 
where  the  list  of  five  criteria  was  reduced  to  two,  one  of  which  concerned  the 
proportion of funding that had to be allocated to staff. 

  The explicit upper limit on the proportion of funded projects that can be awarded 
exceptional  utility  (under  2HP:  ‘No  more  than  10%  of  funded  projects  should 
receive EU co-funding of over 60%’) was discontinued. 

Efforts  were  made  to  improve  the  efficiency  of  the  Programme  by  simplifying  and 
streamlining  procedures.  In  2015,  the  EU  saw  a  large  migration  flux  in  its  territory.  In 
response,  the  Agency  quickly  launched  direct  grants  and  call  for  proposals  for  projects 
addressing the  refugee crisis  and was  able to  sign the selected grant  agreements within 
less than 3 months of the 2015 AWP amendment. This was made easier by the simplified 
administrative procedures introduced in 2014 as well as the portal for online submissions 
and  the  online  evaluation  and  electronic  signature  of  grant  agreements  used  for  the 
Horizon 2020 programme. 

These  simplifications  reduced  administrative  costs  and  improved  the  efficiency  of  the 
Programme.  They  also  reduced  paperwork.  They  helped  reduce  costs  and  made  the 
application processes smooth. The funding portal played a key role by providing manuals 
and useful links.  

However, according to a minority of stakeholders, simplification measures did not reduce 
administrative  burden/costs  which  they  still  found  unreasonable.  These  respondents 
suggested ways to further reduce costs (e.g. by using unit costs and lump sums, and by 
further simplifying the information requested in the application forms). 

g) to what extent were the monitoring processes and resources (at the Commission and 

MS level) cost-effective? 

Monitoring  processes  were  found  to  be  fairly  efficient  and  reasonable.  Key  factors 
enabling  their  efficiency  were  the  relevance  and  clarity  of  indicators  along  with  the 
digitalisation  of  the  processes.  The  monitoring  (and  reporting)  process  helped 
Programme  participants  to  plan  their  work  effectively  and  understand  the  expectations, 
thereby improving the quality of their deliverables. 

88   European  Commission.  (2018),  Report  from  the  Commission  to  the  European  Parliament  and  the 
Council:  Implementation  of  the  third  Programme  of  the  Union's  action  in  the  field  of  health  in  2015. 
Available from: https://ec.europa.eu/health/sites/default/files/funding/docs/ implementation2015_en.pdf  

41 

 
 
 
Although  they  improved  throughout  the  Programme  implementation  period,  mainly 
thanks to digitalisation, the monitoring processes could be  further simplified to achieve 
more efficiency gains, for example through further centralisation of the information.  

The  monitoring  processes  are  still  found  ‘too  heavy’  and  ‘too  detailed’,  especially  for 
in,  who  call  for  further 
individual  experts  brought 
smaller  organisations  and 
simplification. Furthermore, there is still a need for more measurable and comprehensive 
monitoring indicators. 

h) what are the benefits of the reporting systems against their costs and how could they 

be effectively implemented? 

The reporting systems provided several benefits, including the possibility of tracking the 
progress of actions against the original plan. Factors that made reporting more efficient 
included  the  Compass  and  SYGMA  reporting  systems,  which  enabled  beneficiaries  to 
report back to the Commission with less administrative burden and to track projects from 
start to end. Another benefit was the portal which made reporting more efficient. 

Stakeholders consulted as part of this evaluation had mixed views as to whether the costs 
associated  with  the  reporting  were  proportionate  in  relation  to  the  benefits.  The 
Commission’s 4-step reporting cycle was praised, while others highlighted the need for 
better guidance to avoid that the costs of the reporting systems outweigh the benefits.  

i)  summary of costs and benefits identified in the evaluation 

42 

 
 
 
Table 3. Summary of costs and benefits identified in the evaluation 

                               Relevant stakeholders 

EU Citizens  

Businesses/Contractors 
awarded Procurement 
contracts 

Project 
participants/NGOs/Stakeho
lders other than National 
Authorities 

Member States Authorities 
/ National Administrations  

European Commission (DG 
SANTE, CHAFEA, Other 
Commission services 

Costs and benefits 

Quantitative   Qualitative  Quantitative  Qualitative  Quantitative   Qualitative  Quantitative  Qualitative  Quantitative 

Qualitative 

Specific objective 1:  promote health, prevent diseases and foster supportive environments for healthy lifestyles 

Cost or Benefit description 

Costs:  

Direct Costs: Monetary input 
into implementation of best 
practices and other actions 
under specific objective 1 

Recurrent  

N/A 

N/A 

N/A 

N/A 

Direct benefits 

Direct benefits 

procurement 
MEUR 46 

43 

MEUR 30 of 
co-funding 
for Projects 
and Direct 
Grant 

EU co-
funding of 
grants MEUR 
56 

MEUR 32 of 
co-funding of 
Joint Actions 

EU co-
funding Joint 
Actions: 
MEUR 36 

Estimated 
MEUR 139  

N/A 

A prize for 
NGOs is 
organised 
annually  

24 MS using 
evidence-
based good 
practices  

6 MS using 
evidence-
based good 
practices on 
breast 
cancer 
screening 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Online best-
practices 
portal 6 6650 
visitors  

On cancer: 
128 cancer 
registries 
from 29 
European 
Countries, 
Data on 25 
900 000 
records  

Uptake of 
best 
practices 
on: cancer, 
tobacco 
control, 
physical 
exercise, 
nutrition, 
cardiovascu
lar disease 
and 
diabetes 

Direct benefits 

Other direct benefits 

Indirect benefits 

Better health 
outcomes for 
population 
benefiting 
from 
implementati
on of best 
practices:  

Specific objective 2:  protect Union citizens from serious cross-border health threats 

Costs 

Productions 
of 
knowledge 
and tools at 
EU level:  

44 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
N/A 

N/A 

Direct costs linked with 
Programme implementation 

Benefits 

Direct benefits 

Co-funding 
from MS 
MEUR 7 

MEUR 41 
specific 
objective 2 

During the 
COVID-19 
outbreak 
actions of the 
Programme 
have been 
geared, in 
their 
‘emergency 
mode’, 
towards 
combating 
the pandemic 

During the 
COVID-19 
outbreak 
actions of 
the 
Programme 
have been 
geared, in 
their 
‘emergency 
mode’, 
towards 
combating 
the 
pandemic 

Co-funding 
from another 
organisation: 
MEUR 11 

During the 
COVID-19 
outbreak 
actions of the 
Programme 
have been 
geared, in 
their 
‘emergency 
mode’, 
towards 
combating 
the pandemic 

Examples 
Healthy 
Gateways 
Joint 
Action, the 
JA on 
Strengthene
d 
Internationa
l Health 
Regulations 
and 
Preparedne
ss 
(SHARP), 
Training for 
Health 
Professiona
ls on 
isolation, 
waiting 
rooms and 
reception 
areas, 
cleaning, 
appropriate 
PPE, real 
time RT-
PCR testing 
for 
COVID-19 

45 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Production of 
new 
knowledge on 
health-threats 
related issues 
at EU level  

Production 
of new 
knowledge 
on health-
threats at 
EU level 

Production 
of new 
knowledge 
on health-
threats 
related 
issues at 
EU level 

Feasibility 
study 
launched 
for the 
developmen
t of a 
common 
EU 
vaccination 
card 

22 Member 
States 
integrated 
coherent 
approaches in 
the design of 
their 
preparedness 
plan 

JA EMERGE 
improved the 
capabilities 
for rapid 
laboratory 
diagnosis for 
new or 
emerging 
pathogens 

Support to 
interventions 
in 2014-2016 
to limit 
spread of 
Zika and 
Ebola  

Direct benefits 

Direct benefits 

Direct benefits 

Indirect benefits 

N/A 

Procurement 
MEUR 14 

Better 
protection of 
the 
population 
against cross-
border health 
threats 

Better 
protection 
from vaccine-
preventable 
diseases 
through 
interventions 
to improve 
vaccine 
uptake 

46 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Specific objective 3: contribute to innovative, efficient and sustainable health systems 

Costs 

Direct Costs linked with 
implementation of the 
Programme 

Benefits 

Direct benefits 

N/A 

N/A 

Contribution 
from other 
participating 
organisations: 
MEUR 30 

Procurement 
MEUR 37 

Total 
appropriatio
ns MEUR 
110 

Contribution 
of MS 
participating 
in Joint 
Actions: 
MEUR 19 

41 HTAs per 
year 
produced per 
year 

23 MS 
using 
identified 
tools and 
mechanism
s, 
contribute 
to effective 
results in 
their health 
systems  

47 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Direct benefits 

Direct benefits 

Indirect benefits 

Organ 
database 
which has 
helped 34000 
transplants in 
2017 

Setting up of 
the EU 
Health Policy 
Platform 
(HPP) a 
collaborative 
on-line tool  

Strengthenin
g of Health 
Systems of 
the Member 
States for the 
benefit of EU 
citizens and 
patients 

48 

Launching 
of 
THEDAS 
JA (The 
European 
Health 
Dataspace) 
to develop 
principles 
for cross-
border 
secondary 
use of 
health data 

Improvement 
of country 
knowledge in 
the field of 
health through 
regular 
publication of 
The State of 
Health in the 
EU, the 
Country 
Profiles and 
Health at a 
Glance series 

Successful 
implementati
on of 
EUnetHTA 
JA3 which 
led to a 
permanent 
regulation on 
EU 
cooperation 
on HTA 

JA on Market 
Surveillance 
on Medical 
Devices 
reinforced the 
market 
surveillance 
system for 
these devices 
and 
improving 
cooperation 
among all MS 

Cost savings 
for health 
systems by 
use of tools 
such as HTA 
and 
digitalisation 
policies 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Specific objective 4:  facilitate access to better and safer healthcare for Union citizens 

Costs 

Direct costs for implementing 
the Programme 

Contribution 
of other 
organisations: 
MEUR23 

Contribution 
of MS in 
Joint Actions: 
MEUR 15 

Total 
appropriatio
ns MEUR 
85 

Benefits 

Direct benefits 

Direct benefits 

Indirect benefits 

Procurement
: MEUR 27 

24 European 
Reference 
Networks 
(ERNs) 
established 

27 MS using 
the tools 
developed 

24 European 
Reference 
Networks 
(ERNs) 
established 

1185 
healthcare 
providers and 
centre of 
expertise 
joined the 
ERNs 
2,100 virtual 
expert panels 
opened in the 
CPMS 

1.7 million 
patients 
treated by 
ERN 
members 
Patients 
participated 
in 732 
clinical trials 

Easier access 
to treatment 
for patients 
with rare 
diseases 

Cross-cutting issues and overall Programmes costs and benefits 

49 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Overall Administrative Cost 
for the implementing the 
Programme 

Cost for functioning of the 
executive Agency CHAFEA 

MEUR  10 

MEUR 30 

50 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4.1.3 Coherence 
a)  are the actions implemented under the Programme coherent with its objectives? How 

has the coherence of the Programme influenced its effectiveness? 

The  evaluation’s  analysis  showed  that  the  actions  funded  by  the  Programme  were 
coherent  with  each  other  and  aligned  with  the  Programme’s  objectives  and  with  DG 
SANTE’s  activities.  This  is  evidenced  by  a  review  of  DG  SANTE’s  annual  activity 
reports for 2014-2020 and the mapping of its fields of activity to the specific objectives 
and thematic priorities of the Programme, for each year. 

The coherence between the actions facilitates synergies, complementarities and mutually 
reinforcing  mechanisms between actions launched under the same  or different  thematic 
priorities. 

b)  to what extent have the priorities of the Programme led to more synergy, focus and 
coherence  between  the  EU-funded  actions  in  delivering  on  similar  objectives?  Did 
the Programme encourage cooperation with the European Structural and Investment 
Funds  and  other  EU  financial  instruments?  To  which  extent  is  the  Programme 
coherent with wider EU policy and with international obligations? 

Links with other EU financial instruments were built into the design of the Programme. 
Regulation (EU) No. 282/2014 states that the Programme should promote synergies with 
other  EU  programmes  funding  actions  in  the  field  of  health,  such  as  the  Framework 
Programme  for  Research  and  Innovation  2014-2020  (Horizon  2020)  and  the  European 
Structural and Investment Funds (ESIF).  

Synergies were built with the Horizon 2020, for example, in the area of rare diseases and 
ERNs. The Programme’s actions were complementary to Horizon 2020 projects such as 
the European Joint Programme co-fund on Rare Diseases (EJP RD) and the Coordination 
and  Support  Action  for  the  24  ERNs  (ERICA),  as  well  as  EU-funded  collaborative 
research projects (e.g. Solve-RD, Conect4Children). 

Direct links between the Programme and Horizon 2020 are established for cancer and for 
specific thematic priorities (e.g. AMR, chronic diseases, rare diseases, …). This  is  also 
the  case  for  action  on  HTA  under  objective  3  (Contribute  to  innovative,  efficient  and 
sustainable health systems). The Programme aimed to facilitate the uptake of the results 
stemming  from  research  projects  supported  under  Horizon  202089,  including  within  the 
feedback  to  policy  (F2P)  framework  developed  by  the  Research  and  Innovation 
Directorate-General  of  the  Commission  (DG  RTD)  in  2020.  Similarly,  the  Programme 

89 In areas such as personalised medicine, one  health/antimicrobial resistance,  infectious diseases (Ebola, 
Zika), COVID-19 related projects 

51 

 
 
 
 
 
aimed  to  facilitate  the  uptake  of  Horizon  2020  projects’  results  in  the  area  of  effective 
and efficient investment and innovation in public health and health systems (objective 3 – 
Thematic  thematic  priority  3.4  Setting  up  a  mechanism  for  pooling  expertise  at  EU 
level)90. 

Moreover, Horizon 2020 and the ESI  Funds directed funding to health-related activities 
over the Programme implementation period (2014-2020). 

When  considering  EU  action  in  the  field  of  research  and  innovation,  the  Regulation 
establishing  Horizon  2020  included  health,  demographic  change  and  well-being  as  a 
specific objective under the priority ‘Societal challenges’91. Research priorities included 
topics  such  as  personalised  medicine,  health  promotion  and  disease  prevention, 
innovative  health  and  care  systems,  non-communicable  and  infectious  diseases,  rare 
diseases,  antimicrobial  resistance,  global  health  and  the  digital  transformation  in  health 
and care. 

Interlinkages  and synergies between the  Programme and ESIF were sought  and created 
during the Programme implementation period, as results from Programme-funded actions 
served as a basis for actions financed through ESIF. Examples of health-related actions 
financed  by  ESIF  include  the  promotion  of  digital  public  services  through  the 
deployment of digital health solutions and the provision of accessible medical services to 
vulnerable groups92. 

Programme-funded actions contributed to the EU’s wider policies and wider international 
obligations. 

In  particular,  DG  SANTE’s  specific  objectives  for  Programme  spending  were  aligned 
with  and  built  on  wider  EU  policy  priorities:  the  Europe  2020  strategy  for  smart, 
sustainable  and  inclusive  growth  in  2014-2015;  the  Juncker  Commission’s  priorities  in 
2016-2019; and the Von der Leyen Commission’s priorities in 2020. 

For instance, in 2016-2019 the Programme’s thematic priority 1.3 ‘Cost-effective health 
promotion and disease prevention’ and 1.4 ‘Effective, accessible and resilient healthcare 
systems in the EU’ contributed to the Commission priority A new boost for jobs, growth 
and investment in the EU, and in 2020 the thematic priority 2.2 ‘Patients’ access to safe, 
innovative and affordable medicines and medical devices’ contributed to the Commission 
priority Promoting our European Way of Life. 

On  the  alignment  of  the  Programme  with  wider  international  obligations  that  share 
common  objectives  with  the  Programme,  information  reviewed  shows  that  the 

90   European Union., 2014. Regulation (EU) No 282/2014: 
https://eurlex.europa.eu/legalcontent/EN/TXT/PDF/?uri=CELEX:32014R0282&from=EN.  
91   European Union., 2013. Regulation (EU) No 1291/2013 of the European Parliament and of the Council 
establishing Horizon 2020 - the Framework Programme for Research and Innovation (2014-2020) and 
repealing Decision No 1982/2006/EC (OJ L 347, 20.12.2013, p. 104–173).  
92 European Commission, European Structural and Investment Funds 2014-2020. 2020 Summary report of 
the programme annual implementation reports covering implementation in 2014-2019.  

52 

 
 
 
Programme was  aligned  for instance with  the WHO common policy framework Health 
202093. 

In  conclusion,  the  Programme  encouraged  cooperation  and  was  aligned  with  other  EU 
financial instruments that support health-related activities. However, such alignment and 
cooperation  cannot  be  considered  as  fully  achieved,  despite  efforts  made  to  forge  links 
and synergies with other financial instruments. 

c)  to  what  extent  has  the  Programme  proved  complementary  to  other  EU  or  Member 

States targets, interventions, and initiatives in the field of health? 

The evaluation demonstrates that the Programme has been coherent with other EU health 
policies during its timeframe (up to 2020) and that there has been alignment between the 
relevant Commission departments and the different EU policies in the area of health. The 
Programme was also coherent with Member State health-related strategies and initiatives. 

A  mapping  exercise  of  selected  EU  health-related  initiatives  adopted  up  to  2020 
(including  EU  legislation  on  medicinal  products  for  human  use,  medical  devices  and 
tobacco control) showed that those initiatives are aligned with the Programme objectives. 
In particular, initiatives such as the 2012-2020 eHealth action plan, the action plan for the 
EU health workforce94, the 2014-2020 EU action plan on childhood obesity and the One 
Health  action  plan  against  Antimicrobial  Resistance95,  AMR,  were  aligned  with  the 
Programme objectives. 

Furthermore,  a  review  of  the  documentation  demonstrated  alignment  between  different 
Commission services on the direction of health policies. This coherence is also reflected 
in  the  policy  coordination  between  relevant  Commission  departments  and  between  the 
different EU policies and mechanisms involving health. A key example is the European 
Semester, the Commission’s annual cycle of policy coordination, which includes health-
related priorities among its recommendations. Similarly, the activities of DG REFORM 
through the structural reform support programme and of DG REGIO and DG RTD have 
been found to be overall aligned with the Programme objectives. 

93 World Health Organisation (2013), ‘Health 2020. A European policy framework and strategy for the 21st 
century’. Available from: https://www.euro.who.int/__data/assets/pdf_file/0011/199532/Health 
2020-Long.pdf 
94 The 2012-2020 eHealth Action plan and the Action plan for the EU Health workforce were both adopted 
before the entry into force of the Programme and covered the Programme’s implementation period. 
95  COMMUNICATION  FROM  THE  COMMISSION  TO  THE  COUNCIL  AND  THE  EUROPEAN 
PARLIAMENT  A  European  One  Health  Action  Plan  against  Antimicrobial  Resistance  (AMR)  - 
COM/2017/0339 final, see also amr_2017_action-plan_0.pdf (europa.eu) 

53 

 
 
 
 
4.2.  How did the EU intervention make a difference and to whom? 

a)  what is the additional value resulting from the Programme, compared to what could 
reasonably  have  been  expected  from  Member  States  acting  at  national  and/or 
regional  levels,  and  compared  to  what  the  EU  would  have  achieved  without  the 
Programme? 

In accordance with Article 168(7) TFEU, Union action shall  respect the responsibilities 
of the Member States for the definition of their health policy and for the organisation and 
delivery of health services and medical  care. The responsibilities of the Member States 
shall include the management of health services  and medical  care and the allocation of 
the  resources  assigned  to  them.  However,  the  importance  of  EU  action  in  the  field  of 
public health is  acknowledged in the Treaty on the Functioning of the European Union 
(e.g.,  Articles  6  and  168  TFEU),  which  refers  to  the  Union’s  role  in  complementing, 
supporting,  and  coordinating  national  actions.  Within  the  remit  of  its  competences,  EU 
action in the field of health can add value to national efforts and support Member States 
in achieving common objectives and tackling common challenges such as serious cross-
border  health  threats,  preventing  and  managing  non-communicable  diseases,  promoting 
good health, improving access to care and supporting health systems. 

From  the  analysis,  it  emerged  that  the  Programme-funded  actions  provided  EU  added 
value according to the definition in Section 2.2.  

Regarding  the  specific  thematic  areas,  the  Programme’s  midterm  evaluation  identified 
the added value of Programme actions in areas such as capacity building against health 
threats,  pooling  expertise  and  resources  across  the  EU  to  reduce  health  inequalities, 
collaboration  in  the  field  of  health  technology  assessment  (HTA)  and  eHealth,  and 
exchange  and  implementation  of  best  practices  for  promoting  health  and  preventing 
diseases. 

On  the  EU  added  value  of  the  Programme’s  funding  mechanisms,  stakeholders 
mentioned  that  procurement  contracts  led  to  the  production  of  EU-wide  studies  that 
provided valuable information on the public health situation and issues across EU. This 
was  perceived  to  go  beyond  what  would  have  been  achieved  by  individual  Member 
States, acting alone. 

The Programme strongly supported the sharing of best practices. This can be seen in DG 
SANTE’s  online  ‘best  practice  portal’,  launched  in  2018.  The  sharing  of  best  practices 
and  networking  across  Member  States  is  seen  as  an  example  of  the  Programme’s  EU 
added value96. Examples of actions on sharing best practices include the EU Compass for 
action on mental health and wellbeing, the CHRODIS+ Joint Action and the Young5097 

96 LOMBA, N., 2019. The benefit of EU action in health policy: The record to date. 
97  The  YOUNG50  project  aims  to  transfer  the  Italian  best  practice  CARDIO50  project  in  Lithuania, 
Romania, Luxembourg among 50 years old.  
The  objectives  of  CARDIO  50  were  to  estimate  cardiovascular  risk  among  the  50  years  old  population, 
identify  persons  with  inadequate  lifestyles,  new  cases  of  hypertension,  hyper-glycemia  and  hyper 

54 

 
 
 
 
project. The Programme also delivered EU added value by encouraging cooperation and 
coordination on specific policy issues among Member States. This is especially the case 
in areas such as rare diseases, alcohol consumption. 

HTA is  another area of  strong EU added value.  The EUnetHTA Joint Action 3 funded 
under  the  Programme  created  a  collaborative  infrastructure  for  national  and  local  HTA 
authorities  and  enabled  sustainable  cooperation,  which  was  reflected  in  the  recently 
adopted  HTA  Regulation.  EUnetHTA  Joint  Action  3  builds  on  the  lessons  of  earlier 
EUnetHTA  joint  actions  funded  under  previous  health  programmes.  The  overall 
objective of EUnetHTA JA3 was to establish a model for joint work on HTA that would 
continue after its completion. It also set out to increase the amount of high quality HTA 
joint work, promote its uptake and implementation at national, regional and local level, 
and  support  evidence-based,  sustainable  and  equitable  choices  in  healthcare  and  health 
technologies. 

EUnetHTA JA3 (and its predecessors) has achieved its overarching objective and laid a 
strong  foundation  for  sustainable  cooperation  which  is  reflected  in  the  permanent 
framework for joint work set up by the HTA Regulation. This regulation aims to tackle 
the remaining barriers to EU-wide HTA collaboration. While it is acknowledged that the 
longer-term benefits (i.e. the sustainability of health systems, a more efficient allocation 
of  resources  in  healthcare,  greater  innovation  and  transparency,  and  a  higher  level  of 
human health protection) depend on a variety of factors that go beyond the contribution 
of EU  action on  HTA, it  can be reasonably assumed that the outcomes achieved under 
the Programme on HTA are conducive to reaching those impacts. 

The setting  up  of 24  ERNs is  considered a flagship  achievement of the  Programme, as 
ERNs  provide  undisputable  EU  added  value.  The  ERNs  involve  healthcare  providers 
across  Europe,  and  are  an  example  of  how  the  EU  can  add  value  to  Member  States’ 
actions  by  enabling  them  to  coordinate  efforts,  pool  resources  and  take  advantage  of 
expertise across Europe. 

Another  area  where  EU  action  made  a  difference  by  supporting  coordination  and 
cooperation  among  Member  States  was  the  establishment  of  several  EU-wide  data 
systems. These included an EU quality register to ensure the safety of medical devices, a 
platform to facilitate organ transplants, an EU-wide tobacco tracking and tracing system 
to combat the trafficking of illicit tobacco products.  

Lastly,  the  Programme  provided  EU  added  value  by  enabling  mutual  learning  and  the 
development of new knowledge. Many respondents to the evaluation surveys considered 
one  of  the  most  important  examples  of  EU  added  value  of  the  Programme  to  be  the 
‘supporting  networks  for  knowledge  sharing  or  mutual  learning’.  The  Programme 

cholesterolemia,  activate  an  integrated  model  of  assistance  to  help  modify  or  reduce  risk  factors  among 
healthy  subjects,  promote  interventions  to  change  unhealthy  lifestyles  and  increase  knowledge  and 
perceptions of CVD risks among the general population. 

55 

 
 
 
enabled  mutual  learning  and  synergies  between  multiple  stakeholders  (national 
authorities,  healthcare  providers,  patient  organisations,  regulators  and  NGOs)  and 
Member States.  

b)  how  far  have  the  EU  added  value criteria  led to the development  of  proposals that 
better addressed these aspects? Are all of these criteria still relevant? Which criteria 
have been most/least addressed? 

Seven criteria set out in the Programme Regulation identify areas where Programme-
funded actions should provide added value. These are98:  

 
the exchange of good practices between Member States;  
  support networks for knowledge sharing or mutual learning;  
  addressing cross-border threats to reduce their risks and mitigate their 

consequences;  

  addressing certain issues relating to the internal market where the EU has 

substantial legitimacy to ensure high-quality solutions across Member States;  

  unlocking the potential of innovation in health;  
  benchmarking to allow informed decision-making at EU level; and 
 

improving efficiency by avoiding a waste of resources due to duplication and 
optimising the use of financial resources99. 

Evidence  from  the  evaluation’s  consultation  activities  shows  that  these  seven  criteria 
were used in funding proposals to  sufficient extent. The criteria which were considered 
the  most  important  were  the  exchange  of  best  practices,  support  networks  for  mutual 
learning and avoiding inefficient duplication of work. 

Survey  respondents  who  were  involved  in  the  management  and  administration  of  a 
Programme  action  mentioned  that  the  criteria  were  well-defined,  at  least  to  a  moderate 
extent.  However,  a  large  proportion  of  respondents  stated  that  they  did  not  know.  Of 
those who did provide an answer, 30% said the criteria were used, at least to a moderate 
extent.  

This  suggests that  the process  of integrating the  EU added value  criteria  into proposals 
could be made clearer and more systematic. 

The  EU  added  value  criteria 
implementation period and are considered useful for future health programmes.  

throughout 

remained 

relevant 

the  Programme 

98  Regulation  (EU)  No  282/2014  of  the  European  Parliament  and  the  Council  on  the  establishment  of  a 
third  Programme  for  the  Union's  action  in  the  field  of  health  (2014-2020)  and  repealing  Decision  No 
from:https://eurlex.europa.eu/legalcontent/EN/TXT/PDF/ 
1350/2007/EC. 
?uri=CELEX:32014R0282&from=EN [Accessed November 2021]]. 
99  European  Commission.  n.d.  Funding  under  the  third  Health  Programme  2014-2020:  The  European 
Added  Value.  Available  from:  https://ec.europa.eu/chafea/health/programme/documents/factsheets-hp-
av_en.pdf [Accessed July 2022]. 

Available 

56 

 
 
 
 
A large proportion (65%) of survey respondents who were involved in the management 
and  administration  of  a  Programme  action  stated  that  the  criteria  remained  relevant,  at 
least to a moderate extent. 

Moreover,  the  important  developments  and  revisions  initiated  by  CHAFEA  helped 
increase  the  relevance  of  these  criteria  during  the  Programme’s  lifespan.  Its  guide  for 
applicants for project grants released in 2018100, expanded on the list of EU added value 
criteria, with the following additions:  

Impact on target groups; 

 
  Long-term  effect  and  potential  multiplier  effect  such  as  replicable,  transferable, 

and sustainable activities;  

  Contribution  to  complementarity,  synergy,  and  compatibility  with  relevant  EU 
and  Member  State  policies  and  programmes,  including  compatibility  with  the 
European  Platform  on  Rare  Disease  registration  and  the  Commission’s  ERN 
Platform.  

Furthermore, the guide included the following ways to achieve added value:  

Implementing EU legislation; 

 
  Promoting best practice;  
  Benchmarking for decision-making;  
  Reducing cross-border threats;  
  Strengthening free movement of persons;  
  Strengthening networking activities.  

As regards the future relevance of the EU added value criteria, most survey respondents 
indicated  that  the  seven  added  value  criteria  should  be  retained  in  future  health 
programmes.  A minority (20%) stated that the criteria  should be modified. Suggestions 
for improving these criteria included:  

  Ensuring  the  involvement  of  civil  society  actors  (NGOs)  throughout  the 

programme; 

  Putting a stronger focus on health equity, health promotion and education; 
 
  Allocating funding to areas of unmet needs where EU action has particular added 

Including evidence-based work (activities, policies); 

value, such as rare diseases including childhood cancers.  

Further  improvements  to  help  strengthen  the  EU  added  value  of  EU  action  in  health, 
included  enhancing  cooperation  across  the  wider  Commission  services,  notably  across 
Directorate-Generals  of  the  Commission  (DGs)  and  involvement  of  agencies  and  other 
related organisations that would be beneficial in addition to DGs. 

100   Third Health Programme (2014-2020) Project Grants (HP-PJ) Guide for Applicants, European Commission.  

57 

 
 
 
 
 
4.3. 

Is the intervention still relevant? 

OVERALL MATCHING WITH MEMBER STATES’ HEALTH STRATEGIES 

The  Programme’s  midterm  evaluation  identified  a  set  of  public  health  and  healthcare 
needs  and  problems  at  the  time  when  the  Programme  was  established  in  2014101.  A 
majority of those  needs  remained  relevant  throughout  the  Programme’s lifespan (2014-
2020) and were identified as priority areas for participating countries and highlighted by 
all consulted stakeholders. 

During the Programme implementation, the main EU-wide health needs related to health 
promotion and better and safer healthcare. A mapping of national health strategies set out 
in the participating countries at the beginning of the Programme was conducted as part of 
the  evaluation.  This  mapping  revealed  that  the  most  common  priority  area  indicated  in 
country-level  health  strategies  was  objective  4:  better  and  safer  healthcare.  Key  health 
needs  relating  to  this  objective  included  rare  diseases,  patient  safety  and  quality  of 
healthcare.  

The  second  most  common  priority  area  was  objective  1:  health  promotion.  This  was 
confirmed by stakeholders who consider the promotion of healthy behaviours to be a key 
health need in the EU. 

Addressing health and social inequalities is also a key health need in the EU, with major 
health differences  across regions and socio-economic groups reported by several  of the 
stakeholders  interviewed102.  This  issue  was  also  prioritised  by  some  participating 
countries  at  the  beginning  of  the  Programme  (2014-2016).  Specific  groups  or  specific 
inequalities within the EU were identified, including women and children, children with 
cancer and cancer survivors. 

The Programme has  been  overall  relevant  to  the key health needs identified in  the EU. 
This is reflected in the allocation of the Programme funding, which appears proportionate 
to the priority areas expressed by participating countries, under each of the four specific 
objectives and with regard to health inequalities/health determinants and to other cross-
cutting issues. 

101  An  ageing  population,  threatening  the  financial  sustainability  of  health  systems  and  causing  health 
workforce  shortages;  A  fragile  economic  recovery,  limiting  the  availability  of  resources  to  invest  in 
healthcare;  An  increase  in  health  inequalities  between  and  within  Member  States;  An  increase  in  the 
prevalence of chronic disease; Pandemics and emerging cross-border health threats; The rapid development 
of health technologies; Increase in mental health problems (particularly among the young); Other specific 
emergency  situations  which  expose  EU  health  professionals  to  unprecedented  challenges  (for  example, 
dealing with the repercussions of the large increase in refugees); and Threats to environmental health such 
as air quality and pollution monitoring. 
102  Including  an  academic  /  research  stakeholder,  a  governmental  public  health  organisation,  and  an 
organisation representing patients and services users. 

58 

 
 
 
INCREASED  RELEVANCE  OF  PREPAREDNESS  AND  RESPONSE  TO  CROSS-BORDER  HEALTH 
THREATS 
Preparedness and response to health threats (objective 2) was not highlighted as priority 
by many countries. However, health threats are perceived as a challenge best addressed at 
EU and/or international level because of their cross-border and fast changing nature.  

This was due to the fact that health strategies of participating countries were defined at 
the beginning of the Programme over the period 2014-2016, at a time when cross-border 
health threats were not perceived as a first priority. 

The need to strengthen preparedness to health threats increased over time because of the 
frequent  (but moderate)  outbreaks that occurred  at the beginning of the implementation 
period and because of the Ebola and Zika outbreaks which were more significant, even if 
they originated in countries outside the EU territory.  

According to the stakeholders surveyed, the specific objective 2 (health threats) became 
more relevant over time, also because of the severity of communicable diseases ( such as 
HIV/AIDS,  hepatitis,  resurgence  of  tuberculosis  in  certain  countries).  Other  factors 
threats  were: 
relevance  of  preparedness 
justifying 
migrations/population movements, globalisation and environmental threats. 

to  health 

increase 

the 

The  implementation  of  the  Decision  No  1082/2013  (EU)103  on  serious  cross-border 
threats to health, contributed to more awareness on the need to strengthen preparedness 
and  response  to  cross-border  health  threats.  The  tools  provided  by  the  Decision  No 
1082/2013  (EU)  were  complemented  by  the  work  of  Health  Security  Committee,  the 
Early Warning and Response System – EWRS, the Health Emergency Operation Facility 
– HEOF and the Joint Procurement to purchase medical countermeasures, epidemiologic 
surveillance. 

The  COVID-19  outbreak  in  the  first  quarter  of  2020,  revealed  weaknesses  in  the 
preparedness and response to a major cross-border health threat of such magnitude. As a 
consequence  of  the  lessons  learned  from  the  pandemic,  preparedness  and  response  to 
cross-border  health  threats  became  the  first  health  priority  at  EU  and  Member  States 
levels. 

FLEXIBILITY AND ADAPTABILITY TO CHANGING AND UNFORESEEN NEEDS 

The Programme has been flexible and relevant to ongoing and changing health needs.  
For  example,  it  has  effectively  adapted  and  responded  to  the  migrant/refugee  crisis  in 
2015. 

103  Now  repealed  by  Regulation  (EU)  2022/2371  of  the  European  Parliament  and  of  the  Council  of 
23 November 2022 on serious cross-border threats to health 

59 

 
 
 
 
To  adapt  the  Programme  to  evolving  health  needs,  Article  14  of  Regulation  (EU)  No 
282/2014  establishing  the  Programme  states  that  following  the  midterm  evaluation,  the 
list  of  thematic  priorities  in  its  Annex  I  may  be  updated,  where  relevant,  by  removing 
certain  thematic  priorities  from  the  list  and  adding  new  ones.  This  provision  integrated 
flexibility and adaptability into the Programme from the beginning. 

In  addition  to  the  thematic  priorities,  the  Programme’s  midterm  evaluation  aimed  to 
assess the continued relevance of all the Programme objectives. 

However, the midterm evaluation did not recommend any change to the list of thematic 
priorities as it considered them to be still relevant to the health needs of the EU. 

The major health challenge during the Programme’s lifespan was COVID-19 outbreak in 
the first quarter of 2020. Although, this evaluation does not cover the comprehensive EU 
response to COVID-19, it should be noted that following the outbreak of the pandemic, 
the relevant actions funded by Programme were reoriented to the greatest extent possible 
from  an  ‘inter-epidemic  mode’  to  their  ‘emergency/pandemic  mode’  (see  Section  3.2.2 
for more information), with a view to contributing to combat the pandemic, in particular 
in its early stages. 

Stakeholders considered the work on COVID-19 as an example of how the Programme 
has  remained  relevant  to  public  health  changes  in  Europe.  They  highlighted  the  new 
comprehensive  approach  to  mental  health  adopted  by  the  Commission  in  June  2023104 
also  as  a  consequence  of  COVID-19,  and  considered  DG  SANTE’s  recent  call  for 
proposals on this topic as highly relevant. 

COMMISSION WIDER POLICIES, CITIZENS NEEDS AND CROSS-CUTTING PRIORITIES 

The actions funded under the Programme were found to be relevant and aligned with the 
Commission’s wider policies (see Section 4.1.3.b).  

Regarding citizens, the evaluation shows that the Programme has been largely relevant to 
their  needs,  as  confirmed  by  stakeholder  consultations  (PC,  interviews),  even  if  a 
minority of respondents stated that most thematic priorities were not directly relevant to 
people’s needs as they were too broad, and because the practical problems faced by many 
patients  (such  as  long  waiting  lists  to  receive  medical  care)  are  not  in  the  remit  of  EU 
health programmes.  

Certain  topics (e.g. childhood cancers – despite the establishment of ERN PaedCan for 
paediatric cancers) were seen to be insufficiently addressed by the Programme, given the 
magnitude of the issues at stake in these areas. These are concrete areas where EU-level 
action  could  further  increase  the  relevance  of  the  health  programme  to  citizens’  needs. 

104  The  Communication  of  the  Commission  on  a  comprehensive  approach  to  mental  health  has  been 

adopted on 7 June 2023 – COM(2023) 298 final 

60 

 
 
 
Progress  is  being  made  in  this  area  thanks  to  the  EU4Health  programme,  which  funds 
dedicated measures under Europe’s Beating Cancer Action Plan.  

Finally,  the  Programme  addressed  health  inequalities,  which  while  not  a  specific 
objective of the Programme, were to be addressed across all thematic priorities. The box 
below presents relevant findings of the case study on this topic. 

The Programme has supported the following six main actions:  

 
 

 

 

 

the Joint Action Health Equity Europe, (see 3.5.(i)); 
the project AHEAD which aims to address the challenge of medical deserts and 
medical desertification in Europe to help reduce health inequalities; 
the European network to reduce vulnerabilities in health which aimed to bring 
together  NGOs  and  academic  partners  from  different  European  countries  and  to 
contribute to the reduction of EU-wide health inequalities and to better equipped 
health systems to deal with vulnerability factors; 
the project Mig-HealthCare, which aimed to promote effective community-based 
care  models  to  improve  physical  and  mental  health  care  services,  support  the 
inclusion and participation of migrants and refugees in Europe and reduce health 
inequalities; 
the  project  MyHealth,  which  aimed  to  improve  the  healthcare  access  of 
vulnerable  immigrants  and  refugees  newly  arrived  in  Europe  and  focused  on 
women and unaccompanied minors; 

  SH-CAPAC,  which  aimed  to  support  Member  States  in  coordinating,  assessing 
and  planning  their  public  health  response  to  the  challenges  posed  by  migratory 
pressure. 

Despite these positive results and the significant resources invested by the Programme on 
this  policy  area,  overall,  the  theme  of  health  inequalities  is  not  perceived  by  consulted 
stakeholders as sufficiently addressed by the Programme. This might be partly explained 
by the fact that reducing health inequalities was a general, cross-cutting objective of the 
Programme, rather than a specific objective or thematic priority. Therefore, stakeholders 
might be less aware of the role the Programme played in addressing health inequalities. 

The  positive  results  of  the  funded  actions  in  terms  of  increased  cooperation  and 
coordination between different actors, improved knowledge and exchange, can contribute 
in  the long-term  to  building  capacity and  creating infrastructures able to  address  health 
inequalities and social determinants of health. 

CONSEQUENCE OF THE COVID-19 PANDEMIC AND FUTURE HEALTH NEEDS IN THE EU 

Despite the adaptability and flexibility of the Programme described above,  the COVID-
19 pandemic has demonstrated that health systems need to be ready to provide state-of-
the-art  services  and  care  and  to  be  prepared  to  cope  with  epidemics  and  other 
unforeseeable challenges. 

The  pandemic  showed  the  importance  of  testing,  contact  tracing  and  public  health 
surveillance as  essential aspects  of preparedness  and response.  It  also  demonstrated the 
need for a smooth functioning single market and access to personal protective equipment, 

61 

 
 
 
 
 
medicines,  and  medical  devices.  Another  key  lesson  learned  was  that,  while  physical 
infrastructure  could  be  expanded,  the  most  pressing  need  became  the  availability  of 
healthcare  staff  that  were  competent  within  intensive  care  units.  In  some  instances, 
healthcare  service  reservists  and  medical  students  were  called  in  and,  where  needed, 
rapid training was provided. 

Overall, the crisis demonstrated the fragility of national and local health systems and the 
need for strong and coordinated action at EU level to complement Member State health 
policies  to  improve  the  health  of  people  across  the  EU  and  ensure  that  public  health 
protection is taken into account in all EU policies.  

These  new  challenges  and  health  needs  were  fully  taken  into  account  and  addressed  in 
the design of the EU4Health programme105 (2021-2027) which succeeds the Programme. 

The EU4Health programme, adopted as a response to the COVID-19 pandemic, is one of 
the main instruments paving the way to a stronger European Health Union. It has a total 
budget  of  around  EUR  5.3  billion  over  the  whole  2021-2027  multiannual  financial 
framework. 

5.  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? 

5.1.  Conclusions 

Effectiveness 

The Programme produced an array of positive effects during its implementation.  
Knowledge produced by the Programme was used in policymaking, helping to improve 
health  and  healthcare  policies  across  the  EU.  This  was  especially  the  case  for  cancer, 
AMR,  HTA,  alcohol  and  tobacco  control.  The  Programme  also  helped  improve  health 
and healthcare through the implementation of best practices,  the  coordination of efforts 
across Member States and changes to policy and practices at EU level. For example, joint 
actions  as  a  funding  mechanism  enabled  important  collaboration,  fostered  coordination 
efforts  among  Member  States,  facilitated  the  sharing  of  good  practices  and  capacity 
building on a number of pertinent topics. Similarly, the  setting up of 24 ERNs allowed 
for  a  high  level  of  coordination  between  healthcare  providers  across  Europe  to  tackle 
complex or rare diseases conditions and will also allow to boost clinical research on rare 
diseases in the long run.  

105  Regulation  (EU)  2021/522  of  the  European  Parliament  and  of  the  Council  of  24 March  2021 
establishing a Programme for the Union’s action in the field of health (‘EU4Health Programme’) for the 
period 2021-2027, and repealing Regulation (EU) No 282/2014 (OJ L 107, 26.3.2021, p. 1–29) 
see also:  EU4Health programme 2021-2027 – a vision for a healthier European Union (europa.eu) 

62 

 
 
 
 
Funded  actions contributed to  achieving  the  Programme  objectives to  a good extent, in 
particular  for  objective  1  (promote  health,  prevent  diseases,  and  foster  supportive 
environments  for  healthy  lifestyles),  objective  3  (increase  of  health  inequalities 
between/within  Member  States)  and  objective  4  (facilitate  access  to  better  and  safer 
healthcare for EU citizens). 

This  is  evidenced  by  the  overall  positive  trend  of  the  indicators  associated  with  the 
specific  objectives  of  the  Programme,  even  if  these  indicators  did  not  cover 
comprehensively all funded actions under the 23 thematic priorities of the Programme. 

The available data shows that the most effective funding mechanisms were joint actions 
and project grants. According to the evaluation results, factors hindering the achievement 
of  the  Programme  objectives  were  limited  resources,  limited  capacity,  insufficient 
political  will  at  national  level,  difficulties  in  engaging  with  stakeholders.  There  is 
therefore  room  for  strengthened  and  more  effective  EU  action  to  address  those 
limitations and further support Member States. 

As regards objective 2 (protect EU citizens from serious cross-border health threats), the 
relevant  indicator  (‘number  of  Member  States  integrating  coherent  approaches  in  the 
design of their preparedness plans’) showed an increasing trend, indicating the ability of 
the  Programme  to  effectively  meet  this  objective.  Complementary  qualitative  analyses 
confirmed that funded actions contributed to strengthen the preparedness and response of 
the EU and Member States, as they contributed (in coordination with other tools set up in 
DG  SANTE  and  in  the  Commission)  to  respond  to  and  mitigate  the  moderate  health 
threats106 that frequently emerged during the implementation period. 
Despite active contribution of the Programme funded actions to combatting the COVID-
19  pandemic,  particularly  in  its  early  stages,  the  pandemic  uncovered  weaknesses  on 
several aspects of the preparedness and response to a major cross-border health threats at 
EU  and  at  Member  States  level.  It  can  therefore  be  concluded  that  the  objective  2  was 
met only partially or to a moderate extent by the Programme. 

The  exceptional  utility  criteria  intended  to  facilitate  higher  participation  of  low-GNI 
countries  in  the  Programme;  and  stakeholders  did  perceive  the  criteria  as  having  a 
positive  impact.  However,  low-GNI  countries  had  a  lower  overall  participation  rate  in 
Programme  actions  as  coordinators  and  partners  when  compared  with  high-GNI 
countries. Furthermore, programme participation by low-GNI countries did not increase 
over time.  

An  important  measure  of  effectiveness  concerned  the  sharing  and  dissemination  of 
Programme results. The evaluation conclusions suggest that the Programme results have, 
to varying extents, been published by the Commission services and by other stakeholders 

106  among  which:  the  avian  influenza  H7N9  in  human  beings  (2013-2014);  resurgence  of  polio  and 
tuberculosis in certain countries; hepatitis; Ebola outbreak in west Africa and later in Democratic Republic 
of  Congo  –  DRC;  Zika  virus  outbreak;  influenza;  measles;  dengue;  yellow  fever,  West  Nile  fever, 
chikungunya viruses. 

63 

 
 
 
in scientific journals, and that publications resulting from the Programme have been used 
by stakeholders. However, it emerged that Programme beneficiaries faced difficulties in 
publishing and disseminating the results of funded actions.  There is therefore scope for 
the Commission to provide further support to the dissemination of Programme results by 
way of organising knowledge transfer activities.  

The  midterm  evaluation  was  an  important  milestone  in  the  implementation  of  the 
Programme.  The  Commission,  and  notably  DG  SANTE  and  the  executive  agency, 
CHAFEA  have  taken  steps  to  address  its  10  recommendations.  Evidence  suggests  that 
some  of  the  recommendations  have  been  addressed  successfully.  Conversely,  some 
recommendations  were  not  sufficiently  taken  up,  including  spelling  out  how  actions 
targeting  health  promotion  and  health  systems  should  generate  EU  added  value  and 
investing  in  the  resources  necessary  to  improve  systems  for  monitoring  Programme 
implementation.  The  latter  recommendations,  alongside  with  those  which  were  only 
partially met, should be followed up under the new EU4Health programme (and beyond). 

Lastly, the results of the Programme were found to be sustainable overall, with HTA, the 
Joint Action on AMR and ERNs identified as areas of high sustainability. Sustainability 
was  supported  by  the  addition  of  an  obligatory  work  package  on  sustainability  in  the 
funded  actions’  workplans,  as  well  as  through  strong  connections  built  between  key 
stakeholders jointly working on a funded action or through adoption of legislation (or a 
similar  permanent  framework)  resulting  from  the  funded  action.  Challenges  to 
sustainability  were  found  to  be  linked  with  a  lack  of  political  will  in  participating 
countries  or  beneficiaries,  insufficient  integration  of  actions’  results  in  national/local 
policymaking,  insufficient  synergy  and  complementarity  with  other  funding  sources 
undermining the scaling up and further development of the results of an action. 

Efficiency 

The  data  assessed  shows  that  the  Programme  was  relatively  cost-effective  considering 
changes in the health landscape over its implementation period, and the size and scope of 
actions undertaken. In particular, the Programme’s operational costs (administrative costs 
plus the costs allocated to the functioning of the executive agency CHAFEA) were found 
to  be  reasonable  and  comparable  to  those  of  other  EU  programme  of  similar  size. 
Consulted  stakeholders  highlighted  the  positive  impacts  of  work  achieved  with  the 
resources allocated, even in cases where funding was not deemed to be wholly sufficient. 
Flexibility of funding allocation was particularly efficient and underlines a strong success 
factor of the Programme as a whole.  

An  analysis  of  the  total  implementation  costs107  of  actions  funded  under  the  4  specific 
objectives  and  on  cross-cutting  issues,  against  the  qualitative  outcome  of  these  actions 
enables to conclude that the total costs were proportionate to the results and outcomes of 

107 i.e., the total costs incurred by the Commission, Programme beneficiaries, Member States and relevant 
stakeholders. 

64 

 
 
 
the funded-actions, even if the latter are not expressed in monetary terms. The benefits of 
the funded actions potentially outweigh to a significant extent their total implementation 
costs. This potential is further enhanced if longer-term effects108 of the actions are taken 
into account. 

While  operational  and  management  costs  were  reasonable,  administrative  costs  were 
considered by stakeholders as sometimes disproportionately heavy, increasing workload 
of  those  involved  in  actions  and  potentially  putting  countries  with  low  GNI  or  smaller 
organisations off becoming involved, or being involved in future work. 

The distribution of Programme credits among the four specific objectives was efficient in 
that it addressed the key health needs identified during the implementation period, with 
funding  allocation  deemed  critical  to  achieve  expected  results.  A  particular  strength  of 
the Programme was the flexibility of funding allocation, which for example allowed the 
Programme to respond to key emerging health threats.  

There were significant differences in costs and benefits between participating countries, 
as countries with lower GNI were less able to participate in the Programme (especially in 
coordinating roles) and Western European countries led the most actions and received the 
most funding for actions. Accordingly, countries with less capacity and funding did not 
feel  the  same  benefits  as  other  countries.  Although  the  exceptional  utility  criteria 
intended to increase the participation of low GNI countries, differences in capacity still 
prevented  these  countries  from  participating  more  fully.  Further  support  to  these 
countries may be needed in order to improve the situation. 

Over the implementation period, significant efforts were made to improve the efficiency 
of  the  Programme  by  simplifying  and  streamlining  its  procedures.  This  included  the 
introduction  of  electronic  monitoring  and  reporting  mechanisms  via  the  IT  tools 
SYGMA109  and  COMPASS110,  which  were  also  used  for  the  Horizon  2020  programme. 
However,  according  to  certain  beneficiaries,  there  was  scope  to  further  simplify  the 
processes, especially for applications for funding, monitoring and reporting.  

The  cost-effectiveness  of  actions  could  have  been  improved  by  introducing  a  more 
centralised information system (either using systems already in place in the Programme 
portal  or  adding  a  new  one)  dedicated  to  disseminating  information  about  different 
funding possibilities to ensure synergies across projects, to better disseminate the results 
of implemented  actions,  to  coordinate projects and to  allow  direct  communication with 
project officers of the executive agency CHAFEA or of DG SANTE.  

108 e.g., establishment of lasting knowledge/expertise sharing networks; contribution to progressive changes 
in population behaviour, through the implementation of health promotion and disease prevention actions  
109 European Commission’ IT-based grant management system 
110 European Commission’s IT-based workflow system for the Horizon  2020 Framework Programme for 
Research and Innovation and for other EU Programmes of the research family, including the Third Health 
Programme 2014-2020 

65 

 
 
 
Coherence 

The actions funded by the Programme were aligned with its objectives and coherent with 
each  other.  Funded  actions  were  found  to  be  focused  on  thematic  priorities  while  also 
complementing  and  building  on  one  another,  thereby  demonstrating  high  internal 
coherence.  

Overall, the Programme encouraged cooperation and was aligned with other instruments 
financing  health-related  activities,  in  particular  the  European  Structural  and  Investment 
Funds and Horizon 2020111. Moreover, actions funded by the Programme contributed to 
the EU’s wider policies and priorities (i.e. the Europe 2020 strategy for smart, sustainable 
and  inclusive  growth  in  2014-2015,  the  Juncker  Commission’s  priorities  in  2016-2019 
and  the  von  der  Leyen  Commission’s  priorities  in  2020),  and  were  aligned  with  wider 
international  obligations,  in  particular  the  WHO  common  policy  framework  ‘Health 
2020’  and  the  European  action  plan  for  strengthening  public  health  capacities  and 
services. Lastly, the Programme was coherent with other health-related EU policies and 
was largely aligned with Member States’ relevant strategies and initiatives.  

EU-added value 

The  Programme  provided  added  value  in  comparison  with  what  could  have  been 
achieved  by  the  EU  without  the  Programme  and  by  Member  States  acting  alone.  In 
particular,  it  funded  multiple  actions  which  demonstrated  strong  EU  added  value  by 
encouraging  Member  States  to  exchange  best  practices  and  cooperate  and  coordinate 
with  each other on pertinent  policy issues. Furthermore,  it enabled mutual  learning and 
knowledge  exchange  in  areas  such  as  health  promotion,  HTA,  rare  diseases,  AMR, 
alcohol and tobacco policy. 

The seven added value criteria were well-defined and used in the assessment of funding 
proposals. A significant proportion of stakeholders were not aware of the extent to which 
the criteria were defined or used, suggesting that there is scope for making the process of 
integrating the EU added value criteria into proposals, clearer and more systematic. The 
criteria  that  were  considered  the  most  important  were  ‘sharing  of  best  practices’  and 
‘supporting networks for mutual learning’, which corresponds to some of the areas where 
the  Programme-funded  actions  provided  stronger  EU  added  value.  Finally,  the  EU-
added-value  criteria  remained  relevant  throughout  the  Programme’s  implementation 
period  and  are  considered  useful  for  developing  future  health  programmes  and 
establishing health policies and priorities at EU-level. 

111 EU Framework Programme for Research and Innovation 2014-2020. 

66 

 
 
 
 
Relevance 

During the third Health Programme (2014-2020), the main health needs identified across 
the EU related to health promotion and better and safer healthcare. However, there were 
also additional new and emerging needs related to health systems and health and social 
inequalities.  

An  analysis  of  participating  countries’  priorities  and  action  plans  indicates  that  the 
Programme  was  relevant  in  that  it  addressed  the  national  health  needs.  This  was 
particularly  the  case  for  objectives  1  (promote  health,  prevent  diseases  and  foster 
supportive environments for healthy lifestyles) and 4 (facilitate access to better and safer 
healthcare  for  EU  citizens).  The  involvement  of  participating  countries  in  the  design 
phase was instrumental in ensuring the Programme’s relevance. 

The Programme has for the most part remained relevant to changes in health needs over 
time,  and  was  flexible  enough  to  respond  to  emerging  health  needs  such  as  the 
migrant/refugee  crisis  in  2015  and  the  COVID-19  pandemic  in  2020,  even  if  this 
pandemic  showed,  at  EU  level  and  in  the  Member  States,  weaknesses  in  preparedness 
plans and in the ability to respond to a major health threat. 

A number of factors limited the relevance of the Programme. For example, stakeholders 
felt  that  higher  budgets  could  have  helped  address  the  problems  better  (e.g.  health 
inequalities,  mental  health,  and  child  and  infant  health,  including  paediatric  cancers). 
There were, however, funded actions that covered mental health and health inequalities, 
even if these themes were not always perceived by surveyed stakeholders as adequately 
addressed. This is likely because they were not named as specific thematic priorities, so 
they did not receive proper emphasis. 

The  Programme  was  also  relevant  in  that  funded  actions  were  clearly  aligned  with  its 
specific thematic priorities. The funded actions were also aligned with the Commission’s 
wider priorities, which enabled them to respond to the needs of EU citizens, in particular 
in  areas  such  as  the  reduction  of  alcohol  consumption,  tobacco  control,  rare  diseases, 
cancer screening, and implementation of best practices in nutrition, physical activity and 
the prevention of chronic diseases. 

During  the  Programme  implementation  period,  the  relevance  of  preparedness  and 
response to  health threats significantly increased  because of infectious diseases  and the 
moderate (or less severe) outbreaks112 that originated either within the EU territory or in 
third countries. With the COVID-19 outbreak at the beginning of 2020 the protection of 
EU citizens from serious cross-border health threats became the highest health priority. 

112 e.g.,  avian influenza H7N9 in human beings; resurgence of polio and tuberculosis; measles; hepatitis; 
Ebola outbreak in west Africa and in Democratic Republic of Congo; Zika virus; influenza; dengue; yellow 
fever, West Nile fever, chikungunya viruses. 

67 

 
 
 
The  COVID-19  pandemic  demonstrated  the  need  for  the  health  systems  to  be  ready  to 
provide state-of-the-art services and care and to be prepared to cope with epidemics and 
other unforeseeable challenges. Lessons learned from the pandemic in terms of emerging 
challenges  and  new  health  needs  were  considered  and  addressed  in  the  design  of  the 
EU4Health programme (2021-2027) which succeeded the third health programme (2014-
2020). 

5.2.  Lessons learned 

Conclusions and recommendations 

This  evaluation  assessed  the  Programme’s  effectiveness,  efficiency,  coherence,  EU 
added value and relevance so as to learn lessons and pave the way for future EU action in 
health.  The  Programme  has  been  largely  relevant  in  addressing  the  health  needs 
expressed by European countries and citizens over the period of its implementation and 
has  adapted  to  changes  in  health  needs  over  time,  being  flexible  enough  to  respond  to 
emerging health needs such as the migrant/refugee crisis and the COVID-19 pandemic.  

It  has been effective in  achieving  its objectives, as  shown by the  progress made on the 
indicators designed to measure the Programme’s performance. It has also been effective 
in  enabling  more  cooperation  and  coordination  among  Member  States  and  overall 
improvements  in  health  policy  developments  across  the  EU.  Its’  funded  actions 
demonstrated  added  value  and  created  synergies  with  other  national,  EU  and 
international  policies  addressing  health.  Despite  its  overall  success,  there  have  been 
limitations to  what  the  Programme  could  have  achieved, as  outlined above. To  address 
those  limitations  and  ensure  that  EU  action  in  health  is  fit  for  the  complex  and  ever-
evolving  health  landscape,  this  evaluation  sets  out  recommendations  for  EU  action 
structured under four headings, summarised below. 

Further building on the midterm evaluation recommendations 

Building on the midterm recommendations, continued focus should be placed on areas of 
EU added value that have clearly emerged from this analysis.  

  Future EU action in the area of health should continue to encourage cooperation 
and  coordination  among  Member  States  in  areas  such  as  preparedness  and 
response  to  cross-border  health  threats;  rare  diseases,  HTA  and  eHealth,  while 
also fostering exchanges and implementation of best practices in health promotion 
and  disease  prevention,  in  particular  sub-themes  that  have  become  increasingly 
important  (such  as  cardiovascular  diseases,  diabetes,  mental  health,  cancer, 
physical activity, alcohol, tobacco control and nutrition).  

Design of Programme and funding frameworks 

Improving  the  outcomes  and  impacts  of  funded  actions  begins  at  the  design  stage.  A 
number of key findings on the main evaluation criteria suggest that improvements could 
be made to the Programme’s funding structures.  

68 

 
 
  Rethinking  how  cross-cutting  policy 

issues  can  be 

the 
Programme’s priority areas. Any significant health needs should be given explicit 
attention  and  funding,  rather  than  being  included  as  a  ‘cross-cutting  issue’.  For 
example,  although  actions  were funded  on mental  health and health inequalities, 
stakeholders did not always perceive these topics to be adequately addressed. This 
was  probably  because  they  were  not  named  as  specific  thematic  priorities,  even 
though  at  certain  moments  during  the  Programme’s  lifespan,  funding  and 
emphasis was provided. 

integrated 

into 

  The  flexibility  and  adaptability  of  the  Programme  was  one  of  its  key  strengths, 
and  this  aspect  should  continue  and  be  further  developed  to  help  face  sudden 
emergencies or changes in health needs.  

Facilitating and strengthening participation of all countries 

Having all EU Member States participate contributes to strengthening the Programme’s 
outputs,  outcomes  and  impacts.  Full  participation  also  increases  the  added  value  of 
funded actions and should remain a key factor for improvement.  

  Structures  should  be  put  in  place  to  remove  barriers  to  full  participation  by 
through  additional  support  and  relevant 

countries  with  fewer  resources 
mechanisms, schemes and incentives. 

 

Increased  EU-level  resources  dedicated  to  health  issues  could  contribute  to 
addressing national difficulties in participating in the Health Programme.  

  Furthermore,  an  even  stronger  action  at  EU  level  in  brokering  the  existing 
knowledge  and  pooling  the  existing  relevant  information  would  contribute  to 
closing the knowledge gaps, where needed, while also steering national action.  

  Support  to  the  dissemination  of  Programme  results  by  way  of  organising 
knowledge transfer activities (e.g. communities of practice, policy dialogues and 
other events) should be strengthened. 

Ensuring sustainability 

The sustainability of funded actions has a profoundly positive effect on EU and national 
health policies and systems.  

Guiding  and  actively  supporting  beneficiaries  in  conceptualising  and  implementing 
actions to foster sustainability is a key element to consider during future planning.  

Mechanisms  and  support  should  be  provided  to  ensure  sustainability  measures  are 
planned or negotiated with beneficiaries at the start of the funded actions.  

Joint actions have been particularly successful at this, due to the obligation to ensure the 
sustainability of the work and should be considered as good practice. 

69 

 
 
 
ANNEX I:   PROCEDURAL INFORMATION 

−  [Lead DG, Decide reference and, if relevant, Work Programme reference. 
−  Derogations granted and justification. 
−  Organisation and timing. 
−  Consultation of the Regulatory Scrutiny Board (if relevant).  
−  Evidence  used  together  with  sources  and  any  issues  regarding  its  quality  (i.e.  has  the 

information been quality assured?); 

−  Use of external expertise.] 

Lead DG  

The European Commission's Directorate-General (DG) for Health and Food Safety (DG SANTE) is 
the lead DG for this evaluation (PLAN/2020/9070). 

Organisation and timing  

The Commission published a roadmap on the final evaluation of the 3rd Health Programme 2014-
2020 on 20 October 2020 and was open for feedback until 17 November 2020. 

A public consultation ran for 13 weeks from 11 March 2022 to 10 June 2022 with 68 responses 
received in total. 

An interservices steering group (ISSG) was established in September 2020 involving 
representatives from: Secretariat-General, DG AGRI, DG INTPA, DG RTD, DG EAC, ESTAT, 
DG REGIO and JRC.  

The ISSG contributed to the evaluation and ensured that it met the necessary standards for quality, 
impartiality and usefulness. Five meetings of the IISG were held. 

Exceptions to the Better Regulation Guidelines  

None. This evaluation was not selected for scrutiny by the Regulatory Scrutiny Board. 

Evidence, sources and quality  

This evaluation report drew on the following sources of evidence: 

•  Study supporting the final evaluation of the 3rd Health Programme 2014-2020 
•  Study supporting the mid-term evaluation of the 3rd Health Programme 2014-2020 
•  Study supporting the final evaluation of the 2nd Health Programme 2008-2013 
•  Data-gathering  study  on  the  common  financial  framework  for  the  management  of 

expenditure under Regulation (EU) No. 282/2014 

•  Annual Implementation Reports, of the 3rd Health Programme for the years 2014 to 2020 

70 

 
 
 
•  Report from the Commission to the European Parliament and the Council on the operation 
of Directive 2011/24/EU on the application of patients' rights in cross-border healthcare and 
the staff working document (SWD) accompanying this report 

•  Study of the European Parliament 2019, on the Benefit of EU action in health policy: The 

record to date  

•  Special report of the European Court of Auditors, 2019 addressing antimicrobial resistance: 

progress in the animal sector, but this health threat remains a challenge for the EU 

•  Special  report  of  the  European  Court  of  Auditors,  2019:  EU  actions  for  cross-border 

healthcare: significant ambitions but improved management required.  

•  Health for the EU:  a selection of  actions funded under the Third EU Health Programme  – 

Special Edition for the EU Health Programme Conference 30 September 2019 

•  Submissions  to  the  online  public  consultation  from  March  to  June  2021  and  the  factual 
summary report of these as well as the synopsis report of all consultation activities in Annex 
V 

•  Submissions to the targeted stakeholder survey, and the factual summary report of these, as 

well as the synopsis report of all consultation activities in Annex V 

•  Submissions  to  the  focus  groups  conducted  for  each  main  type  of  funding  instruments 
implemented by the Programme (actions grants, joint actions, operating grants, procurement 
contracts, direct grants to international organisations), and factual summary reports of these, 
as well as the synopsis report of all consultation activities in Annex V 

•  Results of stakeholder interviews, and case studies 

71 

 
 
ANNEX II. METHODOLOGY AND ANALYTICAL MODELS USED 

In order to assess the Programme according to the 22 evaluation questions, a three-phase approach 
was devised: 

starting with an in-depth understanding of the programme and its implementation 
then consultation with stakeholders 

- 
- 
-  analysis and synthesis 

1.  Building an understanding of the Programme 

The first phase of the study involved gathering contextual information on the background, structure, 
functioning  and  implementation  of  the  Programme.  The  steps  taken  under  this  phase  are  outlined 
below. 

  Searching  and  checking  data  availability  and  making  and  overview  of  the  functioning  of 

Programme  

  Identification of relevant stakeholders within seven stakeholder groups,113 to be consulted 
with in later phases of the study. Relevant stakeholder groups were identified for this study 
to ensure a varied and robust response to the evaluation questions and full understanding of 
the programme. A stakeholder engagement strategy was prepared to detail how stakeholders 
were to be involved in the evaluation.  

  An analysis of the Programme database was carried out. Which enabled the development 
of  a  single  output  database  containing  relevant  collated  information  on  the  main  funded 
actions,  participating  entities,  and  the  geographical  and  temporal  scope  of  actions  falling 
within  this  evaluation,  in  the  form  of  all  publicly  available  information  about  all  funded 
actions in the database.  

  In-depth review of 61  preliminary documents114 relating  the context, legal  and financial 

basis of Programme. 

  Main  document  review  covering  documents  that  confirm  the  implementation  status  of 
Programme  activities,  and  strategic  documents  which  shed  light  on  the  evolution  of  the 
Programme  to  evolving  needs  and  priorities.  Through  this  review,  the  national  health 
strategies and plans, as well as specific health strategies such as HIV/AIDS action plans, of 
all countries in the scope of the present study were reviewed and the priorities were extracted 
and mapped to the objectives of the Programme. 

113   Public  authorities  (central  government/ministries  of  health,  and  public  health  authorities  or  agencies); 
Academic/research  organisations;  Non-governmental  organisations;  EU  citizens;  Patients  and  service  users  and 
organisations  representing  them;  Consumer  organisations;  Company/business  organisations;  Other  (international 
organisations  e.g.  WHO,  OECD;  Healthcare  service  providers;  Organisations  presenting  healthcare  service 
providers; Healthcare professionals’ associations; Independent experts) 

114   Consisting  of  previous  evaluations  of  the  European  Commissions’  health  programmes,  as  well  as  relevant  EU 

health strategy documents and legal texts on the functioning of the health programmes 

72 

 
 
 
 
 
 
 
  Five  scoping  interviews  were  carried  out  within  DG  SANTE  and  with  Programme 
Committee  members,  to  better  understand  differing  perspectives  of  Programme,  including 
successes and gaps, administrative issues, and the varied funding mechanisms 

  An in-depth analysis of a sample of 18 funded actions relating to the case study topics115 
was  carried  out.  Relevant  documents  were  reviewed,  and  discussions  were  held  with  key 
stakeholders within DG SANTE and HaDEA to inform the development of the case studies. 

2.  Consulting with stakeholders 

In-depth  stakeholder  consultations  were  carried  out  over  the  course  of  the  study:  an  Public 
consultation,  a  targeted  survey,  interviews,  focus  groups  and  social  media  listening  as  elaborated 
below.  

  The Public consultation (PC)  (11 March to 10 June 2022) provided the general public, and 
all  interested  parties,  with  the  opportunity  to  provide  information  and  opinions  on  the 
matters to be addressed in this study. The PC was targeted at all those who have an interest 
in  the  3rd  Health  Programme  but  who  had  not  necessarily  been  directly  involved  in  the 
Programme  design  or  implementation.  Questions  were  therefore  relatively  high-level, 
exploring the overall perception of the Programme, and its relevance to broader health needs 
and objectives. A total of 69 responses were received.116  

  The targeted stakeholder survey (10 March to 13 May 2022) collected further evidence on 
the  views  and  perceptions  of  those  with  direct  experience  of  the  Programme  regarding  its 
relevance  and  implementation  and  performance.  The  survey  was  targeted  at  all  those  who 
have  been  directly  involved  in  the  Programme  design  and/or  implementation  (including 
those having received funding from the Programme) and who were therefore able to answer 
relatively  specific  and  more  detailed  questions  on  the  implementation  and  performance  of 
the Programme. 

  34 Stakeholder interviews were carried out from April to July 2022 to help the study team 
to  understand in  more depth  the design  and implementation  of the  Programme. They  were 
also  used  to  triangulate  findings  drawn  from  other  data  collection  tasks  and  to  fill  gaps  in 
evidence collected through other tasks.  

  Five online focus groups were conducted (May to June 2022) to gain further insights into the 
main funding mechanisms of the Programme117. Between three and 10 stakeholders took part 
in each focus group, which lasted for up to 4 hours.  

  Social media listening118 was conducted to extract data from Twitter between July 2020 – 
July 2022 to understand coverage and trends of discussions on the six case study topics of 
the study.119 

115 Alcohol, Antimicrobial resistance (AMR), health inequalities, nutrition, Health Technology Assessment (HTA), and 
vaccination; 
116 Three responses were identical (including responses to open-ended questions), and so they have been considered as 
one response. The analysis therefore focused on 67 responses; 
117  Project  Grants,  Operating  Grants,  Joint  Actions,  Procurement  contracts,  and  a  final  focus  group  on  all  funding 
mechanisms. Due to a lack of participation and availability of DG SANTE and HaDEA staff, the fifth focus group was 
ended early, and follow-up interviews were scheduled instead. 
118 Social media listening is the process of tracking social media platforms for mentions and conversations related to a 
topic, then analysing these for insights. 
119 Alcohol, Antimicrobial resistance (AMR), health inequalities, nutrition, Health Technology Assessment (HTA), and 

vaccinations 

73 

 
 
 
3.   Analysis and synthesis 

Once  all  data  from  desk  research  and  consultations  with  stakeholders  was  gathered,  in-depth 
analysis and synthesis was undertaken, detailed below.  

The  purpose  of  the  analysis  and  synthesis  phase  was  to  draw  together  the  data  sources  generated 
from the study, to allow for the identification of patterns, divergences and convergences in findings 
per  evaluation  criteria.  Data  sources  were  analysed  separately  to  identify  key  findings  per 
evaluation question. The findings were compared per evaluation question across the study activities, 
noting  divergence  and  convergence  of  evidence  and  accounting  for  differences  in  views  per 
stakeholder group. In preparation for analysis, data was organised into useable formats, e.g., writing 
up  interview  notes  and  focus  group  notes,  cleaning  and  organising  PC  and  targeted  survey  data 
files. Then, data was analysed as below.  

Qualitative data analysis 

The  following  steps  were  carried  out  to  utilise  qualitative  data  gathered  through  the  document 
review and stakeholder consultations: 

  Key  findings  were  drawn  from  the  document  review  to  provide  documentary  evidence 

relating to each relevant evaluation question.  

  Relating  to  the  funded  actions  database,  qualitative  information  including  abstract,  priority 

area, and coordinator, was analysed related to each relevant evaluation question.  

  Open-ended questions from the PC and targeted stakeholder survey were manually reviewed 

and coded for key themes. 

  The  notes  from  the  focus  groups  were  reviewed  and  key  findings  were  summarised  by 

evaluation criteria.  

  The  notes  from  the  interviews  were  reviewed  and  coded  into  a  master  file  showing  key 
issues by stakeholder group. This was then reviewed by evaluation criteria and trends were 
summarised into the final report. 

Quantitative data analysis  

The  following  steps  were  carried  out  to  utilise  quantitative  data  gathered  through  the  document 
review and stakeholder consultations: 

  Responses  to  close-ended  questions  within  the  PC  and  targeted  stakeholder  survey  were 
processed  using  univariate  analysis  (proportions,  averages),  disaggregated  by  question  and 
key  variables.  Responses  were  also  processed  using  bivariate  analysis,  including  cross-
tabulations. 

  Tables  and  graphs  of  key  points  were  created  for  the  social  media  listening  and  further 

explanatory text was drafted to provide insights into these findings. 

Case studies 

The  case  studies  provide  a  deep  dive  on  a  specific  theme  within  Programme.  The  approach  of 
contribution  analysis120  was  used  to  enable  the  identification  of  concrete  links  between  thematic 
objectives and their specific outcomes and impacts. The level of contribution from the Programme 

120  Contribution  analysis  involves  unpacking  the  intervention  logic  for  specific  activities  of  Programme,  isolating  the 
hypothesis  (or  hypotheses)  underpinning  the  various  steps  involved  –  e.g.,  from  outputs  to  outcomes,  or  from 
outcomes to impacts – and exploring to which extent the evidence available supports the hypothesis. 

74 

 
 
 
at each of these steps was considered based on a thorough review of the evidence; as well as other 
contributing factors in influencing the outcomes.  

The  case  studies  were  used  to  provide  evidence  to  answer  certain  questions,  related  to  the 
effectiveness of the Programme. Additionally, findings from the case studies were used to provide 
evidence to answer other evaluation questions as needed.  

Limitations and robustness of study findings 

Key  strengths  of  the  evaluation  include  the  identification  of  links  between  inputs,  outcomes  and 
impacts of specific actions of the Programme, which was achieved through the use of contribution 
analysis  and  presented  in  the  case  studies  of  the  Programme.  Furthermore,  despite  challenges  in 
engaging stakeholders as outlined below, the study engaged with a rank of stakeholders from across 
the main groups identified through the study. Engagement with those involved in the management 
and  design  of  the  Programme  was  particularly  high  through  the  interview  and  focus  group 
consultations.  It  was  therefore  possible  to  corroborate  insights  from  such  relevant  stakeholder 
groups across the multiple consultations to yield reliable evidence and data to produce a thorough 
assessment  underpinning  the  evaluation.  Additionally,  the  extensive  document  review  provided  a 
solid basis for the study and generated key line of enquiry to be investigated through the stakeholder 
consultations.  

However, some limitations related to the fact that the evaluation study was developed mainly using 
the publicly available database and limited participation from stakeholders.  

The Health Programme database121, contained only actions funded through grant agreements and did 
not include comprehensive information such as detailed financial data and outputs of the actions. 

Thus, the analysis of the database may be limited or may not be representative due to the types of 
funded actions which are included. For instance, half of the actions included were half a year long, 
and over one third of them were still ongoing when finalising this evaluation report. As not all of 
the actions of the Programme are included in the database, this means the analysis may be biased to 
include more short-term actions and not consider longer ones. However, through the consultations 
undertaken as part of this study, stakeholders were able to provide feedback on all ongoing actions 
as well as completed actions, regardless of their implementation maturity, as long as they fell under 
Programme funding.  

This shortcoming and related data gaps were mitigated by the provision of separate complementary 
documents  and  specific  data,  such  as  the  list  of  all  procurement  contracts  concluded  under  the 
Programme.  For  the  operating  grants  awarded  to  Non-governmental  Organisations  the  final 
technical  reports  were  made  available  to  the  contractor,  as  well  as  the  final  reports  and  detailed 
technical deliverables and outputs of a sample of actions funded by the Programme.  

These  measures,  combined  with  triangulation  of  data  from  different  sources,  enabled  to  arrive  at 
evidence-based results confirmed by more than one information source. 

121 Health Programme DataBase - European Commission (europa.eu) 

75 

 
 
 
Moreover,  due  to  the  timing  of  the  study  (being  undertaken  after  the  commencement  of  the 
EU4Health Programme) and the number of other public-health priorities being faced by all relevant 
stakeholder  groups  during  the  study  period  (COVID-19,  war  in  Ukraine),  the  stakeholders’ 
engagement was limited.  

The online targeted survey and  PC yielded fewer replies than anticipated, despite a dissemination 
campaign  and  several  targeted  reminder  emails.  This  may  be  due  to  a  lack  of  engagement  by 
stakeholders and other contextual factors (including delays to the overall study timeline, the study 
being run after the launch of the new health programme- and thus a risk of de-prioritisation of the 
previous programme). A larger number of survey responses would have provided greater depth to 
the  qualitative  analysis,  however  an  equal  representative  response  from  all  relevant  stakeholders 
was received and allowed for an assessment to be made. 

Similarly,  for  the  stakeholder  interviews,  despite  multiple  communications  to  stakeholders,  the 
targets  per  stakeholder  group  were  not  fully  met  for  two  groups:  ‘government  policymakers’  and 
‘healthcare service providers and organisations representing them’. While the target was to have 45 
participants  in  the  interviews,  despite  substantial  attempts  to  engage  with  stakeholders,  34 
stakeholders participated in total.  

76 

 
 
1 
1a 

1b 

ANNEX III. EVALUATION MATRIX  

Evaluation questions 

Judgment criteria 

Quantitative and qualitative indicators 

Research methods 

Sources for document review 

Relevance 
To what extent have the Programme’s scope, including its objectives and priorities been relevant to health needs across the EU, considering their evolution over the evaluation period? 
To what extent did the objectives 
and priorities of the Programme, 
its actions and other activities, 
address health and healthcare 
needs and problems at EU-level 
over the evaluation period?  

• 

• 

• 

• 

• 

• 

• 

Programme Implementation 
documentation 
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme 

To what extent have each of the 
objectives and priorities remained 
relevant to health and healthcare 
needs and problems at EU-level 
over time and up until 2020? 

• 

• 

• 

• 

• 

• 

Problems, needs and their 
drivers identified as part of 
the Programme were 
correctly defined. 
No relevant problems or 
needs were left out of the 
Programme at the time. 
The implementation mode 
of the Programme was 
relevant given needs and 
context at the time. 

The problems and drivers 
that led to the choice of the 
Programme’s objectives are 
still relevant. 
Each of the objectives and 
actions have remained 
relevant considering 
changes in:  
science and technological 
progress in the area of 
health and healthcare 
solutions developed at 
national level, by public, 
private and not-for-profit 
actors  
changes in prevalence & 
severity of NCDs & CDs  

[Quantitative] A majority of 
stakeholders agree that 
problems/needs/drivers were 
correctly defined 
[Qualitative] Expert 
stakeholders’ recollection of 
problems at the time 
[Qualitative] Available 
literature from 2014-2020 
reflects the 
problems/needs/drivers of the 
Programme 

[Quantitative] A majority of 
stakeholders agree that 
problems/needs/drivers remain 
relevant defined 
[Qualitative] Expert views on 
extent to which there is still a 
need to focus on each of the 
Programme’s priority areas 
[Qualitative] Extent to which 
Member States still require 
support in the areas identified 
by the Programme  
[Qualitative] Extent to which: 
EU knowledge needs to be 
improved and in which domains 
health officials and healthcare 
providers use new knowledge 

77 

• 

• 

• 

• 

• 
• 

• 

Document review looking 
at sources from 2014-2020 
that address problems, 
needs and drivers related 
to health and healthcare 
PC 
Targeted stakeholder 
surveys 
Stakeholder interviews  
Social media analysis 
Focus groups 

• 
• 

• 
• 
• 

• 

• 

• 

•  Mapping of project 

database 
PC 
Targeted stakeholder 
surveys 
Stakeholder interviews,  
Social media analysis 
Focus groups 

• 
• 

• 
• 
• 

Document review, 
particularly: 
review of data on 
prevalence and severity of 
NCDs & CDs 
literature reviews on the 
state of play in health & 
healthcare research & 
innovation 

• 

• 

• 

Programme Implementation 
documentation 
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  
EU-level collected data on health 
indicators to help understand the 
relevance of the Programme 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
• 

The implementation mode 
of the Programme remains 
relevant given evolving 
needs and context. 

• 

• 

developed through the 
Programme 
[Qualitative] Overview and 
assessment of health and 
healthcare solutions 
(technologies, therapies, 
products) developed between 
2014-2020 
[Quantitative & qualitative] 
Changes in prevalence, 
incidence and severity of NCDs 
and CDs 

2 
2a 

2b 

3  
3a 

To what extent were the Programme’s thematic priorities sufficiently covered by the funded actions to achieve the Programme’s objectives and Commission’s wider priorities? 
To what extent were the 
Programme’s funded actions 
aligned with the thematic 
priorities of the Programme? 

• 
•  Mapping of project 

There is a clear alignment 
between funded actions 
and the specific thematic 
priorities set out by the 
Programme. 

Document review 

• 

• 

• 

• 

• 

Programme Implementation 
documentation  

database 
Targeted stakeholder 
surveys 
Stakeholder interviews 
Focus groups 

• 
• 

To what extent were the funded 
actions aligned with the 
Commission’s wider priorities? 

• 

There is a clear alignment 
between funded actions 
and wider Commission 
priorities. 

• 

• 

• 

• 

[Qualitative] Available 
information from the 
Programme period reflects 
alignment between funded 
actions and Programme 
priorities 
[Quantitative] A majority of 
stakeholders agree that funded 
actions align with thematic 
priorities 
[Quantitative] % of total 
number of funded actions that 
align with specific Programme 
themes  
[Qualitative] Available 
information from the 
Programme period reflects 
alignment between funded 
actions and wider Commission 
priorities 
[Quantitative] A majority of 
stakeholders agree that funded 
actions align with wider 
Commission priorities 

Document review 

• 
•  Mapping of project 

database 
Targeted stakeholder 
surveys 
Stakeholder interviews, 
particularly with SANTE & 
CHAFEA and NFPs & PCs 
Focus groups 

• 

• 

• 

• 

• 

Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

How relevant is the Programme to EU citizens, and in particular, is the Health Programme close to citizens and responding to their needs? 
How relevant is the Programme 
to EU citizens? 

• 

• 

• 

Each of the objectives and 
actions have remained 
relevant considering 
changes in public/citizens’ 
expectations and 
behaviours in relation to 
health and healthcare 

[Quantitative] A majority of 
stakeholders agree that funded 
actions are relevant to 
public/EU citizens’ expectations 
and behaviours in relation to 
health and healthcare 
[Qualitative] Available 
information from the 

• 

Document review, 
particularly: review of 
existing data on 
public/citizens’ 
expectations & behaviours 
PC 
Social media analysis 

• 
• 

Strategic Documents (policies/reports) to 
understand relevance of the Programme  
EU-level collected data on health indicators to help 
understand the relevance of the Programme 

78 

 
 
 
 
 
3b 

4 
4a 

Is the Programme close to 
citizens and responding to their 
needs? 

• 

Actions funded under the 
Programme are directly 
relevant/responding to the 
needs of EU citizens 

Effectiveness 
What have been the (quantitative and qualitative) effects of the Programme? 
To what extent has the 
Programme contributed to a 
more comprehensive and uniform 
approach to health and 
healthcare in the EU? 

• 

•  Measures implemented by 
Member States are aligned 
with the Programme 
National programmes and 
actions reflect evidence and 
evidence-based approaches 
developed through 
Programme funding 
Health data is more robust, 
timely and comparable 
across EU Member States 

• 

4b 

• 

To what extent has the 
Programme contributed to 
improvements in health and 
healthcare in the EU and at 
Member State level? 

• 

• 

• 

• 

Programme actions that 
have led to new knowledge 
and evidence have been 
used in the development of 
policy and decision-making 
Programme actions have 
led to improvements in 
health and healthcare in 
the EU and at MS level in 
terms of:  
Implementation of best 
practices by MS 
Coordination of efforts 
across MS 

• 

Stakeholder interviews 

• 

• 

• 

Document review, 
particularly: review of 
existing data on 
public/citizens’ 
expectations & behaviours 
Targeted stakeholder 
survey 
Social media analysis 

Document review 

• 
•  Mapping of project 

database  
PC 
Targeted stakeholder 
surveys 
Stakeholder interviews  
Focus groups 
Case studies 

• 
• 

• 
• 
• 

Document review 

• 
•  Mapping of project 

database 
PC 
Targeted stakeholder 
surveys 
Stakeholder interviews  
Focus groups 
Case studies 

• 
• 

• 
• 
• 

Strategic Documents (policies/reports) to 
understand relevance of the Programme  
EU-level collected data on health indicators to help 
understand the relevance of the Programme 

• 

• 

• 

• 

• 

• 

Programme Implementation 
documentation 
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  
EU-level collected data on health 
indicators to help understand the 
relevance of the Programme 

Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  
EU-level collected data on health 
indicators to help understand the 
relevance of the Programme 

Programme period 
demonstrates that funded 
actions are relevant to 
public/citizens’ expectations 
and behaviours in relation to 
health and healthcare 
[Quantitative and Qualitative] 
Survey data & other research 
on public/citizens’ expectations 
& behaviours in relation to 
health & healthcare 
[Qualitative] Available 
information on the extent to 
which funded actions directly 
address the needs of citizens 

[Qualitative] Level of alignment 
or divergence between national 
level actions in relation to 
Programme priorities and 
actions 
[Qualitative] Level or degree of 
MS use of evidence and 
evidence-based approaches 
developed under the 
Programme 
[Qualitative] Extent to which 
health data is more robust, 
timely and comparable across 
EU Member States 
[Quantitative & Qualitative] 
Number and content of 
scientific studies and best 
practice, guidance, etc. 
developed as part of the 
Programme 
[Qualitative] Extent of 
implementation of best 
practices by EU MS 
[Qualitative] Coordination 
efforts by EU MS 
[Qualitative] Changes in EU 
policy and practice  
[Qualitative] Stakeholder views 

79 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

 
 
 
 
 
 
 
 
 
 
• 

Changes in policy and 
practice at EU level 

4c 

• 

To what extent has the 
Programme contributed to 
the EU’s influence on health 
and healthcare standards, 
policies and practices at 
international level? 

• 

• 

Programme outputs have 
been used at an 
international level 
The EU’s coordination with 
international bodies in the 
field of health has been 
strengthened in 
Programme priority areas 

on how studies, reports and 
evidence produced through the 
Programme contributed to 
decision making at EU or 
national level 
[Qualitative] Extent to which 
stakeholders attribute 
improvements to the 
Programme 
[Qualitative] Extent to which 
documentation corroborates 
stakeholder views on 
relationship between new 
knowledge and policy-making 
or decision-making 
[Qualitative] Extent to which 
factors other than the 
Programme can explain any 
improvements 
[Qualitative] Expert views on 
how the Programme 
contributed to the EU’s 
influence on standards, policies 
and practices at global level 
(WHO, SDGs) 
[Quantitative &Qualitative] 
extent to which documentation 
and other stakeholder 
interviews confirm expert views 

• 

• 

• 

• 

• 

Document review 

• 
•  Mapping of project 

database 
Targeted stakeholder 
surveys 
Stakeholder interviews  
Focus groups 
Case studies 

• 

• 
• 
• 

Programme Implementation 
documentation 
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

• 

• 

• 

5 
5a  

To what extent have the Programme’s objectives (general and specific) been met? To what extent can factors influencing the observed achievements be linked to the EU intervention? 
To what extent have the funded 
actions contributed to achieving 
the objectives of the Programme?   

For a subset of the actions: 

• 
• 

• 

• 

• 

• 

Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

• 

There is a clear indication 
that funded actions meet 
the Programme’s objectives 
The actions funded by the 
Programme have led to 
high-quality, publicised and 
influential outputs that 
support Programme 
objectives  

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews 

• 
•  Mapping of project 

• 
• 

database 
Focus groups 
Results of analysis under 
EQ4 on Programme effects 

[Qualitative] Available 
information from the 
Programme period 
demonstrates that funded 
actions meet the Programme’s 
objectives 
[Qualitative] Quality of outputs 
from funded actions 
[Quantitative] A majority of 
stakeholders agree that funded 
actions meet the Programme’s 
objectives 
[Quantitative &Qualitative] 
Information on publication & 
dissémination efforts  

• 

• 

• 

80 

 
 
 
5b 

6 
6a 

Regarding the objectives partially 
met or unmet, which factors 
hindered the achievement of the 
objectives?  
• 

• 

• 

The cause and effect chain 
for achieving the objectives 
was effective 
The explanatory factors 
that hinder and enable 
achieving Programme 
objectives can be identified 

• 

• 

• 

[Qualitative] Influence of 
actions on decision-making 
[Qualitative] Impact on 
achieving strategy objectives 
[Qualitative] For question 5a, 
where objectives have not been 
met, assessment of what has 
contributed to objectives not 
being met  

• 

• 

Review of evidence 
gathered in support of 
question 5a 
Focus groups 

• 

• 

Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

How effective was the introduction of "exceptional utility" criteria in the Regulation establishing the Programme in order to incentivise the participation of low GDP countries? 
To what extent did the 
‘exceptional utility’ criterion 
incentivise participation of low 
GDP countries? 
• 

• 
• 

• 

• 

• 

• 

• 

Programme Implementation 
documentation  

Programme participation 
by low GDP countries has 
increased:  
over time 
as compared to 
participation in the 2nd HP 
As compared to 
participation during the 
first half of the Programme 
period (i.e. since the mid-
term evaluation)  

• 

• 

• 

• 
Document review 
•  Mapping of programme 
data, particularly:  
Participation by low GDP 
countries/ organisations 
(number & geographic 
distribution) 
Projects (number and 
types) funded under the 
criterion in comparison to 
regular funding  
Funding allocations 
(proportions and amounts) 
to low GDP countries/ 
organisations overall and 
under the criterion 
Targeted stakeholder 
surveys 
Stakeholder interviews  

• 

• 

• 

[Quantitative] Trends in 
participation of low GDP 
countries over the Programme 
period and compared to 2HP  
[Quantitative] Success rates of 
applicants seeking to benefit 
from the criterion, and as 
compared to success rates for 
regular funding 
[Quantitative & Qualitative] 
Number of low GDP countries 
participating, levels of funding 
provided 
[Qualitative] Stakeholder views 
on extent to which low GDP 
country participation was 
incentivised by the criterion 
[Qualitative] Stakeholder views 
on any changes or 
improvements made since the 
mid-term evaluation that 
improved or might have led to 
improved participation (even if 
this is not directly reflected in 
the quantitative results) 
[Qualitative] Stakeholder views 
on any factors leading to lower 
participation by low GDP 
countries that is specific to the 
criterion and Programme in 
general (e.g. awareness / 
understanding of the criterion) 
[Qualitative] Stakeholder views 

81 

• 

• 

• 

• 

• 

 
 
 
 
 
7 

7a 

7b 

7c 

8 
8a 

8b 

on wider factors that may 
influence participation external 
to the criterion (e.g. securing 
co-financing, administrative 
capacity to manage actions in 
the MS) 

To what extent are the Programme’s actions/outcomes/results published by Commission services, by Programme beneficiaries and other stakeholders? To what extent are they made accessible to the 
international scientific and health community and to the wider public in the EU? 
To what extent are Programme 
results published? 
• 

[Quantitative & Qualitative] 
Number, type and source of 
publications 

• 

• 

• 

• 

• 

Programme results have 
been published by:  
Commission services 
Programme beneficiaries 
Other stakeholders 

• 
• 
• 

To what extent are publications 
made accessible to the wider 
scientific and health community 
and to the public? 

To what extent are the results 
used by stakeholders? 

• 

• 

Publications are available 
to wider stakeholders and 
the public 

Publications have been 
used by other stakeholders 
in research or other 
activities 

• 

• 

[Quantitative & Qualitative] 
Number & type of publications 
available Open Access (green & 
gold) 

[Qualitative] Stakeholder views 
on quality of deliverables and 
dissemination efforts 

To what extent have the recommendations from previous evaluations been implemented? 
To what extent has SANTE 
maintained a focus on thematic 
areas of strong EU added value 

• 

• 

• 

• 

To what extent has SANTE 
strengthened and built links 
between the HP and wider 
Commission & EU policy agenda 
to maximise impact? 

SANTE has undertaken 
activities that focus on 
areas identified as having 
the most EU added value 
since the mid-term 
evaluation 
SANTE has identified MS 
needs where the 
Programme can provide 
added value and acted on 
these 
SANTE has learned from 
experiences of successful 
coordination and extended 
these efforts where they 
have worked and/or 
undertaken new such 

• 

• 

• 

[Qualitative] Documented 
priorities and actions reflect a 
focus on areas of greatest 
added-value 
[Qualitative] Stakeholder views 
on extent to which areas of 
greatest added value have 
been prioritised and acted upon 

[Qualitative] Programme 
actions indicate an effort to 
coordinate with wider 
Commission & EU policy 
agenda 
[Qualitative] Stakeholder views 

82 

Document review, 
including programme 
monitoring & reporting 
data 

•  Mapping of project 

• 

database- focus on outputs 
and outcomes 
Document review, 
including programme 
monitoring & reporting 
data 

•  Mapping of project 

database 
Document review 

• 
•  Mapping of project 

database 
Social media analysis 

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews,  
Results of analysis under 
EQ18 on EU added value of 
the Programme  

Document review 
Targeted stakeholder 
surveys 
PC 
Stakeholder interviews,  
Results of analysis under 

• 

• 
• 

• 
• 

• 
• 

• 
• 
• 

Programme Implementation 
documentation 
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme 

• 

• 

• 

• 

• 

• 
• 

• 

• 
• 

• 

Programme Implementation 
documentation 
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme 

Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

Document review supporting EQ18 
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

Document review supporting EQ21 
Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

 
 
 
 
8c 

8d 

8e 

To what extent has SANTE spelt 
out how action targeting health 
promotion & health systems 
should generate EU added value?  

To what extent has SANTE refined 
Programme thematic priorities 
and streamlined them in 
EU4Health to focus spending on 
areas with the greatest potential 
impact? 

To what extent have SANTE & 
CHAFEA refined the EU-added 
value criteria and fully integrated 
these into the application 
process? 

• 

• 

• 

• 

• 

• 

efforts 

SANTE has defined in detail 
the mechanisms by which 
best practices should be 
taken up in practical terms 
and timescales for doing so 
since the mid-term 
evaluation 
This information has been 
shared with key 
stakeholders 
Operational objectives for 
the next Programme have 
been revised to detail how 
the Programme should 
generate added value 
Thematic priorities have 
been refined since the mid-
term evaluation to reflect 
more concretely their 
anticipated results 
Thematic priorities have 
been streamlined for the 
next Programme to avoid 
overlap or ambiguity as 
well as any redundancies 

The EU-added value criteria 
have been refined & their 
use integrated more fully 
into the application process 

8f 

To what extent has SANTE 

•  Multi-annual planning has 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

on extent to which coordination 
has occurred  
[Qualitative] Guidance has 
been created which details how 
actions generate added value  
[Qualitative] Operational 
objectives for the next 
Programme have been clearly 
revised to take this into account 
[Qualitative] Stakeholder views 
on extent to which this 
information has been shared 
with stakeholders and reflected 
in the next Programme’s 
objectives 

[Qualitative] Documented 
refinements to existing 
priorities to reflect anticipated 
results 
[Qualitative] Documented 
efforts to streamline priorities 
in the EU4Health programme 
[Qualitative] Stakeholder views 
on extent to which existing 
priorities have been refined and 
streamlined for the future 
[Qualitative] Stakeholder views 
on the extent to which the EU 
added value criteria have 
improved & been used in a 
more integrated way in the 
application process 
[Qualitative] Stakeholder views 
on the extent to which 
applicants & assessment 
panellists understand the EU 
added value criteria & how to 
apply them 
[Qualitative] Review of 
Programme documents 
demonstrate these changes, 
including development of ‘how 
to guides’ or other guidance 
[Qualitative] Stakeholder views 

83 

coherence EQs  

• 
• 
• 

Document review 
Stakeholder interviews,  
Results of analysis under 
EQ18 on EU added value of 
the Programme 

• 
• 

Document review 
Stakeholder interviews 

• 
• 

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews,  

• 
•  Mapping of project 

• 
• 

• 

• 

• 

Document review supporting EQ18 
Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

• 
• 

Document review supporting EQ19 
Programme Implementation 
documentation  

• 

database 
Results of analysis under 
EQ19 on the EU added 
value criteria 

• 

Document review 

• 

Programme Implementation 

 
 
integrated multi-annual planning 
with existing programme 
processes? 

been integrated with the 
formal priority setting 
process since the mid-term 
evaluation 

8g 

8h 

To what extent have SANTE & 
CHAFEA developed a broader 
strategy to increase participation 
from poorer MS & 
underrepresented organisations? 

To what extent have SANTE & 
CHAFEA invested in the resources 
necessary to improve systems for 
monitoring programme 
implementation? 

• 

• 

• 

• 

• 

A broader strategy to 
increase participation from 
poorer MS & 
underrepresented 
organisations has been 
developed since the mid-
term evaluation 
Low-GDP MS are 
participating in the 
Programme at higher rates 
or granted greater 
proportions of funding since 
the mid-term evaluation 
The exceptional utility 
criterion is being used more 
since the mid-term 
evaluation 
Additional resources have 
been allocated to 
monitoring systems since 
the mid-term evaluation 
Real-time, accurate 
information about HP 
implementation is available 
to programme managers 

8i 

To what extent have SANTE & 
CHAFEA implemented and used 

• 

Programmatic and action 
specific monitoring 

• 

• 
• 

• 

• 

• 

• 

• 

• 

on the extent to which multi-
annual planning has become 
more formally incorporated 
into priority setting 
[Qualitative] Stakeholder views 
on the extent to which they feel 
more connected to the multi-
annual planning processes 

[Qualitative] A strategy from 
increasing participation has 
been developed and/or 
implemented 
[Quantitative] Resources 
allocated/dedicated to 
increasing participation from 
low GDP MS over the 
Programme period & compared 
with mid-term evaluation 

[Quantitative] Resources 
allocated/dedicated to 
Programme monitoring & 
trends in resource levels so 
allocated over the Programme 
period 
[Qualitative] An electronic 
monitoring system has been 
developed and/or implemented 
[Qualitative] Stakeholder views 
on the extent to which 
additional resources have been 
focused on monitoring 
[Qualitative] Stakeholder views 
on the extent to which 
monitoring data has become 
more accurate / easier to 
access / less fragmented 
[Qualitative] Programme 
documents demonstrate 

84 

• 

• 

• 
• 

• 
• 

• 
• 
• 

Targeted stakeholder 
surveys 
Stakeholder interviews,  

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews  
Results of analysis under 
EQ6 on exceptional utility 
criterion 

• 

• 
• 

• 

• 

documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

Document review supporting EQ6 
Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

Document review 
Stakeholder interviews  
Results of analysis under 
EQ16 on monitoring 
processes & resources 

• 
• 

Document review supporting EQ16 
Programme Implementation 
documentation  

• 
• 

Document review 
Stakeholder interviews  

• 
• 

Document review supporting EQ16 
Programme Implementation 

 
 
 
programmatic and action specific 
monitoring indicators? 

8j 

To what extent has dissemination 
of action results been improved?  

• 

• 

• 

indicators have been 
introduced and used since 
the mid-term evaluation 
Existing programme 
monitoring indicators have 
become more 
comprehensive since the 
mid-term evaluation 
A system for reporting on, 
collecting and presenting 
data on action specific 
indicators has been put in 
place 

Dissemination of action 
results has clearly increased 
and the quality of 
dissemination efforts has 
improved since the mid-
term evaluation 

• 

• 

• 

• 

implementation of more 
specific monitoring indicators 
[Qualitative] Programme 
documents demonstrate that a 
system for monitoring action 
specific indicators has been put 
in place 
[Qualitative] Stakeholder views 
on the extent to which 
programme & action specific 
monitoring indicators have 
been implemented & used, as 
well as a system for monitoring 
action indicators has been put 
in place 
[Quantitative & Qualitative] 
Trends in number, type and 
source of publications since the 
mid-term evaluation and 
comparison with mid-term 
results 
[Qualitative] Stakeholder views 
on quality of dissemination 
efforts since the mid-term 
evaluation and comparison 
with mid-term results 

• 

Results of analysis under 
EQ16 on monitoring 
processes & resources 

documentation  

Document review 

• 
•  Mapping of project 

• 
• 

database 
Stakeholder interviews  
Results of analysis under 
EQ7 

• 
• 

Document review supporting EQ7 
Programme Implementation 
documentation  

9 
9a 

How are the results and effects of the Programme likely to last at the end of its implementation if funding ceases to exist (self-sustainability)? 
To what extent are the 
Programme results and effects 
likely to be sustainable?  
• 

• 

• 

• 

• 

Document review 

• 
•  Mapping of project 

Programme results and 
effects demonstrate 
evidence of being continued 
regardless of Programme 
funding (utilising 
sustainability plans where 
they have been requested 
in projects) 

[Qualitative] Stakeholder views 
on the sustainability of 
Programme results 
[Qualitative] Reviewing specific 
levers and barriers to 
sustainability of how the work 
funded by Programme has been 
sustained.  

database 
Targeted stakeholder 
surveys 
Stakeholder interviews  
Case studies 
Results of analysis under 
effectiveness EQ4 

• 
• 
• 

10 

Efficiency 
To what extent has the Programme been cost effective? 

• 

85 

• 
• 
• 

• 

Document review supporting EQ4 
Project documentation in database 
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  
EU-level collected data on health 
indicators to help understand the 
relevance of the Programme 

 
 
 
 
 
 
 
 
10a 

To what extent could the same results 
have been achieved with fewer 
resources? 

10b 

Regarding Programme objectives 
partially met or unmet, what have 
been the opportunity costs? 

• 

• 

• 

Programme actions 
have deviated from 
their planned resource 
budgets 
Presence of 
Programme actions 
which had high costs 
but low impacts  

The degree to which 
the impacts foreseen 
for the Programme 
have matched the 
impacts achieved, and 
where there are 
discrepancies, an 
assessment of the 
opportunity costs of 
these 

• 

• 

• 

• 

• 

• 

• 

• 

Document review 

• 
•  Mapping of project 

• 
• 

database 
Stakeholder interviews  
Responses to effectiveness 
questions  

[Quantitative] Comparison of 
planned vs actual 
implementation budgets 
[Qualitative] Expert opinion on 
planned and actual budgets 
and factors influencing 
deviations (including reasons) 
[Quantitative / Qualitative] 
Action and causal chain 
effectiveness 
[Quantitative] Actual 
implementation budgets 
[Qualitative] Assessment of 
effectiveness of the Programme  

[Quantitative & Qualitative] 
Degree of objective 
achievement (e.g. assessment 
of Programme effectiveness) 
[Quantitative & Qualitative] 
Typology of benefits 
anticipated for each of the 
objectives 
[Qualitative] Assessment of any 
discrepancies between the 
expected and achieved impacts 

• 

• 

• 

• 
• 

Document review, namely 
considering:  
Response to effectiveness 
question EQ5 
Impacts anticipated for 
Programme  
Stakeholder interviews  
Cost-effectiveness analysis 

Programme Implementation 
documentation  

Programme Implementation 
documentation  

• 

• 
• 

• 

• 

11 

11a 

To what extent are the costs associated with the Programme proportionate to the benefits it has generated? What factors are influencing any particular discrepancies? How do these factors link to the 
Programme? 
To what extent are the Programme 

[Quantitative & Qualitative] 

Costs associated with 

Document review 

Programme Implementation 

• 

• 

• 

• 

86 

 
 
 
 
 
 
costs proportional to the expected 
results? 

11b 

What factors influence any disparities 
between Programme costs and 
expected results? 

11c 

To what extent are these factors 
linked to the Programme? 

the Programme are 
reasonable and kept to 
the minimum necessary 
in order to achieve the 
expected results, 
including:  
Programme 
operational costs 
(design & 
implementation) 
•  Management costs for 

• 

• 

funding 
Administrative costs for 
applicants & CHAFEA 

•  Monitoring & reporting 

• 

• 

costs for MS and the 
Commission  
Identification of 
factors, both internal 
and external to the 
Programme related to 
any disproportionate 
costs found 

The degree to which 
factors identified as 
creating 
disproportionate costs 
can be directly linked to 
the Programme  

Typology and accounting of 
costs associated with the 
implementation of the 
Programme 
[Quantitative & Qualitative] 
Typology and (where possible) 
accounting of benefits 
associated with the 
implementation of the 
Programme 
[Qualitative] Stakeholder views 
on the extent to which costs are 
reasonable given the objectives 
and expected results  

[Qualitative] Stakeholder views 
on factors that 
disproportionately affect costs 
relative to expected benefits 

[Qualitative] Stakeholder views 
on the extent to which 
identified factors are directly 
linked to the Programme 

• 

• 

• 

• 

•  Mapping of project 

documentation  

database 
PC 
Targeted stakeholder 
surveys 
Stakeholder interviews  
Cost-effectiveness analysis 
Analysis of efficiency EQs, 
especially 14-17 

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews  
Cost-effectiveness analysis 
Analysis of efficiency EQ12 

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews  
Cost-effectiveness analysis 
Analysis of efficiency EQ 12 

• 
• 

• 
• 
• 

• 
• 

• 
• 
• 

• 
• 

• 
• 
• 

• 

• 

• 

• 

• 

Programme Implementation 
documentation  

Programme Implementation 
documentation  

12 

12a 

To what extent do factors linked to the Programme (e.g. the number of priorities, available financial and human resources, various financial mechanisms, established procedures, intended results, political focus) 
influence the efficiency with which the observed achievements were attained? What other factors influence the costs and benefits? 
How does the design and 
implementation of the Programme 
influence the efficiency with which 
achievements were attained? 

Programme Implementation 
documentation  

• 
•  Mapping of project 

Document review 

• 

• 

• 

• 

• 

database 
Targeted stakeholder 
surveys 
Stakeholder interviews  
Cost-effectiveness analysis 

• 
• 

Identification of factors 
relating to the 
implementation of the 
Programme resulting in 
discrepancies in the 
efficiency of achieving 
its objectives 
Identification of the 
magnitude to which 
factors related to the 
implementation of the 
Programme influence 

• 

[Quantitative & Qualitative] 
Typology and accounting of 
costs associated with the 
implementation of the 
Programme 
[Quantitative & Qualitative] 
Typology and (where possible) 
accounting of benefits 
associated with the 
implementation of the 
Programme 
[Quantitative / Qualitative] 

• 

• 

87 

 
 
 
 
 
 
 
 
 
 
12b 

What other factors influence the costs 
and benefits of the Programme? 

the efficiency of 
achieving its objectives 

• 

• 

Identification of factors 
external to the 
implementation of the 
Programme resulting in 
discrepancies in the 
efficiency of achieving 
its objectives 
Identification of the 
magnitude to which 
factors external to the 
implementation of the 
Programme influence 
the efficiency of 
achieving its objectives 

• 

• 

• 

• 

• 

• 

• 

Action and causal chain 
effectiveness 
[Quantitative] Actual 
implementation budgets 
[Qualitative] Stakeholder 
assessment of factors 
influencing the effectiveness of 
the Programme  
[Quantitative & Qualitative] 
Typology and accounting of 
costs associated with the 
implementation of the 
Programme 
[Quantitative & Qualitative] 
Typology and (where possible) 
accounting of benefits 
associated with the 
implementation of the 
Programme 
[Quantitative / Qualitative] 
Action and causal chain 
effectiveness 
[Quantitative] Actual 
implementation budgets 
[Qualitative] Stakeholder 
assessment of factors 
influencing the effectiveness of 
the Programme  

13 
13a 

To what extent was the distribution of Programme credits among the four thematic priorities efficient? 
To what extent were the four 
thematic priorities allocated 
Programme credits in line with EU 
health priorities? 

• 

• 

• 

13b 

To what extent was the funding 
allocation considered to be critical to 
achieve the expected results? 

• 

• 

• 

Identification of the 
degree to which 
Programme funding 
was distributed across 
the four thematic 
priorities 
Identification of EU 
health priorities  

Assessment of whether 
the allocation of 
resources under the 
four thematic priorities 
is proportional to the 
expected results 

[Quantitative] Actual 
implementation budgets 
[Qualitative] Mapping of EU 
health priorities onto the 
Programme's four thematic 
priorities 
[Qualitative] Stakeholder 
assessment of the alignment of 
the four thematic priorities and 
EU health priorities 
[Quantitative] Actual 
implementation budgets 
[Quantitative & Qualitative] 
Typology and accounting of 
costs associated with the 
implementation of the 
Programme 

• 

• 

• 

88 

• 

• 

• 

• 

• 

• 

• 
• 

• 
• 

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews  
Cost-effectiveness analysis 

• 
Document review 
•  Mapping of project 

• 
• 

• 
• 
• 
• 

database 
Stakeholder interviews  
Cost-effectiveness analysis 

Document review 
Stakeholder interviews  
Cost-effectiveness analysis 
Secondary data analysis of 
relevant health indicators 
(i.e. Eurostat, 
Eurobarometer, OECD, 

Programme Implementation 
documentation  

Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  
Programme Implementation 
documentation  

Programme Implementation 
documentation  

 
 
 
 
 
 
 
WHO) 

• 

• 

• 

[Quantitative & Qualitative] 
Typology and (where possible) 
accounting of benefits 
associated with the 
implementation of the 
Programme  
[Qualitative] Stakeholder 
assessment of the role of the 
distribution of funding 
allocation in the achievement 
of Programme objectives 
[Quantitative] Analysis of 
MS/EU level health trends and 
wider macroeconomic 
indicators  

Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  
EU-level collected data on health 
indicators to help understand the 
relevance of the Programme 

• 

• 

• 

• 

• 

• 
• 
• 

database 
Targeted stakeholder 
surveys 
Stakeholder interviews  
Cost-effectiveness analysis 
Secondary data analysis of 
relevant health indicators 
(i.e. Eurostat, 
Eurobarometer, OECD, 
WHO) 

14 
14a 

If there are significant differences in costs (or benefits) between participating countries, what is causing them? How do these differences link to the Programme? 
What if any differences in costs (or 
benefits) occurred between 
participating countries? 

• 
•  Mapping of project 

Document review 

• 

• 

• 

• 

• 

• 

• 

14b 

What factors resulted in any 
differences observed (where 
significant)? 

Assessment of the 
distribution of 
Programme funding 
across Member States 
Identification of the 
costs incurred by 
Member States in the 
implementation of the 
programme 
Identification of the 
benefits accrued by 
Member States in the 
implementation of the 
programme 

Assessment of the 
significance of factors 
relating to programme 
design and 
implementation (i.e. 

[Quantitative] Actual 
implementation budgets 
[Quantitative & Qualitative] 
Typology and accounting of 
costs associated with the 
implementation of the 
Programme 
[Quantitative & Qualitative] 
Typology and (where possible) 
accounting of benefits 
associated with the 
implementation of the 
Programme  
[Qualitative] Stakeholder 
assessment of the role of the 
distribution of funding 
allocation, type of funding 
mechanism, thematic priorities, 
and thematic objectives in the 
achievement of Programme 
objectives 
[Quantitative] Analysis of 
MS/EU level health trends and 
wider macroeconomic 
indicators  
[Quantitative] Actual 
implementation budgets 
[Quantitative & Qualitative] 
Typology and accounting of 
costs associated with the 

89 

• 

• 

• 

• 

• 

• 
• 

• 
• 

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews 
Cost-effectiveness analysis 

• 

• 

Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

 
 
 
 
14c 

To what extent can the differences be 
linked to the Programme itself? 

• 

Secondary data analysis of 
relevant health indicators 
(i.e. Eurostat, 
Eurobarometer, OECD, 
WHO) 

• 
• 
• 
• 

Document review 
Stakeholder interviews 
Cost-effectiveness analysis 
Secondary data analysis of 
relevant health indicators 
(i.e., Eurostat, 
Eurobarometer, OECD, 
WHO) 

• 

• 
• 

• 

• 

• 

EU-level collected data on health 
indicators to help understand the 
relevance of the Programme 

Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  
EU-level collected data on health 
indicators to help understand the 
relevance of the Programme 

• 

• 

• 

funding mechanism, 
thematic objective, 
thematic priority, 
funding levels) in 
creating differences in 
the costs and benefits 
experienced by 
Member States 
Assessment of the 
significance of factors 
external to the 
programme (i.e. wider 
health trends, country-
level factors) in 
creating differences in 
the costs and benefits 
experienced by 
Member States 

Assessment of the 
significance of factors 
relating to the 
programme in 
determining differences 
in the cost and benefits 
observed in Member 
States, relative to 
factors external to the 
programme 

• 

• 

• 

• 

• 

• 

• 

• 

implementation of the 
Programme 
[Quantitative & Qualitative] 
Typology and (where possible) 
accounting of benefits 
associated with the 
implementation of the 
Programme  
[Qualitative] Stakeholder 
assessment of the role of the 
distribution of funding 
allocation, type of funding 
mechanism, thematic priorities, 
and thematic objectives in the 
achievement of Programme 
objectives 
[Quantitative] Analysis of 
MS/EU level health trends and 
wider macroeconomic 
indicators  
[Quantitative] Actual 
implementation budgets 
[Quantitative & Qualitative] 
Typology and accounting of 
costs associated with the 
implementation of the 
Programme 
[Quantitative & Qualitative] 
Typology and (where possible) 
accounting of benefits 
associated with the 
implementation of the 
Programme  
[Qualitative] Stakeholder 
assessment of the role of the 
distribution of funding 
allocation, type of funding 
mechanism, thematic priorities, 
and thematic objectives in the 
achievement of Programme 
objectives 
[Qualitative] Stakeholder 
assessment of the role of 
external factors in the 
achievement of Programme 
objectives 

90 

 
 
 
 
• 

[Quantitative] Analysis of 
MS/EU level health trends and 
wider macroeconomic 
indicators  

15 
15a 

15b 

16 

16a 

• 

• 

• 

• 

• 

• 

• 
• 

Programme Implementation 
documentation  

To what extent is there scope to 
further reduce costs? 

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews 

The simplification 
measures led to a 
reduction in the 
administrative costs for 
applicants and CHAFEA 

To which extent did the simplification measures contribute to the efficiency of the Programme? Was there further scope for simplification to make the Programme implementation more efficient? 
To what extent did the simplification 
measures reduce administrative costs 
for applicants and CHAFEAh? 

[Qualitative] Programme 
information indicating the 
extent to which simplification 
measures reduce 
administrative costs 
[Qualitative] Stakeholder 
assessment of the extent to 
which simplification measures 
reduce administrative costs 
[Qualitative] Programme 
information indicating the 
extent to which simplification 
measures can be improved to 
further reduce administrative 
costs 
[Qualitative] Stakeholder 
assessment of the extent to 
which simplification measures 
can be improved to further 
reduce administrative costs 
To what extent were the monitoring processes and resources (at the Commission and MS level) cost-effective? How the role and benefits of the monitoring systems [i.e. to plan and promote the results of the 
Health Programme and finally to incite stakeholders (internal and external) to make use of them] are assessed, against the costs of these monitoring systems (also considering any administrative burden 
involved)? 
To what extent do the monitoring 
processes enable efficient 
management of supported actions? 

Assessment of 
potential 
improvements to the 
simplification measures 
to further reduce costs 

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews  

Programme Implementation 
documentation  

Programme Implementation 
documentation  

• 
• 

• 
• 

• 

• 

• 

• 

• 

• 

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews 
Cost effectiveness analysis 

• 
• 

• 

•  Monitoring resources 
are adequate to 
support the established 
processes 
The monitoring 
framework includes 
indicators, targets, and 
objectives that enable 
effective measurement 
of results 

•  Monitoring processes 
are effective across all 
MS 

•  Monitoring processes 
enable effective 
dissemination and 
promotion of 

[Qualitative] Programme 
information indicating the 
effectiveness of monitoring 
processes  
[Qualitative] Stakeholder 
assessment of the effectiveness 
of monitoring processes in the 
management of supported 
actions and the dissemination 
and promotion of results 

• 

• 

91 

 
 
 
 
16b 

To what extent are the monitoring 
costs proportional to the expected 
results? 

Programme results 

•  Monitoring does not 

entail resources beyond 
the minimum necessary 
to achieve the expected 
results 

[Quantitative] Data on 
monitoring costs across the 
implementation period at 
Commission and (if available) 
Member State level 
[Qualitative] Stakeholder 
assessment of the 
proportionality of monitoring 
costs relative to the 
effectiveness of the monitoring 
processes 

• 

• 

• 

What are the benefits of the reporting systems against their costs and how could they be effectively implemented? 
What benefits have resulted from the 
reporting system? 

• 

• 

17 
17a 

17b 

What are the costs of the reporting 
system and are these proportionate in 
relation to the benefits? 

17c 

Are there any ways in which the 
reporting systems could be more 
effectively implemented? 

Identification of 
benefits to 
stakeholders resulting 
from the reporting 
system 

• 

• 

Identification of costs 
to stakeholders 
resulting from the 
reporting system 

Assessment of 
improvements in the 
implementation of the 
reporting system 

• 

• 

• 

• 

• 

[Qualitative] Programme 
information identifying benefits 
of the reporting system 
[Qualitative] Stakeholder 
assessment of the benefits 
resulting from the 
implementation of the 
reporting system 
[Qualitative] Programme 
information indicating the costs 
resulting from the reporting 
system 
[Qualitative] Stakeholder 
assessment of the costs 
resulting from the 
implementation of the 
reporting system 
[Qualitative] Programme 
information indicating ways the 
reporting systems could be 
effectively implemented 
[Qualitative] Stakeholder 
assessment of the scope for 
potential improvements to be 
made to the reporting system 

• 
• 

• 
• 

• 
• 

• 

• 
• 

• 

• 
• 

• 

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews 
Cost effectiveness analysis 

• 

Programme Implementation 
documentation  

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews 

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews 

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews 

• 

Programme Implementation 
documentation  

Programme Implementation 
documentation  

Programme Implementation 
documentation  

• 

• 

• 

• 

18 

18a 

EU-added value 
What is the additional value resulting from the Programme, compared to what could reasonably have been expected from Member States acting at national and/or regional levels, and compared to what the EU 
would have achieved without the Programme? 
Why has action at EU-level been 
the most appropriate? To what 
extent have the results produced 

[Qualitative] Stakeholder views 
on the extent to which EU level 
action (i.e. the Programme) 

Programme Implementation 
documentation  
Strategic Documents 

Improvements cannot be 
viewed as a result of 
Member States efforts and 

PC 
Document review 
Targeted stakeholder 

• 
• 
• 

• 

• 

• 

• 

92 

 
 
 
 
 
 
 
 
 
 
 
 
under the Programme gone 
beyond what Member States 
would have achieved in its 
absence (considering public and 
private initiatives at 
international, national and local 
levels)?  

18b 

To what extent have the results 
produced under the Programme 
gone beyond what the EU would 
have achieved in its absence? 

initiative alone, i.e. 
Member States took actions 
as a result of the 
Programme that would 
otherwise not have taken 
place, or would have 
occurred more slowly or to 
a lesser extent 
There is a clear link 
between the characteristics 
of health and healthcare 
challenges and the need for 
action at EU-level 
There was no detrimental 
impact on existing Member 
State actions in respect of 
health and healthcare (i.e. 
the EU Strategy did not 
disrupt or slow existing 
activity or activity that was 
already planned) 
Areas for EU action are 
appropriate in view of EU 
and national competencies 
The Programme has led to 
results that go beyond what 
the EU would have 
achieved in its absence 

• 

• 

• 

• 

• 

• 

provided added-value 
[Qualitative] EU dimension vs 
national dimension of the 
problems the Programme has 
aimed to resolve 
[Qualitative] Evidence that MS 
actions have been 
helped/incentivised (and not 
harmed) by the Programme 

• 
• 
• 

surveys 
Stakeholder interviews 
Focus groups 
Synthesis of evidence 
collected through other 
EQs  

• 

(policies/reports) to understand 
relevance of the Programme  
EU-level collected data on health 
indicators to help understand the 
relevance of the Programme 

• 

• 

[Qualitative] Stakeholder views 
on the potential impact of 
discontinuing the Programme 
[Qualitative] Expert views on 
what if scenarios involving the 
discontinuation of the 
Programme 

• 
• 
• 
• 
• 

Document review 
Stakeholder interviews 
Social media analysis 
Focus groups 
Synthesis of evidence 
collected through other 
EQs  

• 

• 

• 

Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  
EU-level collected data on health 
indicators to help understand the 
relevance of the Programme 

19 

19a 

• 

How far have the EU added value criteria led to the development of proposals that better addressed these aspects? Are all of these criteria still relevant? Which criteria have been most/least 
addressed? 

To what extent have the EU 
added value criteria led to 
proposals that better address the 

• 

The eight added-value 
criteria are well-defined 
and evidenced in funding 
proposals 

• 

• 

[Qualitative] Extent to which 
each of the criteria are 
assessed to be well-defined 
[Qualitative] Extent to which 

Document review 

• 
•  Mapping of project 

database 
Targeted stakeholder 

• 

• 

Programme Implementation 
documentation  

93 

 
 
need for added value?  

• 

19b 

To what extent are the added 
value criteria still relevant?  

19c 

To what extent have the added 
value criteria been addressed in 
funding proposals?  

• 

• 

The cause-effect chain can 
be established between the 
added value criteria as 
applied in proposals and 
the extent to which the 
Programme has produced 
results with added value 

The added value criteria are 
relevant in light of current 
added-value needs & 
priorities 

The added-value criteria 
have all been addressed in 
funding proposals, and in 
proportion to their relative 
importance 

• 

• 

• 

• 

• 

• 

each of the criteria have been 
evidenced in funding proposals 
[Qualitative] Evidence of 
linkages between added value 
as applied in proposals and 
Programme results 
[Qualitative] Stakeholder views 
on the added value criteria, 
including their definition, and 
the relationship between their 
use in proposals and 
Programme results 
[Qualitative] Extent to which 
each of the criteria are 
assessed to be aligned with 
added value needs & priorities 
for health 
[Qualitative] Stakeholder views 
on the relevance of added value 
criteria in relation to current 
needs & priorities 
[Qualitative & Quantitative] 
Measurement of the extent to 
which each of the criteria have 
been applied in funding 
proposals 
[Qualitative] Stakeholder views 
on the relevance of added value 
criteria in relation to current 
needs & priorities 

surveys 
Stakeholder interviews 

• 

• 
• 

• 

• 
• 

• 
• 

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews 

• 

• 

Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

• 

Programme Implementation 
documentation  

Document review 
Targeted stakeholder 
surveys 
Stakeholder interviews  
Analysis of added value 
EQ19a 

20 
20a 

• 

• 

• 

• 

• 

Coherence 
Are the actions implemented under the 3rd Health Programme coherent with its objectives? How has the coherence of the Programme influenced its effectiveness? 
How well have the actions 
implemented under the 
Programme aligned with its 
objectives, over time and up until 
2020? Conversely, are there any 
gaps, areas of tension or 
inconsistencies? 
Where there have been 
inconsistencies or gaps, what has 
caused these? What have been 
the impacts?  

[Qualitative] Extent to which 
each of the actions have 
supported the objectives of the 
Programme 
[Qualitative] Extent to which 
actions have been aligned with 
one another where relevant  
[Qualitative] Evidence of 
tensions between different 
objectives or actions 
undertaken as part of the 
Programme 
[Qualitative] Evidence of 

The actions undertaken as 
part of the Programme 
have appropriately 
addressed its objectives  
The actions undertaken 
have been aligned with 
each other where possible  
There have been no 
tensions between the 
actions undertaken as part 
of the Programme and no 
gaps in terms of actions 
taken in relation to 
Programme objectives 

database 
Stakeholder interviews 
Focus groups 
Intervention logic mapping 
to identify potential 
synergies, 
complementarities or 
tensions 

Document review, 
including outputs and 
deliverables associated 
with actions undertaken 
•  Mapping of programme 

• 
• 
• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

94 

 
 
 
 
 
 
 
 
20b 

How has the coherence of the 
Programme influenced its 
effectiveness? 

• 

Programme effectiveness 
(as assessed in 
effectiveness EQs) was 
influenced by Programme 
coherence (as assessed in 
other coherence EQs) 

sufficient uptake of 
opportunities so that objectives 
are well-covered in relation to 
actions 
[Qualitative] Stakeholder 
insights on factors leading to 
inconsistencies and gaps 
[Qualitative] Stakeholder 
insights on impacts of 
inconsistencies and gaps 
[Qualitative] Evidence of 
Programme coherence 
influencing effectiveness 
[Qualitative] Stakeholder 
insights on factors leading to 
coherence influencing 
effectiveness (or a lack thereof) 
[Qualitative] Stakeholder 
insights on impacts of 
coherence (or lack thereof) on 
Programme effectiveness 

• 

• 

• 

• 

• 

• 
• 
• 

• 

Document review 
Stakeholder interviews  
Analysis of effectiveness 
EQs 
Analysis of other coherence 
EQs 

• 

Programme Implementation 
documentation  

• 

To which extent have the priorities of the Programme led to more synergy, focus and coherence between the EU-funded actions in delivering on similar objectives? Did the health Programme 
encourage cooperation with the European Structural and Investment Funds and other EU financial instruments? To which extent is the Programme coherent with wider EU policy and with 
international obligations? 

21 

21a 

To what extent have the 
Programme priorities led to more 
synergy, focus and coherence 
between the funded actions over 
time and up until 2020?  
Where there have been 
inconsistencies or a lack of focus 
and coherence, what has caused 
this? What have been the 
impacts?  

• 

• 

• 

Programme priorities are 
reflected in the coherence 
of funded actions (and with 
reference to the assessment 
of coherence in EQ20). 
Programme actions are 
clearly focused in relation 
to the priority areas.  
Programme actions clearly 
exhibit synergies with one 
another and in relation to 
priority areas.  

21b 

Did the health Programme 
encourage cooperation with the 

• 

Provisions for cooperation 
were established within the 
Programme 

• 

• 

• 

• 

• 

• 

[Qualitative] Evidence of the 
alignment of priorities with 
funded actions  
[Qualitative] Evidence of focus 
and synergies between 
priorities and funded actions  
[Quantitative] Analysis of 
planned and realised funding 
for actions in relation to 
Programme priorities 
[Qualitative] Expert 
stakeholders agree that that 
coherence, focus and synergies 
exist between funded actions 
and priorities 
[Qualitative] Insights from 
stakeholders on factors leading 
to any lack of coherence, focus 
and/or synergy. 
[Qualitative] Evidence of 
provisions being made in 
Programme documentation to 

95 

• 
• 

• 
• 

Document review 
Targeted stakeholder 
surveys, particularly with 
NCAs 
Stakeholder interviews 
Focus groups 

• 

• 

Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

• 
• 
• 

Document review 
Stakeholder interviews 
Focus groups 

• 

• 

Programme Implementation 
documentation  
Strategic Documents 

 
 
European Structural and 
Investment Funds and other EU 
financial instruments? 

• 

Cooperation activities were 
undertaken with the 
European Structural and 
Investment Funds and other 
EU financial instruments 

To what extent has the 
Programme been aligned with 
wider EU policy and international 
obligations with common 
objectives?  
Where there have been 
inconsistencies or gaps, what has 
caused these? What have been 
the impacts?  

•  Wider EU policies 

• 

incorporate, are aligned 
with, and/or do not 
contradict the Programme, 
in particular:  
Research & Innovation 
framework programmes 
EU Cohesion Policy 
Food and food safety 
International obligations 
with common objectives 
are aligned with and/or do 
not contradict the 
Programme, in particular:  
•  WHO European Action Plan 

• 
• 
• 

• 

(EAP-PHS)  
Sustainable development 
goals 

(policies/reports) to understand 
relevance of the Programme  

• 

• 

• 

Programme Implementation 
documentation  
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  
EU-level collected data on health 
indicators to help understand the 
relevance of the Programme 

• 

• 

• 
• 

• 

Document review, 
including mapping of wider 
EU policies and 
international obligations 
related to health and 
healthcare, to be compiled 
based on:  
Expert/DG SANTE 
recommendations 
NCA survey 
Complementary document 
review 
Stakeholder interviews  

cooperate with other EU 
financial instruments 
[Qualitative] Evidence of 
cooperation activities being 
undertaken with other EU 
financial instruments 
[Quantitative] Analysis of 
planned and realised funding 
for actions where cooperation 
was undertaken 
[Qualitative] Insights from 
stakeholders on factors leading 
to cooperation and/or any 
areas where there was a lack of 
cooperation and reasons for 
this. 
[Qualitative] Comparison of 
wider EU policies between 
2014-2020 against Programme 
objectives 
[Qualitative] Comparison of 
international obligations with 
common objectives between 
2014-2020  
[Qualitative] Expert assessment 
of how EU Programme 
objectives are reflected in wider 
EU policies and vice versa 
[Qualitative] Expert assessment 
of how EU Programme 
objectives are reflected in 
meeting international 
obligations with common 
objectives 

• 

• 

• 

• 

• 

• 

• 

• 

• 

To which extent has the Programme proved complementary to other EU or Member States targets/interventions/initiatives in the field of health? 

To what extent has the 
Programme been coordinated 
and complementary with other 
EU-level policies in the field of 
health over time and up until 
2020? 
Where there have been 
inconsistencies or gaps, what has 

• 

• 

• 

Other EU policies and 
related activities in the field 
of health incorporate 
and/or do not contradict 
the Programme, in 
particular:  
EU Framework on mental 
health & well-being 
Directive 2011/21/EU on 
patients’ rights to cross-

• 

• 

• 

[Qualitative] Comparison of 
other relevant EU-level policies 
and interventions from 2014-
2020 against Programme 
objectives 
[Qualitative] Expert assessment 
of how Programme objectives 
are reflected in other relevant 
EU-level health policies 
[Qualitative] Insights from DG 

• 

• 

• 
• 

Document review, 
including mapping of EU 
interventions related to 
health to be compiled 
based on:  
Expert/DG SANTE 
recommendations 
Survey of NCAs 
Complementary document 
review 

• 

Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

96 

21c 

22 
22a 

 
 
 
caused these? What have been 
the impacts?  

• 

• 

• 

border healthcare 
Decision 1082/2013/EU on 
serious cross-border health 
threats 
ECDC Early Warning & 
Response System 
EU legal frameworks for 
medical products & medical 
devices 
Tobacco legislation 

interventions/initiatives 
developed between 2014-
2020 in the field of health 
incorporate and/or do not 
contradict the Programme 

• 
•  Member State 

22b 

To what extent has the 
Programme been coordinated 
and complementary with 
Member State 
interventions/initiatives in the 
field of health over time and up 
to 2020? 
What have been the drivers for 
this?  
Where there have been 
inconsistencies or gaps, what has 
caused these? What have been 
the impacts? 

• 
• 

Stakeholder interviews  
Focus groups 

• 

• 

• 
• 

• 
• 

Document review, 
including mapping of EU 
interventions related to 
health to be compiled 
based on:  
Expert/DG SANTE 
recommendations 
Survey of NCAs 
Complementary document 
review 
Stakeholder interviews 
Focus groups 

SANTE, other DGs and EU 
Agencies, as well as other 
stakeholders on factors leading 
to inconsistencies and gaps 
[Qualitative] Insights from 
stakeholders on impacts of 
inconsistencies and gaps 

[Qualitative] Comparison of 
relevant Member State policies 
and interventions from 2014-
2020 in relation to Programme 
objectives 
[Qualitative] Expert assessment 
of how Programme objectives 
are reflected in Member State 
health policies and 
interventions 
[Qualitative] Insights from 
Member State representatives 
and other stakeholders on 
factors leading to 
inconsistencies and gaps 
[Qualitative] Insights from 
stakeholders on impacts of 
inconsistencies and gaps 

• 

• 

• 

• 

• 

• 

• 

Documents on MS-level policies in 
the field of health and related 
analysis and commentaries, e.g., MS 
health strategies 
Strategic Documents 
(policies/reports) to understand 
relevance of the Programme  

97 

 
 
 
ANNEX IV. OVERVIEW OF BENEFITS AND COSTS 

Table 1a. Overview of costs and benefits identified in the evaluation 

                               Relevant stakeholders 

EU Citizens  

Businesses/Contractors 
awarded Procurement 
contracts 

Project 
participants/NGOs/Stakeh
olders other than National 
Authorities 

Member States Authorities 
/ National Administrations  

European Commission 
(DG SANTE, CHAFEA, 
Other Commission 
services 

Costs and benefits 

Quantitative   Qualitative  Quantitati

ve 

Qualitativ
e 

Quantitative   Qualitative  Quantitativ

Qualitative  Quantitativ

Qualitative 

e 

e 

Specific objective 1:  promote health, prevent diseases and foster supportive environments for healthy lifestyles 

Cost or Benefit description 

Costs:  

Direct Costs: Monetary input 
into implementation of best 
practices and other actions 
under specific objective 1 

Recurrent  

Direct Cost: Administrative 
costs 

One-off and 
recurrent  

MEUR 32 
of co-
funding of 
Joint 
Actions 

EU co-
funding for 
Joint 
Actions: 
MEUR  

MEUR 30 of 
co-funding 
for Projects 
and Direct 
Grant 

Administrati
ve Costs 
linked with 
participation 
of Actions 
implemented 
by the 
Programme 
(provision of 
information 

Estimanted 
amount of 
MEUR 
139 of 
operaional 
appropriati
on 

Overall 
administrat
ive costs 
for 
running 
the entire 
Programm
e are 
presented 
for all 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

Costs 
linked 
with 
preparatio
n or 
response 
to call for 
tenders 

98 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Direct benefits 

Direct benefits 

Direct benefits 

nomination 
and 
coordination 
with national 
entities 
participating 
in Joint 
actions) 

EU co-
funding of 
grants 
MEUR 56 

Total 
amount of 
awarded 
procurem
ent 
contracts: 
MEUR 46 

specific 
objectives 
and cross-
cutting 
actions 

N/A 

EU co-
funding for 
Joint 
Actions: 
MEUR 36 

24 MS 
using 
evidence-
based good 
practices 
on nutrition 
and in 
particular  
in the area 
of saturated 
fat 

6 MS using 
evidence-
based good 
practices on 
EU 
guidelines 
and 
accreditation 
for breast 
cancer 
screening 

A prize for 
NGOs is 
organised 
annually to 
encourage 
the 
organisatio
ns which 
make 
significant 
contributio
ns in the 
field of 
health 

Online 
best-
practices 
portal 
launched in 
2018: 6 
6650 
visitors 

On cancer: 
128 cancer 
registries 
from 29 
European 
Countries 
provide data 
to the 

99 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
from all 
EU 
Member 
States 

Commission 
with more 
than 25 900 
000 records 
so far 

Implementati
on of and 
uptake of 
best practices 
at national 
leve in the 
fields of: 
cancer, 
tobacco 
control, 
physical 
exercise, 
nutrition, 
cardiovascula
r disease and 
diabetes 

Specific objective 2:  protect Union citizens from serious cross-border health threats 

Other direct benefits 

Indirect benefits 

Better health 
outcomes for 
population 
benefiting 
from 
implementati
on of best 
practices:  

Productio
ns on 
knowledg
e and tools 
at EU 
level: e.g., 
study on 
alcohol-
related 
harm in 
the EU 

100 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Costs 

Direct costs linked with 
Programme implementation 

N/A 

N/A 

N/A 

N/A 

Direct Cost: Administrative 
costs 

One-off and 
recurrent  

Benefits 

Direct benefits 

Costs 
linked 
with 
preparatio
n or 
response 
to call for 
tenders 

Co-funding 
from another 
organisation: 
MEUR 11 

Administrati
ve Costs 
linked with 
participation 
of Actions 
implemented 
by the 
Programme 
(provision of 
information 
nomination 
and 
coordination 
with national 
entities 
participating 
in Joint 
actions) 

Co-funding 
from 
Member 
States: 
MEUR 7 

EU co-
funding for 
Joint 
Actions: 
MEUR  

MEUR 41 
devoted to 
specific 
objective 2 

Overall 
administra
tive costs 
for 
running 
the entire 
Programm
e are 
presented 
for all 
specific 
objectives 
and cross-
cutting 
actions 

During the 
COVID-19 
outbreak 
actions of the 
Programme 
have been 
geared, in 
their 
‘emergency 
mode’, 
towards 

Key 
examples 
of actions 
are the 
Healthy 
Gateways 
Joint 
Action, the 
Joint 
Action on 
Strengthen

During the 
COVID-19 
outbreak 
actions of 
the 
Programme 
have been 
geared, in 
their 
‘emergenc
y mode’, 

Key 
examples of 
actions are 
the Healthy 
Gateways 
Joint Action, 
the Joint 
Action on 
Strengthened 
International 
Health 

101 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
combating 
the pandemic 

ed 
Internation
al Health 
Regulation
s and 
Preparedne
ss 
(SHARP), 
Training 
for Health 
Professiona
ls on 
isolation, 
waiting 
rooms and 
reception 
areas, 
cleaning, 
appropriate 
PPE, real 
time RT-
PCR 
testing for 
COVID-19 

Production 
of new 
knowledge 
on health-
threats 
related 
issues at 
EU level 

towards 
combating 
the 
pandemic 

Regulations 
and 
Preparedness 
(SHARP), 
Training for 
Health 
Professionals 
on isolation, 
waiting 
rooms and 
reception 
areas, 
cleaning, 
appropriate 
PPE, real 
time RT-
PCR testing 
for COVID-
19 

Production of 
new 
knowledge 
on health-
threats 
related issues 
at EU level 

As of 2202, 
22 Member 
States 
integrated 
coherent 
approaches 
in the 
design of 
their 
preparedne
ss plan 

Joint 
Action 
EMERGE 

Feasibility 
study 
launched for 

Production 
of new 
knowledge 
on health-
threats 
related 
issues at 
EU level  

Direct benefits 

Direct benefits 

N/A 

Procurem
ent 
contracts 
awarded: 
MEUR 14 

102 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
the 
development 
of a common 
EU 
vaccination 
card 

improved 
the 
capabilities 
for rapid 
laboratory 
diagnosis 
for new or 
emerging 
pathogens 

Support to 
interventio
ns in 2014-
2016 to 
limit 
spread of 
Zika and 
Ebola  

Direct benefits 

Indirect benefits 

Better 
protection of 
the 
population 
against cross-
border health 
threats 

Better 
protection 
from 
vaccine-
preventable 
diseases 
through 
interventions 
to improve 
vaccine 
uptake 

Specific objective 3: contribute to innovative, efficient and sustainable health systems 

Costs 

Direct Costs linked with 
implementation of the 
Programme 

N/A 

N/A 

Contribution 
from other 
participating 
organisations

Contributio
n of MS 
participatin
g in Joint 

Total 
appropriati
ons under 
specific 

103 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
N/A 

N/A 

Direct Cost: Administrative 
costs 

One-off and 
recurrent  

Costs 
linked 
with 
preparatio
n or 
response 
to call for 
tenders 

: MEUR 30 

Administrati
ve Costs 
linked with 
participation 
of Actions 
implemented 
by the 
Programme 
(provision of 
information 
nomination 
and 
coordination 
with national 
entities 
participating 
in Joint 
actions) 

Actions: 
MEUR 19 

EU co-
funding for 
Joint 
Actions: 
MEUR  

objective 
3: MEUR 
110 

Overall 
administra
tive costs 
for 
running 
the entire 
Programm
e are 
presented 
for all 
specific 
objectives 
and cross-
cutting 
actions 

Benefits 

Direct benefits 

Direct benefits 

Procurem
ent 
contracts 
awarded 
to 
contractor
s: MEUR 
37 

Establisheme
nt of an 
organ 
database 

Setting up of 
the EU 
Health Policy 
Platform 

104 

As of 2020, 
41 Health 
Technolog
y 
Assessmen
ts (HTAs) 
produced 
per year 

23 MS using 
identified 
tools and 
mechanisms, 
in order to 
contribute to 
effective 
results in 
their health 
systems  

Successful 
implement
ation of the 
3rd Joint 

Launching of 
the THEDAS 
Joint Action 
(The 

Improveme
nt of 
country 
knowledge 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
which has 
helped 34000 
transplants in 
2017 

(HPP) a 
collaborative 
on-line tool 
which makes 
it easier for 
health-
related 
interest 
groups, 
stakeholders 
and 
Commission 
services to 
cooperate  

Action of 
HTA 
(EUnetHT
A) which 
led to a 
permanent 
regulation 
setting up 
cooperatio
n 
framework 
between 
MS on 
HTA 

European 
Health 
Dataspace) to 
facilitate the 
European 
Health Data 
Space by 
developing 
principles for 
cross-border 
secondary 
use of health 
data 

in the field 
of health 
through 
regular 
publication 
of Health 
information
, in 
particular: 
The State of 
Health in 
the EU, the 
Country 
Profiles and 
Health at a 
Glance 
publication 
series 

The joint 
action on 
Market 
Surveillanc
e on 
Medical 
Devices 
has yield 
significant 
added 
value by 
reinforcing 
the market 
surveillanc
e system 
for these 
devices and 
improving 
cooperatio
n among 

105 

Direct benefits 

 
 
 
 
 
 
 
 
 
 
 
Indirect benefits 

Strengthenin
g of Health 
Systems of 
the Member 
States for the 
benefit of 
EU citizens 
and patients 

Specific objective 4:  facilitate access to better and safer healthcare for Union citizens 

Costs 

Direct costs for 
implementing the 
Programme 

N/A 

N/A 

Direct Cost: Administrative 
costs 

One-off and 
recurrent  

Contribution 
of other 
organisations
: MEUR23 

Administrati
ve Costs 
linked with 
participation 
of Actions 
implemented 
by the 
Programme 
(provision of 
information 
nomination 

Costs 
linked 
with 
preparatio
n or 
response 
to call for 
tenders 

106 

all MS 

Achieving 
cost 
savings for 
the benefit 
of health 
systems 
through 
appropriate 
use of tools 
such as 
HTA and 
through 
digitalisati
on policies 

Contributio
n of MS in 
Joint 
Actions: 
MEUR 15 

EU co-
funding for 
Joint 
Actions: 
MEUR  

Total 
amount of 
appropriati
ons 
MEUR 85 

Overall 
administra
tive costs 
for 
running 
the entire 
Programm
e are 
presented 
for all 
specific 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
objectives 
and cross-
cutting 
actions 

and 
coordination 
with national 
entities 
participating 
in Joint 
actions) 

24 European 
Reference 
Networks 
(ERNs) 
established 

1185 
healthcare 
providers and 
centre of 
expertise 
joined the 
ERNs 

2,100 virtual 
expert panels 
opened in the 
CPMS 

24 
European 
Reference 
Networks 
(ERNs) 
established 

27 MS 
using the 
tools 
developed 

Benefits 

Direct benefits 

Direct benefits 

Indirect benefits 

1.7 million 
patients 
treated by 
ERN 
members 

Patients 
participated 
in 732 
clinical trials 
within the 
ERNs 

Easier access 
to treatment 
for patients 
with rare 
diseases 

Cross-cutting issues and overall Programmes costs and benefits 

Procurem
ent 
contract 
awarded: 
MEUR 27 

107 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Overall Administrative Cost 
for implementing the 
Programme 

Cost for functioning of the 
executive Agency CHAFEA 

MEUR  10 

MEUR 30 

108 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ANNEX V. STAKEHOLDERS CONSULTATION - SYNOPSIS REPORT  

1 

INTRODUCTION 

This  document  provides  a  synopsis  or  summary  of  the  consultation  activities  conducted  under 
the  Study  to  support  the  ex-post  evaluation  of  the  European  Commission’s  3rd  Health 
Programme  2014-2020.  It  was  prepared  in  accordance  with  the  Better  Regulation  Guidelines 
Tool 55.122  

This report is structured as follows: 

Section 2 presents the approach to the consultations 
Section 3 provides an overview of the results. 

2 

APPROACH TO THE CONSULTATION  

The objective of the consultations was to collect qualitative and quantitative information from 
stakeholders on their views of the 3rd Health Programme (Programme).  

2.1 

Stakeholder selection 

Eight stakeholder groups (policymakers, governmental public health organisations, international 
research  organisations,  non-governmental 
public  health  organisations,  academic  and 
organisations,  healthcare  service  providers  and  organisations  representing  them,  healthcare 
professionals’ associations, and patients and services users and organisations representing them) 
were identified by the study team. The study team then aligned these categories with the groups 
set out in the Public consultation  (PC) typology, which are already set by DG SANTE, to ensure 
comparability during the analysis and reporting phase of the study. 

Stakeholders  were identified using the public-facing  database123 for the Programme. The study 
team exported all stakeholder organisations from this database using a web scraper developed by 
the team and cleaned the dataset. This enabled us to get a longlist of stakeholder organisations 
who participated in the Programme. DG SANTE also shared the contact details of the National 
Focal Points and of some Programme Committee members who had agreed to be contacted for 
the study. While the study team web scraped the organisations of all those who received funding 
through grants, there may have been gaps in the list for stakeholders who received other types of 
funding (e.g. via Procurement Contracts). This was because such information was not stored in 
the public facing database nor was there a list that could be shared with us for the purposes of the 
consultation  exercise.  Further,  specific  named  contacts  were  not  available  in  the  public-facing 
database.  Therefore,  the  team  conducted  desk  research  to  collate  publicly  available  contact 
names and email  addresses from  the websites of the identified organisations and through other 

122 https://ec.europa.eu/info/sites/default/files/file_import/better-regulation-toolbox-55_en_0.pdf 
123 European Commission (2022) Health Programme Database. Available at: https://webgate.ec.europa.eu/ 
chafea_pdb/health/ 

109 

 
 
 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

desk review sources. In some cases, the study team asked for assistance from HaDEA and DG 
SANTE  to  review  the  list  to  ensure  it  complied  with  their  understanding  of  the  Programme’s 
stakeholders, to identify missing stakeholders and/or fill gaps in the contact details. 

The lists of stakeholders contacted can be found in Annex 1. An overview of the stakeholder 
groups invited to participate in the consultations is show in Table 2a. 

Table 2a  Summary of stakeholders contacted through the consultations 

Cross-cutting stakeholders contacted for all consultations 

Additional stakeholders 

603124 

9 

Group-specific contact for individual consultations 

Type of organisation 

Targeted survey and PC 

Interviews 

Policymakers (EU institutions, 
national government 
representatives) 

Governmental public health 
organisations 

International public health 
organisations  

Academic and research 
organisations   

100125 

23 

15 

Non-governmental organisations 

53 

Healthcare service providers and 
organisations representing them  

Healthcare professionals’ 
associations  

Patients and services users and 
organisations representing them 

14 

22 

0 

43 

83 

5 

6 

7 

4 

9 

7 

TOTAL 

830126 

173 

2.2  Consultation activities 

2.2.1 Advertising the consultations 

The study team created and used a Stakeholder Network127 on the Health Policy Platform128 to 
disseminate  information  on  the  consultation  activities.  The  study  team  copied  key  information 

124 The “additional stakeholders” from the survey were: companies/business organisations; consumer organisations; 
lead  or  partner  organisations  of  funded  actions;  EU  citizens;  independent  thematic  experts;  and  public 
authorities. 

125 Stakeholders for the survey were identified as “public authorities” 
126  This  figure  does  not  include  a  generic  mailing  sent  from  our  survey  software  tool  to  143  general  stakeholder 

contacts. These contacts were gathered as part of the general stakeholder mapping.  

127 Network gathering health stakeholders on specific health policy areas 

110 

 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

onto the AGORA network129 so that all stakeholders could have the chance to participate in the 
consultation  activities.  This  information  was  mirrored  in  the  weekly  Health  Policy  Platform 
newsletters. 

In order to increase the number of responses, communication around the consultations was brief 
and  informative,  clearly  outlining  the  importance  of  the  consultations  and  encouraging 
participation,  as  well  as  clearly  illustrating  how  to  participate.  The  communications  asked 
respondents  to  share  the  information  amongst  their  own  network,  encouraging  their  peers  to 
participate in the consultations.  

The first email sent to stakeholders included an explanation on key details including: (a) details 
of the study; (b) consent procedures for taking part in the study (it was clearly stated that taking 
part in this research was voluntary); (c) attribution of information (information and quotes were 
not  attributed  to  individuals,  unless  explicitly  approved);  and  (d)  audio-recording  of  the 
interviews  and  focus  groups  (for  accuracy  and  note-taking  purposes,  and  only  with  specific 
consent).  This was also accompanied by an accreditation letter from DG SANTE. 

As the PC and the survey were launched simultaneously, communications around these activities 
sought  to  clearly  highlight  the  difference  between  the  PC  and  the  targeted  survey  (to  ensure 
participants  were  aware  of  the  most  appropriate  method  for  them  to  provide  their  views).  In 
addition, to make sure that participants responded to the most relevant questionnaire (either PC 
or target survey), there was a filtering question at the start of both questionnaires on whether the 
participant  (or  their  organisation)  had  been  directly  involved  in  the  Programme  design  or 
implementation. If it became apparent that the respondent was using the wrong questionnaire, a 
prompt appeared encouraging them to switch to the other consultation activity. 

Table 3a details the activities undertaken to boost the response rate to the consultations. 

Table 3a – Activities undertaken to advertise the consultations 

Consultation method 
PC 

Survey 

Activities undertaken to advertise the consultation 

-  Emails from ICF to all contacts in the database collated by ICF 

-  Emails from DG SANTE to their stakeholders (including all 

members of the Programme Committee) and the contact database 
collated by ICF 

-  The Health Programme webpage: 

https://ec.europa.eu/chafea/health/funding/index_en.htm 

-  DG SANTE webpage: 

https://ec.europa.eu/info/departments/health-and-food-safety_en 

-  The Health Policy Platform: https://webgate.ec.europa.eu/hpf/ 

(and the HPP newsletter) 

-  Emails from DG SANTE to their stakeholders and the contact 

database collated by ICF 

128 European Commission (2022) Health Policy Platform. Accessible : https://webgate.ec.europa.eu/ 
hpf/#:~:text=The%20EU%20Health%20Policy%20Platform%20is%20an%20interactive%20tool%20to,actions%20
among%20a%20wide%20audience.  
129 Online space accessible to all users of the Health Policy Platform 

111 

 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

Interviews 

Focus groups 

-  Advertisements on the Health Policy Platform including the 

Stakeholder Network for the study and the Agora Network which 
were mirrored in the Health Policy Platform newsletter 

-  Emails from ICF to selected contacts in the database collated by 

ICF 

-  Asking interviewees to advertise the study to their network 

-  Emails from ICF to selected contacts in the database collated by 

ICF 

- 

- 

- 

Focus group participants sharing the invitation email with 
contacts who were involved in their funded action 

Focus group participants were given the opportunity to do a 
follow-on interview if they had further feedback to provide 

ICF worked with DG SANTE to identify the most suitable 
stakeholders for the focus group with EU institutions on 
Procurement contracts and all funding mechanisms 

2.2.2 Targeted survey 
The purpose of the targeted survey was to collect evidence on the views and perceptions of those 
with  direct  experience  of  the  Programme  regarding  its  relevance  and  implementation  and 
performance.  The  survey  was  targeted  at  those  who  were  directly  involved  in  the  Programme 
design and/ or implementation (including those who received funding from the Programme) and 
could therefore answer specific questions about Programme implementation and performance. 

Identification and recruitment of participants 
The identification and recruitment of participants was in line with section 2.2.1 on contacting 
stakeholders. 

Conducting the targeted survey 

The  targeted  survey  was  implemented  using  ICF’s  survey  platform  (Qualtrics).  Before 
participating  in  the survey, respondents  were provided with  a privacy  statement  to  ensure they 
were  informed  of  their  rights  under  the  General  Data  Protection  Regulation  (GDPR)130  in 
relation  to  the  collection  and  retention  of  their  data  and  that  they  understood  that  their 
participation was on a voluntary basis. The collected personal data and all information related to 
the consultation were recorded in a secured and protected database hosted at ICF’s secure data 
centre within the European Union. The database is not accessible from outside ICF. Inside ICF, 
the database can be accessed using a User-ID/Password. Any data transferred between ICF and 
DG  SANTE  was  done  using  the  secure  file-sharing  system  OneDrive,  which  is  produced  and 
maintained by Microsoft. 

130 https://gdpr-info.eu/  

112 

 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

The study team kept the survey open from 10 March to 13 May 2022. The survey was originally 
planned to be kept open from 10 March to 21 April. Despite multiple email reminders, there was 
a lack of  response from  stakeholders,  and so the  survey deadline was extended to  6 May. The 
study  team  sent  out  two  email  communications  to  stakeholders,  one  to  all  contacts  notifying 
them of the extension and encouraging their participation, and another tailored to those who had 
started  but  not  completed  the  survey.  For  this  latter  group,  the  study  team  offered  tailored 
support  including  organising  a  phone  call  to  fill  in  the  survey  with  them,  or  to  organise  an 
interview  instead.  The  study  team  also  provided  Word  versions  of  the  questionnaire  when 
requested  so  that  multiple  people  within  an  organisation  could  contribute  to  the  survey 
submission.  The  survey  deadline  was  then  extended  again  to  13  May  to  encourage  more 
participants to take part. All stakeholders were informed of this including those who had started 
but  not  completed  the  survey.  DG  SANTE  also  emailed  all  stakeholders  identified  in  the  ICF 
contacts file to encourage participation. 

Analysis 
The  questions  asked  in  the  survey  covered  the  following  themes:  effectiveness,  efficiency, 
relevance, coherence, and EU added value. Analysis included: cross-tabulations of closed answer 
questions and qualitative analysis of additional textual feedback provided by respondents in open 
answer  questions  and  through  position  papers  uploaded  to  support  their  responses.  Manual 
qualitative analysis was used to provide insight into the themes being discussed. 

2.2.3. Public consultation  

A  Public  consultation  (PC)  was  undertaken  to  provide  the  general  public  and  all  interested 
parties with the opportunity to provide information and opinions on the matters to be addressed 
in the study. The PC was targeted at all those who have an interest in the  Programme but who 
had  not  necessarily  been  directly  involved  in  the  Programme  design  or  implementation. 
Questions  were  therefore  relatively  high-level,  exploring  the  overall  perception  of  the 
Programme, and its relevance to broader health needs and objectives. The PC asked respondents 
to give their view on the effectiveness, efficiency, relevance, EU added value and coherence of 
the Programme.  

Identification and recruitment of participants 

Steps taken to identify stakeholders can be found in section 2.2.1. 

Conducting the PC 

The PC was carried out using the EU Survey tool. The PC was made accessible from the Health 
Policy Platform site of the European Commission: https://webgate.ec.europa.eu/hpf/. 

The  PC  was  launched  on  10  March  2022.  The  PC  was  originally  planned  to  close  on  3  June 
2022.  However,  during  the  last  2-3  days  before  closure  of  the  PC,  DG  SANTE’s  server  was 

113 

 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

down,  and  the  online  questionnaire  was  inaccessible  to  respondents.  DG  SANTE  therefore 
extended the deadline by 1 week to 10 June.  

Analysis 

The  analysis  of  the  PC  included:  cross-tabulations  of  closed  answer  questions  and  qualitative 
analysis  of  additional  textual  feedback  provided  by  respondents  in  open  answer  questions  and 
through  position  papers  uploaded  to  support  their  responses.  Manual  qualitative  analysis  was 
used to provide insight into the themes being discussed. 

2.2.4. Interviews 

Targeted  telephone  interviews  aimed  to  help  the  study  team  to  understand  in  more  depth  the 
design  and  implementation  of  the  Programme.  The  interviews  were  also  used  to  help  us 
triangulate  findings  drawn  from  other  data  collection  tasks  and  fill  gaps  in  evidence  collected 
through other tasks or where the study team identified contradictory evidence. 

Identification and recruitment of participants 

The study team selected potential interviewees based on their field of knowledge and expertise, 
their level of involvement with the Programme and on their likely ability to provide information 
on  key  issues  of  the  evaluation.  Accordingly,  interviewers  tailored  the  questions  for  each 
interview to explore specific points, rather than aiming to cover all aspects of the evaluation with 
each interviewee. In this way, the study team aimed to make maximum and efficient use of the 
time-constrained consultation period and of the resources available. 

The  study  team  planned  to  carry  out  up  to  45  interviews  and  anticipated  speaking  with 
representatives  from  eight  stakeholder  groups  to  ensure  the  consultation  activity  was 
representative of different perspectives. The study team proposed to conduct the most interviews 
with government policymakers since these include the national focal points who were pivotal in 
shaping  Programme  priorities.  The  study  team  also  proposed  to  conduct  many  interviews  with 
national  public  health  organisations  as  they  also  played  a  vital  role  in  bringing  health-specific 
knowledge and understanding at MS level. For the remaining stakeholder groups, the study team 
allocated two or three interviews each. The study team also allocated a few interviews to be used 
as needed across the stakeholder groups based on response rate and gaps in the study. 

Once  a  stakeholder  responded  to  the  invitation,  the  study  team  followed  up  with  a  short 
questionnaire to facilitate tailoring of the topic guides. The study team also followed up by email 
to  schedule  a  telephone  call  or  virtual  meeting  (as  preferred  by  the  stakeholder)  and  find  a 
suitable date and time for an interview. 

After  being  invited  to  interview,  each  stakeholder  that  did  not  respond  to  the  invitation  was 
contacted  up  to  three  additional  times.  A  detailed  log  of  all  invited  interviewees,  contacts  and 
consent  was  systematically  stored  on  password  protected  computers,  which  helped  ensure 
effective and efficient interview scheduling.  

114 

 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

Conducting the interviews 

The  interviews  followed  a  semi-structured  topic  guide,  tailored  to  the  involvement  of  the 
stakeholder  in  the  Programme.  Each  interview  was  conducted  by  phone  or  Microsoft  Teams. 
Each interview lasted approximately 40-60 minutes. The working language of the interviews was 
English.  

A high-level summary of topics to be covered in the interviews (in the form of an abridged topic 
guide) was sent to interviewees in advance, to allow time for interviewees to prepare. 

Interviews were recorded upon interviewee’s consent. Recordings were stored on secure servers 
during  the  study  to  ensure  the  completeness  and  accuracy  of  qualitative  and  quantitative  data 
collected. 

Analysis 

A summary of key points  was drafted by the interviewer after  each interview using the audio-
recording to identify specific details and obtain direct quotes where needed. Interview write-ups 
were analysed thematically in order to match points discussed in each interview to the questions 
in the study’s analytical framework. This was done per stakeholder group to allow analysis per 
group. 

2.2.5. Focus groups 

The objective of the focus groups was to gain further insight into the main funding mechanisms 
of the Programme. 

The study team conducted five focus groups, covering each of Project Grants, Operating Grants, 
Joint Actions, Procurement contracts, and a final focus group on all funding mechanisms. Due to 
a  lack  of  participation  and  availability  of  DG  SANTE  and  HaDEA  staff,  the  fifth  focus  group 
was concluded early, and follow-up interviews were scheduled instead. 

Identification and recruitment of participants 

There is no fixed ideal number of participants for a focus group, as this depends on the level of 
experience  of  the  participants,  how  sensitive  the  topic  is,  how  complex  the  questions  are,  and 
how long the team has for the discussion. For each focus group, the study team aimed to recruit 
between 5 and 12 participants. Recruiting a minimum of five participants meant  that the study 
team  had  enough  participants  to  engage  in  a  meaningful  discussion  and  gather  sufficient 
feedback  from  a  variety  of  stakeholders.  Limiting  the  focus  groups  to  a  maximum  of  12 
participants meant that participants would be more comfortable and willing to speak, that each 
participant  could  have  an  opportunity  to  share  insights  and  observation,  and  facilitators  could 
more efficiently moderate the discussion so that it stayed on topic. 

115 

 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

When a stakeholder responded to the invitation, the study team followed up with an email with 
further information including an agenda for the focus group and asking for the name and contact 
email of any colleagues who may want to attend with them. The link to join the focus group and 
a guidance note was then shared in advance with all attendees. For each focus group, the study 
team  kept  a  detailed  log  of  all  invited  participants  responses.  After  being  invited  to  the  focus 
group,  each  stakeholder  that  did  not  respond  to  the  invitation  was  contacted  up  to  three 
additional times. 

Conducting the focus groups 

The focus groups took place virtually, online. The benefit of this was that more individuals were 
able to participate and from different locations. The study team conducted the focus groups via 
Microsoft Teams and used tools such as Sli.do which allowed for instant polls, word-clouds and 
quantification of stakeholder feedback. 

In advance of the focus groups, the study team provided a guidance note to participants so they 
could consider the topics of the focus group in advance. 

The focus groups started with a presentation on the Programme with emerging findings from the 
study  to  date,  a  plenary  session  to  discuss  overall  views  of  the  Programme  and  the  specific 
funding mechanism in  question, followed by sub-groups to  discuss specific evaluation  criteria, 
and finally a plenary session to share views as a group. Each focus group lasted for up to 4 hours 
depending on the topics to be covered and the participation of attendees. 

Analysis 

Notes  about  the discussions in  the plenary sessions  and breakout  rooms  were summarised in  a 
report  for  each  focus  group.  These  reports  were  organised  by  evaluation  question  to  enable 
findings to be easily integrated into the main report. 

Limitations  

The  online  survey  and  PC  yielded  fewer  replies  than  anticipated,  despite  a  dissemination 
campaign and reminder  emails.  This  may be due to  a lack of engagement  by stakeholders and 
other contextual factors (including delays to the overall study timeline, the study being run after 
the  launch  of  the  new  health  programme,  and  thus  a  risk  of  de-prioritisation  of  the  previous 
programme).  A  larger  number  of  survey  responses  would  have  provided  greater  depth  to  the 
qualitative  analysis,  but  the  coverage  of  stakeholder  interests  was  good,  with  no  obvious  gaps 
(see section 3 for further details). 

Similarly, for the stakeholder interviews, multiple invitations were sent to stakeholders from 30 
March 2021 to 27 June2021, however targets per stakeholder group were not met for two groups: 
government policymakers, and healthcare service providers and organisations representing them. 
The target for government policymakers was 20 and 10 interviews were conducted; the target for 
healthcare service providers and organisations representing them was 2-3, and 1 interview was 

116 

 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

conducted.    While  the  target  was  to  have  45  participants  in  the  interviews,  despite  substantial 
attempts to engage with stakeholders, 34 stakeholders participated in total.  This figure includes 
follow-up interviews which were scheduled to compensate for a focus group with DG SANTE 
staff on ‘Procurement contracts’ and ‘All funding mechanisms’. This is because the participation 
from DG SANTE was limited due to lack of staff availability. 

3 

OVERVIEW OF CONTRIBUTIONS  

The  sample  of  organisations  consulted  with  covered  a  range  of  sectors,  engagement  with  the 
Programme and geographic areas. All the key stakeholder groups were covered by at least one 
activity.  

Targeted Survey 

While  no  specific  quota  was  set,  the  study  team  aimed  to  receive  at  least  70  responses  (the 
number received in the mid-term evaluation). Due to a limited engagement by stakeholders and 
other  contextual  factors,  a  total  of  32  fully  completed  responses  were  received.  Most  of  these 
came from public authorities (20 responses, 62%), half of which were from central government 
or a ministry of health (10 responses, 50%) and the other half were public health authorities or 
agencies  (10  responses,  50%).  Seven  responses  were  also  received  from  non-governmental 
organisations (22%), and five from academic/research organisations (16%). No responses were 
received from consumer organisations, or from company/business associations.  

Almost  three  quarters  of  survey  respondents  (23  responses,  72%)  worked  for  an  organisation 
focused on only one country, while the rest (9 responses, 29%) worked for an organisation with a 
pan-European or international focus.  

Almost  all  survey  respondents  were  either  directly  involved  in  the  implementation  of  the 
Programme  (16  responses,  50%),  or  stakeholders  who  benefited  from  the  Programme  (14 
responses,  44%).  Only  one  stakeholder  directly  involved  in  the  design  of  the  Programme 
responded to the survey (3%), and only one respondent said they were not directly involved in 
the  Programme  but  only  had  an  interest  in  it  (3%).  No  responses  were  received  from 
stakeholders directly involved in the evaluation of the Programme.  

Respondents who said they were directly involved in the Programme or benefited from it were 
asked about their awareness of the different types of funding instruments. Almost all said they 
were aware of Joint Actions (30 responses, 97%). Most respondents were also aware of Project 
Grants  (20,  65%)  and  Operating  Grants  (13,  42%).  However,  less  than  a  third  of  respondents 
were aware of the Health Policy Platform and Health Award/Health Prizes (11, 35%), and even 
fewer  knew  about  Direct  Grants  to  international  organisations  (8,  26%)  and  Procurements 
Contracts (7, 23%).  

Out of the 32 respondents, a majority (20 responses, 63%) had been involved in the management 
and administration of an action from the Programme (e.g. filled in an application form).  

117 

 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

Public consultation  

Whereas no specific quota was set, the study team aimed to receive  at least 133 responses (the 
number received in the mid-term evaluation).  

Due to a limited engagement by stakeholders and other contextual factors, a total of 69 responses 
were  received  to  the  PC.  Three  responses  were  identical  (including  responses  to  open-ended 
questions),  and  so  they  were  considered  as  one  response.  Analysis  therefore  focused  on  67 
responses. More than a quarter of these came from public authorities (18 responses, 27%). These 
public  authorities  were  mostly  national  (14  responses),  but  a  few  answers  were  also  received 
from  local  public  authorities  (2  responses),  as  well  as  regional  or  international  authorities  (1 
response  each).  Eleven  of  these  were  public  health  authorities  or  agencies,  and  seven  were 
central governments or ministries of health. Responses were also received from EU citizens and 
academic/research institutions (16 responses each, 24%), and from NGOs (15 responses, 22%). 
In addition, a few responses were received from companies/business organisations (2 responses, 
3%).  

Respondents came from 22 different countries (AT, BE, BG, CH, CY, CZ, DE, DK, EL, ES, FI, 
FR, HR, IE, IT, LT, PL, PT, SE, SI, SK, UK). The most commonly represented countries were 
Spain (11 responses, 16%), Belgium and Italy (7 responses each, 10%) and Poland (6 responses, 
9%).  

Just  over  half  of  respondents  said  that  they  had  applied  for  funding  from  the  Programme  (34 
responses,  51%).  Just  over  a  third  said  that  they  had  not  applied  for  funding  through  the 
Programme (23 responses, 34%) and the rest either said they were not aware, or that the question 
was not applicable to them (10 responses, 14%). More than four in ten respondents said that they 
had received funding from the Programme (28 responses, 42%). Almost half of respondents said 
they  had  never  received  funding  (30  responses,  45%)  and  the  rest  said  they  did  not  know  (9 
responses, 13%).  

The  type  of  funding  instruments  that  were  most  familiar  to  respondents  was  Joint  Actions  (26 
responses,  39%),  followed  by  Project  Grants  (20  responses,  30%)  and  Operating  Grants  (8 
responses, 12%). Only five or fewer respondents said they were familiar with the Health Policy 
Platform  and  Health  Award/Health  Prize  (5  responses,  7%),  Direct  Grants  to  international 
organisations (3 responses, 4%) or Procurement Contracts (1 response, 1%). More than half of 
respondents did not provide an answer to this question. Unsurprisingly, when asked about what 
types of funding instruments they benefited from, respondents cited the same instruments: Joint 
Actions  (20  responses,  30%),  Project  Grants  (13  responses,  19%),  and  Operating  Grants  (3 
responses, 4%).  

Interviews  

An overview of the type of stakeholders who participated in the interviews is shown in 0.  

118 

 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

Table 4a  Stakeholder interview participants 

Types of stakeholders 

Number of participants 

Policymakers (EU institutions, national government representatives)  

Governmental public health organisations  

International public health organisations   

Academic and research organisations 

Non-governmental organisations  

Healthcare professionals’ associations   

Healthcare service providers and organisations representing them 

Patients and service users and organisations representing them 

Totals 

Focus Groups 

10 

7 

2 

4 

4 

4 

1 

2 

34 

There were between 3-10 participants at each focus group (Table 5a). Overall, the groups tended 
to  be  somewhat  homogenous,  however  this  was  intentional  as  only  certain  groups  have  used 
certain mechanisms under the Programme. For example, Operating Grants were given to NGOs, 
so NGO representatives comprised most of the focus group participants on this topic. 

Table 5a.  Focus group participants 

1: Project 
Grants 

2: Operating 
Grants 

3: Procurement 
contracts 

4: Joint 
Actions 

5: Procurement 
mechanisms 

Total 

1 

1 

1 

2 

2 

Government 
policymakers 

Governmental public 
health organisations 

International public 
health organisations 

Academic and 
research 
organisations  

Non-governmental 
organisations 

Healthcare service 
providers and 
organisations 
representing them 

Healthcare 
professionals’ 
associations  

8 

1 

3 

11* 

5 

0 

5 

10 

3 

0 

8 

2 

4 

4 

0 

119 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

Patients and services 
users and 
organisations 
representing them 

Total 

7 

9 

8 

10 

3 

0 

37 

*Note the three participants in focus group 5 were also present at focus group 3, therefore the totals do 
not sum 

There  were  no  stakeholders  at  any  focus  groups  from  the  following  stakeholder  groups: 
international public health organisations, healthcare professionals’ associations, and patients and 
services users and organisations representing them. Further information about the distribution of 
participants by focus group is given below: 

•  1 (Project Grants): Attendees worked for seven different organisations and had taken part 

in seven projects. 

•  2 (Operating Grants): Three of the participants were from the same NGO. 

•  3 (Procurement Contracts): All but one participant were from DG SANTE. The remaining 

participant was from HaDEA. 

•  4 (Joint Actions): Attendees were from seven organisations (three organisations had two 

representatives each). Attendees represented seven Joint Actions. 

•  5 (All funding mechanisms): All attendees were from DG SANTE. 

4 

ANALYSIS OF THE REPLIES 

The  following  subsections  summarise  the  evidence  collected  and  analysed  across  the 
consultation activities.  

Relevance 

During  the  implementation  of  the  Programme,  the  main  health  needs  in  the  EU  related  to 
health  promotion  and  better  and  safer  healthcare.  An  interviewed  academic  /  research 
stakeholder, as well as a few NGOs, reported that the promotion of healthy behaviours (objective 
1) was a key health need in the EU.131 There were also some reported key health needs which 
related  to  objective  4  (better  and  safer  healthcare),  for  example  an  academic  /  research 
stakeholder reported that visibility of rare diseases was a key healthcare need. Some interviewed 

131  For  example,  a  stakeholder  from  an  organisation  representing  patients  and  services  users  reported  the  main 
healthcare needs included a lack of accessibility of PrEP (a preventative drug for HIV), and a need to reduce 
stigma and discrimination related to HIV and AIDS.  

120 

 
 
 
 
 
 
 
 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

stakeholders132 also reported that health and social inequalities represent a key health need in the 
EU as there are health differences across regions and socio-economic groups. There were also a 
few identified needs under objective 3 (health systems).133  

The Programme has largely been relevant to these key health needs in the EU. More than three 
quarters of PC respondents said that the Programme correctly identified the health and healthcare 
needs and problems at the time of its development, to at least a moderate extent (52 responses, 
77%).  Similarly,  a  large  majority  of  survey  respondents  said  that  all  four  of  the  Programme's 
specific objectives were relevant in relation to EU health needs at the time of the Programme's 
development.  PC  respondents  and  interviewees134  believed  that  all  four  of  the  Programme’s 
specific objectives were very relevant in relation to EU health needs. In the PC, objective 1 was 
rated as the most relevant to EU health needs (46 responses, 69%). Objective 1 was also deemed 
relevant by most survey respondents (29 out of 32 or 91%). In the survey, objective 3 was seen 
as  the  most  relevant  out  of  the  four  priorities,  with  almost  all  respondents  considering  it  was 
relevant to at least a moderate extent (31 out of 32, 97%).  

There were some factors about the Programme which enabled the Programme to address the 
most  important  health  needs.  In  interviews,  some  government  policy  makers  (at  the  regional, 
national,  and  EU  level)  reported  that  the  Programme  was  aligned  with  national-level  priorities 
potentially due to the involvement of participating countries in designing parts of Programme. In 
the focus group on procurement mechanisms, a government and policy maker reported that each 
unit  in  their  organisation  contributed  to  defining  the  Health  Programmes,  ensuring  that  health 
needs are covered throughout the Programme because all policy units are involved. 

The Programme has for the most part remained relevant to changes in health needs over time. 
In  the  survey,  respondents  were  asked  about  the  extent  to  which  the  Programme's  specific 
objectives had remained relevant over time. More than two thirds (20 responses, 67%) said that 
objective 2 had become more relevant over time, mainly due to new and emerging cross-border 
health threats during the time of the Programme135 and the severity of communicable diseases. 
This  was  a  higher  proportion  than  for  the  other  three  specific  objectives  (between  12  and  13 
responses, or between 39% and 42%). One specific change in health needs was that in 2015, the 

132  Including  an  academic  /  research  stakeholder,  a  governmental  public  health  organisation,  and  an  organisation 

representing patients and services users 

133 Including a lack of training in certain procedures or conditions or about health inequalities (academic / research 
stakeholder; healthcare professionals’ organisation) and a lack of capacity to monitor and/or respond to serious 
cross-border health threats (academic / research stakeholder; governmental public health organisation). 

134 Including  some governmental policy makers and governmental public health organisations, a few  stakeholders 
from organisations representing patients and services users, an academic / research stakeholder, a stakeholder 
from  a  healthcare  professionals’  organisation,  and  a  stakeholder  from  an organisation  representing  healthcare 
service providers 

135  A  note  in  the  survey  indicated  to  respondents  that  Covid-19  was  not  in  the  scope  of  this  study.  However, 
respondents did mention Covid-19 as a factor explaining why this specific objective became more relevant over 
time.  Other  factors  mentioned  by  respondents  included  cross-border  movement/migrations,  globalisation  and 
environmental threats. 

121 

 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

EU  was  impacted  by  increased  migration;  some  governmental  public  health  organisations  felt 
this need was addressed, however some stakeholders136 reported that refugee and migrant health 
was not a topic adequately addressed by the Programme. Another major health challenge during 
the period of the Programme was the Covid-19 pandemic, and for the most part stakeholders137 
felt  this  need  was  well-addressed.  However,  in  the  focus  groups  on  Project  Grants,  a 
governmental  public  health  organisation  reported  it  would  have  been  beneficial  if  the 
Programme had more leeway to act on unanticipated priorities through contingency funding.  

There  were  some  limiting  factors  to  the  relevance  of  the  Programme  overall.  Overall,  a  few 
stakeholders138 indicated that the objectives of the Programme were not always as aligned to key 
health needs as they could have been. A few governmental public health organisations reported 
that healthcare needs were not addressed because of factors to do with participating countries, for 
example  the  countries  were  not  involved  enough,  or  experienced  financial  difficulties  which 
hindered  their  ability  to  participate  in  activities.  Another  somewhat  commonly  reported 
limitation  to  the  relevance  of  the  Programme  was  insufficient  funding.139  Most  stakeholders 
consulted  in  the  focus  group  on  Project  Grants  considered  that  the  thematic  priorities  of  the 
Programme were too broad.  

There were a few notable topics or needs which the Programme did not adequately address. In 
the  PC,  a  large  proportion  of  respondents  said  that  some  relevant  problems  or  needs  were  not 
identified by the Programme at the time of its development (30 responses, 45%). Some consulted 
stakeholders140  did  not  feel  the  Programme  adequately  addressed  needs  related  to  health 
inequalities.  For  example,  a  few  stakeholders  from  an  organisation  representing  patients  and 
service users also reported the work on access to healthcare and health inequalities has not been 
done  comprehensively,  particularly  around  patient  empowerment.  A  few  stakeholders141  also 
reported the Programme did not adequately address mental health or wellbeing. PC respondents 
said that although the Programme acknowledged the high prevalence of mental health problems, 
they felt that the issue was not  extensively  included as a key thematic priority in  and of itself. 
They  added  that  the  Programme  could  have  been  a  key  tool  in  integrating  a  psychosocial 
approach  to  mental  wellbeing,  considering  and  linking  to  the  social  and  environmental  factors 
that  undeniably  play  a  role  in  community  positive  mental  health.  Some  stakeholders  also 

136 An interviewed academic stakeholder and participants in the focus group on Project Grants. 
137  Academic  /  research  stakeholders;  stakeholder  from  an  organisation  representing  patients  and  services  users; 

NGO. 

138  including  a  few  governmental  public  health  organisations,  a  few  NGOs,  and  a  few  healthcare  professionals’ 

organisations. 

139 Reported by an NGO and a stakeholder from a healthcare professionals’ organisation. 
140 OPC respondents; interviewed governmental public health organisation; A few stakeholders from an organisation 

representing patients and services users; academic / research stakeholders. 

141 an interviewed academic / research stakeholder and participants in the focus group on Project Grants 

122 

 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

reported the Programme did not adequately address environmental issues142, including interplays 
between the climate and health, as key health needs in the EU. 

There was clear alignment between funded actions and the specific thematic priorities set out 
by the Programme. In the survey and interviews143, a large majority of respondents said that the 
Programme's  funded  actions  were  aligned  with  the  Programme's  four  specific  objectives.  In 
particular,  85%  of  respondents  to  the  survey  said  actions  were  aligned  to  a  large  or  moderate 
extent  with  objective  1.  However,  a  national  governmental  policy  maker  reported  that  some 
objectives  of  Programme  were  implemented  or  used  more  than  others.  For  example,  actions 
related to health security (objective 2) were not used often. Two EU-level policy makers reported 
that the objectives  and thematic priorities were  very broad and  wide-reaching,  therefore it was 
not possible to address them all with the same level of intensity or funding.  

The funded actions were also aligned with the Commission’s wider priorities. In the survey, a 
large majority of respondents said that the Programme’s thematic priorities were relevant to the 
Commission's  wider  priorities  to  a  large  (36%)  or  moderate  extent  (30%).  More  than  30%  of 
respondents  said  the  Programme's  thematic  priorities  were  relevant  to  a  large  extent  to  the 
following  two  Commission's  wider  priorities:  "Promoting  our  European  Way  of  Life"  and 
"Europe  2020  strategy  for  smart,  sustainable  and  inclusive  growth".  Notably,  there  were  large 
rates of “I don’t know” responses to this question (between 10-33% of respondents varying by 
Commission priority), illustrating how those involved with Programme may not have been aware 
of the Commission’s wider priorities.  

The  Programme  has  largely  been  relevant  to  citizens’  needs.  In  the  survey,  almost  90%  of 
respondents believed that the Programme’s thematic priorities were relevant in light of citizens’ 
perceptions  of  key  health  issues  in  the  EU,  to  at  least  a  moderate  extent  (28  responses,  89%). 
Similarly, almost 90% of respondents believed that the Programme responded to citizens’ health 
needs,  to  at  least  a moderate  extent (27 responses, 86%).  However,  a national  public authority 
involved  in  the  Programme  implementation  said  that  these  were  not  relevant  at  all  due  to  a 
mismatch  of  health  priorities  between  the  Programme  and  the  national  context,  citing  that,  in 
their country, the waiting list to receive medical services was a greater problem and that this was 
not resolved by the Programme thematic priorities. Two EU-level NGOs who benefited from the 
Programme  noted  that  the  funding  opportunities  for  childhood  cancer  were  valuable  but 
insufficient to address the magnitude of the issues in this disease area. 

There  has  been  some  variation  in  the  engagement  of  citizens  with  the  Programme.  Some 
interviewees reported there were differences in the engagement of citizens in the Programme. A 
few  stakeholders  reported  that  there  were  differences  across  the  Member  States  in  the 
participation  and  effectiveness  of  work  relating  to  migrants  (academic  /  research  stakeholder). 

142 academic / research stakeholder and a governmental public health organisation. 
143  an  academic  /  research  stakeholder,  a  few  governmental  policy  makers  and  governmental  public  health 

organisations, a stakeholder from an organisation representing patients and services users. 

123 

 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

An NGO reported that Balkan countries are facing severe funding problems and are struggling 
with  more  basic  services,  and  different  countries  have  different  needs  and  interests.  Further,  a 
stakeholder from an organisation representing patients and services users reported there has not 
been much investment in disseminating the Programme, which has limited citizen engagement. 
Interestingly,  a  stakeholder  from  an  organisation  representing  patients  and  services  users 
reported that in many cases citizens may be engaged in actions of the Programme but not know 
there is a Programme behind it.  

Effectiveness 

Consulted  stakeholders  reported  that  overall,  the  Programme  contributed  to  a  more 
comprehensive  and  uniform  approach  to  health  issues  across  the  EU.  Most  targeted  survey 
respondents  (20  respondents,  63%)  and  PC  respondents  (37  responses,  55%)  believed  that 
measures implemented by Member States were aligned with the specific objectives and thematic 
priorities  of  the  Programme,  at  least  to  a  moderate  extent.  Similarly,  most  survey  respondents 
believed  that  national  programmes  and  actions  reflected  evidence  and  evidence-based 
approaches developed through Programme funding (23 respondents said this was true to at least 
a moderate extent, 72%). 

However, some limitations exist mostly due to national differences in terms of organisation of 
health  systems  and  national  priorities.  Some  stakeholders  who  contributed  to  the  PC  noted 
limitations  to  the  alignment  of  measures  implemented  by  Member  States  with  the  specific 
objectives and thematic priorities of the Programme, mostly because national health systems are 
complex and sometimes fragmented infrastructures and national priorities do not always reflect 
Programme priorities. 

Overall, consulted stakeholders reported that the knowledge produced by the Programme was 
used  in  policymaking  and  the  Programme  contributed  to  improvements  in  health  and 
healthcare  in  the  EU  and  at  Member  State  level.  For  instance,  several  interviewed  national 
policymakers  reported  that  actions  funded  under  the  Programme,  including  Joint  Actions  and 
other  funded  projects,  influenced  national  strategies,  helped  establish  national  plans  and  create 
national legislation. Moreover, 79% of survey respondents believed that the Programme actions 
led to new knowledge and evidence which were used in the development of policy and decision-
making at least to a moderate extent.  

Overall,  most  survey  respondents  believed  that  the  Programme  actions  led  to  general 
improvements  in  health  and  healthcare  in  the  EU  and  at  Member  State  level  (23 
respondents  said  this  was  true  to  at  least  a  moderate  extent,  73%).  Respondents  said  that  the 
Programme  contributed  to  improvements  mainly  in  the  following  areas:  vaccination  in  the  EU 
and at Member State level (19 respondents said this was true to at least a moderate extent, 60%), 
AMR  prevention  in  the  EU  and  at  Member  State  level  (18,  57%),  and  the  creation  of  a  well-
functioning HTA system in Europe (18, 57%).   

124 

 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

Most consulted stakeholders believed that the Programme contributed to some extent to the 
EU’s influence on health and healthcare standards, policies and practices at international 
level.  Overall,  public  authorities  surveyed  as  part  of  this  study  believed  that  the  results  of 
Programme (e.g., establishment of Joint Actions and European Reference Networks, evaluations 
and studies, establishment of EU-wide data systems) were used at an international level, and that 
the EU’s coordination with international  bodies in the field  of health had been strengthened in 
Programme  priority  areas  (18  respondents  said  these  two  statements  were  true  to  at  least  a 
moderate extent, 90%). 

Not many consulted stakeholders were familiar with the exceptional utility criteria, but those 
who were felt it increased participation from some countries. In the survey, a majority of public 
authorities  said  they  did  not  know  whether  their  Member  State  applied  for  funding  under  the 
exceptional  utility  criteria  (14  respondents,  70%).  In  the  focus  group  on  Joint  Actions,  an 
academic / research organisation stakeholder reported that in two Joint Actions they worked on, 
they used the exceptional utility criteria as they had several low Gross National  Income (GNI) 
countries  involved,  so  more  budget  could  go  to  low  GNI  countries.  Nevertheless,  some 
stakeholders  felt  the  criteria  increased  participation  from  low-GNI  countries144.  Some  public 
authorities reported factors which contributed to their country's participation, including securing 
co-financing, followed by the administrative capacity to manage actions in the Member State and 
language skills.145  

However,  there  may  have  been  some  limitations  to  the  benefits  of  the  exceptional  utility 
criteria.  In  the  survey,  only  six  academic/research  organisations  and  NGOs  (50%  of  survey 
respondents) said that the scope of the exceptional utility criteria reduced the differences in costs 
and benefits between countries. Similarly, more than half of surveyed respondents said they did 
not know whether simplification measures related to the exceptional utility criteria had reduced 
administrative costs (17 responses, 53%). Those who did provide an answer tended to say that 
these  measures  did  not  reduce  administrative  costs,  or  only  to  a  small  extent.  A  number  of 
factors determined stakeholders’ decision not to apply for funding under the exceptional utility 
criteria including that the criteria is not always easy to use146; the lack of administrative capacity 

144  “Low-GDP”  and  “high-GDP”  are  used  here  to  refer  to  countries  which  did  and  did  not  meet  the  exceptional 

utility criteria, respectively. 

145 Further, in interviews, some EU-level government policy makers felt that there were more partners participating 
low-GDP  countries  due  to  added  benefits  from  the  exceptional  utility  criteria.  A  stakeholder  from  HADEA 
mentioned  information  sessions  run  by  the  Agency  as  particularly  useful  for  alerting  potential  beneficiaries  to 
actions.  Further,  in  the  focus  group  on  joint  actions,  a  governmental  public  health  organisation  reported  that  the 
criteria  were  sensible  and  effective  for  partners  who  worked  heavily  on  the  action.  In  the  same  focus  group,  a 
stakeholder  from  a  Governmental  public  health  organisation  reported  that  the  criteria  made  it  much  easier  for 
partners  to  participate  as  a  40%  contribution  is  prohibitive  to  some  partners,  so  the  20%  level  makes  it  more 
accessible.  
146 Reported by an academic / research organisation stakeholder in the focus group on Joint Actions 

125 

 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

to manage actions in the Member State; administrative burden (once a project is up and running); 
complexity of the application process.147  

Some consulted stakeholders believed that Programme results have been published to a good 
extent.  Some  interviewed  stakeholders  and  participants  in  the  focus  group  on  Joint  Actions 
confirmed  that  Programme  results  and  outputs  were  published  on  the  HaDEA  dedicated 
database. Moreover, different stakeholders from academic institutions and national policymakers 
reported  that  scientific  publications  linked  to  Programme  actions  were  published  in  scientific 
journals.  Stakeholders  noted  that  barriers  to  accessing  results  of  funded  actions  include  that 
many deliverables were delayed due to the Covid-19 crisis, as well as a lack of clarity regarding 
where publications can be found. 

Publications resulting from the Programme are available to wider stakeholders and the public 
to  a  moderate  extent.  Most  survey  respondents  said  they  had  access  to  publications  resulting 
from  the  Programme's  actions/outcomes/results  at  least  to  a  moderate  extent  (23  respondents, 
73%).  Some  interviewed  stakeholders  and  participants  in  the  focus  group  on  Joint  Actions 
indicated  that  dissemination  activities  were  effective  in  reaching  the  scientific  community  and 
the wider public. 

However,  improvements  to  the  dissemination  of  results  are  needed.  Several  consulted 
stakeholders reported limitations to access to publications and dissemination activities, including 
a  lack  of  contact  between  researchers  and  the  private  sector;  weak  engagement  with  health 
services and healthcare professional; and lack of a systematic way to monitor the extent to which 
Programme beneficiaries disseminate findings after a project. 

Data emerging from the consultation activities shows that stakeholders have used outputs and 
results from Programme activities. Some results used by stakeholders included outputs from the 
EUnetHTA which supported legislation; results of CHRODIS and CHRODIS + which generated 
screening guidelines; outputs of the RARHA Joint Action and the Oramma projects; the Health 
at a Glance publications, chronic disease reports and reports on pharmaceuticals.  

Moreover,  Programme  results  have  been  reported  as  impactful  by  different  stakeholders,  for 
instance  by  raising  awareness  among  patients  and  healthcare  providers  in  the  field  of  digital 
health,  tackling  scepticism  and  helping  realise  a  European  Health  Data  Space;  proving  to 
ministries  of  health  the  effectiveness  of  undertaken  interventions  and  creating  an  impact  on 
citizens at the local and regional level. 

Evidence reported by stakeholders suggests that some of the previous recommendations of the 
Programme  have  been  addressed.  This  includes  strengthening  and  building  links  between  the 
Programme and wider Commission & EU policy agenda to maximise impact. There is evidence 

147 Three public authorities responding to survey and a stakeholder from an NGO in the focus group on project 
grants 

126 

 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

that there is room to improve systems for monitoring programme implementation, and thus other 
recommendations are likely yet to be implemented. 

Some consulted stakeholders felt the effects of the Programme were sustainable. In the survey, 
six  respondents  (19%)  thought  that  the  results  of  the  Programme  were  very  sustainable. 
Similarly,  some  interviewees  and  focus  group  participants  felt  the  actions  were  sustainable. 
Some  elements  or  aspects  of  the  Programme  itself  seemed  to  help  ensure  projects  would  be 
sustainable following their conclusion. For example, according to a governmental public health 
organisation,  adding  the  compulsory  work  package  4  on  sustainability148  was  a  key  success 
factor of Programme. Stakeholders from an international organisation felt that when funds were 
more  structured,  sustainability  was  more  assured149.  Another  key  to  ensuring  sustainability 
seemed  to  be  the  relationships  and  connections  built  through  an  action.150  Some  results  have 
been sustainable because of the actions of participating countries.151  

Some consulted stakeholders mentioned specific topics which were seen as having particularly 
high  sustainability.  Survey  respondents  highlighted  the  following  specific  fields  as  having 
achieved  most  sustainability:  Health  Technology  Assessments  (8  responses,  25%),  vaccination 
policies (5, 16%) and antimicrobial resistance (4, 13%). According to an NGO and an EU-level 
government policymaker, the Programme increased knowledge and skills in crisis preparedness 
for  professionals  in  the  health  sector  and  NGOs,  and  evidence  generated  on  this  topic  will  be 
used  beyond  the  Programme.  There  were  also  certain  funded  actions  which  seemed  to  be 
especially  sustainable,  including  many  Joint  Actions.152  A  few  stakeholders  also  highlighted 
sustainability  within the European Reference Networks, and finally, the SCIROCCO153  project 
has created sustainable outputs. 

However, there were some reported challenges to sustainability.  

148 work package introduced as part of all funded actions to ensure demonstration of sustainability after funding 
period ends 
149 Further, a stakeholder from DG SANTE highlighted that implementation of best practices are a positive way of 
ensuring sustainability as it is moving from an older to a newer system. Related to best practices, in the focus group 
on joint actions, an academic/research organisation stakeholder discussed how the transfer of good practices to other 
regions  across  Europe  needs  to  be  supported  by  guidelines  which  in  turn  support  knowledge  from  expert 
beneficiaries.  Overall,  a  stakeholder  from  a  governmental  public  health  organisation  reported  that  there  was 
increased planning around sustainability from Member States and the Commission regarding the Programme. 
150   Reported by academic/research stakeholders and a governmental public health organisation  
151   An EU-level government policy maker felt that the Programme allowed Member States to see whether actions 
are  suitable  and  if  they  are,  they  can  apply  for  other  funding,  and  indeed  an  interviewed  academic/research 
stakeholder stated that many projects received more funding to continue beyond the Programme. In the focus group 
on joint actions, an academic/research organisation stakeholder highlighted policy dialogues as a useful approach to 
make  actions  more  sustainable,  commenting  on  good  buy-in  from  policymakers  in  member  states.  As  a  specific 
example of Member States creating sustainability, an EU-level government policy maker mentioned that Member 
States drafted and introduced their national cancer strategies following Programme.  
152 EUnetHTA, AMR Joint Actions, Joint Action on alcohol, Shipsend JA, CHRODIS and CHRODIS+, JA Healthy 
Gateways, and a joint action involving promotion of policy dialogues for media advertising beverages and food for 
children. 
153 Scaling Integrated Care in Context 

127 

 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

For purposes of this study, we consider sustainability to mean the extent to which the results of 
funded actions are likely to last once funding from the Programme has ceased. Most respondents 
thought the results of the Programme were somewhat sustainable (21 responses, 66%). Further, 
two stakeholders154 reported that results from Programme were not always integrated into policy. 
It  seems  that  some  threats  to  sustainability  were  regarding  issues  with  sustained  funding 
including  due  to  a  lack  of  permanent  funding  in  the  EU  budget.155  There  may  also  be  barriers 
related  to  political  will  or  interest  to  continue  with  work;  some  survey  respondents  stated  that 
results might not be used nor capitalised on fully by Member States due to a lack of interest and 
involvement  from  national  authorities  which  leads  to  results  of  funded  actions  remaining  at  a 
local level. Some stakeholders156 also mentioned that the design of the programme or actions did 
not lend themselves to increasing sustainability: in the survey, an EU public authority involved 
in the Programme design explained that results were mostly too limited in scale and/or ambition 
to  be  sustainable,  and  that  sustainability  was  not  “in  the  DNA  of  the  Programme  or  the 
participants”. 

Efficiency 

The  Programme  was  largely  viewed  as  cost-effective,  with  high  quantities  of  work  achieved 
with a low budget.  Generally, stakeholders interviewed across all groups, actions, and funding 
mechanisms  felt  that  the  Programme  was  relatively  cost-effective,  with  many  highlighting  the 
quantity  and  quality  of  work  achieved  with  a  small  budget.  Respondents  to  the  stakeholder 
survey  considered  some  costs  associated  with  the  Programme  to  be  reasonable  and  kept  to  a 
minimum necessary to achieve expected results. Deemed to be most reasonable (to a moderate 
extent) by those who provided an answer were management costs for funding (10 respondents, 
50%)  and  Programme  operational  costs  (design  and  implementation)  (8  respondents,  40%). 
Several  stakeholders  reported  achieving  more  than  they  expected  to  with  the  funding  they 
received, and those who had worked on Joint Actions particularly stressed the cost-effectiveness 
of  their  work.157  In  the  survey,  internal  factors  which  positively  influenced  the  Programme's 
results were identified as collaboration between Member States and development of guidance to 
assist funding applicants (22 responses each, 69%), followed by facilitation/coordination of the 
Programme  by  DG  SANTE/CHAFEA  (20  responses,  63%).  External  factors  that  positively 
influenced the Programme's results were identified as science and technological progress in the 
area of health and healthcare (25 responses, 79%), followed by solutions developed at national 

154 A stakeholder from an organisation representing patients and services users in interview and a governmental 
public health organisation in the focus group on project grants 
155 A stakeholder from DG SANTE. 
156   EU public authority involved in the Programme design, EU-level government policymaker, NGO, government 
public health organisation, Healthcare Professionals’ Association. 
157 Including a stakeholder from a governmental public health organisation and a government official/policymaker 

128 

 
 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

level,  or  by  private  or  non-for-profit  actors  (19  responses,  60%)  and  changes  in  citizens’ 
opinions or perspectives on health systems (13 responses, 41%).   

While  there  were  some  deviations  from  the  originally  planned  resources,  this  was  expected 
and not seen as an issue. Some stakeholders reported deviating from planned resource budgets 
due to personal costs, partners leaving the project, the Covid-19 pandemic, lack of Member State 
capacity, or changing priorities over the course of the action. However, due to the high number 
of partners involved in some actions and the duration of projects, it was generally expected that 
changes would be made to budgets. Several stakeholders (particularly those involved in Project 
Grants  and  Joint  Actions)  were  grateful  that  the  budget  could  be  changed  without  having  to 
request an amendment from HaDEA; budgets were permitted to be transferred between allowed 
institutions at a certain percentage because of Covid-19. The flexibility of the management of the 
budget in the Programme was thus identified as a key success in cost efficiency.  

There  were  some  factors  of  the  Programme  which  meant  that  costs  may  not  have  been 
proportional  to  all  benefits.  A  large  proportion  of  respondents  said  some  types  of  costs  were 
either not reasonable or only to a small extent: administrative costs for applicants and CHAFEA 
(8 responses, 40%), and monitoring and reporting costs for Member States and the Commission 
(5 responses, 25%). Factors influencing any disparities between Programme funded action costs 
and the expected results were to do with additional costs associated with Programme preparation, 
coordination, administration, and delivery, followed by additional costs for personnel, and, to a 
lesser extent, by additional costs for materials.  

According to survey respondents, external factors which had a negative influence on the results 
of the Programme included changes in  prevalence and severity of communicable diseases,  and 
the demographic context affecting health and sustainability of health systems (9 responses each, 
29%),  followed  by  new  and  emerging  cross-border  health  threats  during  the  time  of  the 
Programme (6 responses, 19%).2 Stakeholders in interviews and focus groups stressed the need 
for  tasks  to  be  commensurate  with  the  budget  if  costs  are  to  be  outweighed  by  benefits. 
Stakeholders  who  had  utilised  Project  Grants  highlighted  that  projects  focused  on  scaling  up 
findings,  exchange,  and  promotion  of  best  practices  needed  to  receive  more  support  that  is 
proportional to the expected benefits. Stakeholders in the focus group on Operating Grants also 
underlined that appropriate budgets attract good candidates.  

Evolving and established procedures of the programme impacted its efficiency. The number of 
partners  involved  in  an  action  was  identified  by  stakeholders  as  a  factor  which  sometimes 
negatively influenced the efficiency with which achievements were attained.158 Another factor in 

158 A government official/policy maker in the focus group for Procurement Mechanisms mentioned that there were 
too  many  partners  in  the  HTA  Joint  Action  and  suggested  that  the  maximum  number  of  partners  should  be 
indicated by the European Commission in the eligibility criteria. This sentiment was echoed by a stakeholder 
from  the  government  officials  and  policymakers’  group,  who  reported  difficulty  in  coordinating  a  number  of 

129 

 
 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

is 

international  organisations 

observed disparities between costs and benefits was co-funding, for example a stakeholder from 
an  international  organisation  reported  that  40%  co-funding  within  a  Direct  Grant  Agreement 
from 
future 
collaborations.159 Several stakeholders  also mentioned that timing of projects and funding caps 
could  negatively  impact  the  efficiency  with  which  achievements  are  attained,  for  example  the 
projects  had  short  durations  and  insufficient  accountability  mechanisms.160  More  emphasis  on 
planning for sustainability before an action begins and sustainability mechanisms when an action 
ends  was  perceived  as  a  way  to  marry  costs  with  benefits.  Available  financial  and  human 
resources was identified as a defining factor in efficiency of achievements.  

'unbearable'  and  may  negatively 

impact 

There were few stakeholders who reported that programme objectives were unmet or partially 
unmet.  Further,  programme  credits  were  distributed  efficiently  between  the  four  thematic 
priorities. A majority of stakeholders consulted in interviews and focus groups felt that there was 
an  efficient  distribution  of  Programme  credits  among  the  four  thematic  priorities  and  several 
stakeholders  mentioned  priorities  being  in  line  more  widely  with  EU  objectives.  For  example, 
those who had received Operating Grants largely agreed that they were in line with Programme 
objectives, and stakeholders in the Procurement Contracts focus group also felt that the funding 
was aligned with EU objectives. 

Most  stakeholders  considered  funding  allocation  to  be  critical  to  achieve  expected  results. 
Stakeholders from the Operating Grants focus group felt that funding allowed them to plan and 
deliver on projects with (financial) security. Those in governmental public health organisations 
also emphasised how invaluable funding was to achieving results: one stakeholder reported that 
funds would not have been directed to the identified priorities without Programme, and another 
stakeholder  from  the  same  group  highlighted  that  funding  was  critical  for  enabling  low-GNI 
countries  to  achieve  results  with  other  Member  States.  A  stakeholder  from  a  healthcare 
professionals’  association  also  highlighted  how  external  stakeholders  would  not  have  been 
engaged in innovations in healthcare systems in the same way without the funding. 

There  were  some  differences  in  costs  between  countries,  caused  by  several  factors.  Some 
academic/research  organisations  and  NGOs  believed  there  were  differences  in  costs  between 
countries. Some respondents believed these differences were caused by differing staff expenses, 
which impacted achievable goals and work performance. Some stakeholders also identified cost 

entities  working  on  a  Joint  Action.  If  one  partner  does  not  submit  cost,  then  it  cannot  be  funded  by  the 
Commission. 

159  Further,  a  stakeholder  from  an  NGO  who  utilised  a  Project  Grant  also  felt  that  the  standard  co-funding 
requirement (40%) is too high and the application process for the 80% co-funding is difficult, particularly for 
smaller  organisations.  A  stakeholder  from  the  same  focus  group  in  an  academic/research  organisation  also 
discussed the difficulty of co-funding in ERNs (but acknowledged this as a point which has been addressed in 
EU4HEALTH).  Similarly,  a  stakeholder  from  a  governmental  public  health  organisation  stressed  that  20% 
commitment to their own funding was challenging for NGOs and may prevent them from contributing despite 
the value they add to projects. 

160 reported by a stakeholder from a government public health organisation 

130 

 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

differences  which  could  be  linked  to  the  Programme.161  Survey  results  suggested  a  larger 
perceived difference in benefits gained through the Programme; stakeholders believed that tasks 
and  the  level  of  involvement  of  Member  States  in  projects/actions  dictated  to  what  degree 
countries benefited from the Programme. The exceptional utility criteria were perceived by half 
of respondents as a factor which reduced differences in costs and benefits. Other factors affecting 
differences were identified in the survey as: organisational capacity to deliver funded actions (8 
responses,  67%),  administrative  burden  of  applying  for  and  receiving  funding  (7  responses, 
59%), and countries' public health capacity to apply for and manage funding (6 responses, 50%). 
Stakeholders  interviewed  identified  geographical  location  as  another  factor  which  caused 
differences in benefits between participating countries.162  

While some respondents did not know much about the simplification measures, those who did 
generally felt they contributed to the efficiency of the Programme. 32% of respondents in the 
survey  did  not  know  whether  the  simplification  measures  contributed  to  the  efficiency  of  the 
Programme,  and  those  who  did  answer  were  divided.  Ways  in  which  simplification  measures 
were  deemed  to  be  efficient  were  in  the  introduction  of  electronic  tools  for  the  submission  of 
proposals,  management  of  grants  and  e-reporting  and  monitoring  (subject  to  the  system 
functioning  efficiently),  the  introduction  of  a  negotiation  process  for  Joint  Actions,  and  the 
ability for beneficiaries to transfer resources between different cost categories without the need 
for  an  amendment.  Stakeholders  consulted163  generally  felt  that  simplification  measures  had 
reduced  administrative  costs  and  improved  efficiency  of  the  Programme,  for  example  by 
reducing paperwork and improving operational running of Joint Actions.164  

Many  stakeholders  felt  that  there  was  further  scope  to  reduce  costs.  Most  proposed  changes 
were related to application processes.165 Further simplification of administrative processes was a 
common  improvement  suggested  by  stakeholders.  One  stakeholder  from  a  healthcare  service 
provider/  organisation  representing  them  praised  the  reduction  in  paperwork,  but  felt  that 

161 A stakeholder from a governmental public health organisation felt that the divergence in daily payment amounts 
from participating countries in Joint Actions should be reconsidered. One stakeholder from a healthcare service 
provider/ organisation representing them highlighted that payments to those working on projects were adjusted 
to countries' levels, meaning different people were paid different amounts.  

162  Stakeholders  in  focus  groups  on  Joint  Actions  and  Operating  Grants  particularly  felt  that  Western  European 
countries were still overrepresented in the Programme, and a stakeholder in the focus group on Project Grants 
felt that this was due to the difficulty faced by low GNI countries in meeting co-funding rates. A stakeholder 
from a governmental public  health organisation highlighted that countries with low GDP struggled to see the 
same  benefits  of  Joint  Actions  due  to  not  having  the  resources  and  capacity  to  participate.  However,  one 
stakeholder from the government officials/policy makers group highlighted that even with more funding, some 
countries still would not have had the resources or capacity to participate effectively in the Programme.  

163 Including in the Project Grants focus group and the Operating Grants focus group. 
164 Reported by a stakeholder from research/academic organisation 
165  One  government/policy  maker  stated  that  more  flexibility  was  still  needed  in  Project  Grant  funding  for  Joint 
Actions.  The  stakeholder  worked  on  a  Joint  Action  on  vaccination  where  the  Ministry  of  Health  were 
nominated as the competent authority to work with a university, but they were not able to justify the affiliated 
entity  aspect  of  the  university.  Another  stakeholder  from  a  healthcare  professionals’  association  felt  that 
applications for ERNs should not be annual, but every 5-10 years to reduce administrative burden.   

131 

 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

locating documents was difficult, especially when trying to find out why a project was declined. 
Another  stakeholder  from  a  healthcare  professionals’  association  highlighted  that,  prior  to  the 
pandemic,  some  financial  officers  required  face-to-face  meetings  whereas  others  allowed  e-
meetings; this affected the financing as travel expenses were difficult to cover. 

Monitoring costs were largely seen as reasonable and cost-effective. A majority of respondents 
who were involved in the management and administration of an action from the Programme said 
that the monitoring costs were reasonable and kept to the minimum necessary in order to achieve 
the  expected  results,  at  least  to  a  moderate  extent  (11  responses,  55%).  According  to  some 
respondents, the key factors enabling efficiency were the relevance of indicators (10 responses, 
50%) and the level of clarity of the indicators (9 responses, 45%).  

However, there were a few challenges with the monitoring processes. While some stakeholders 
noticed  improvements  in  the  monitoring  process,  many  still  felt  it  could  be  simplified 
(particularly stakeholders from healthcare professionals’ associations and NGOs). A stakeholder 
belonging  to  the  governmental  policymakers  group  mentioned  that  experts  they  worked 
alongside  struggled  with  the  budgeting  table  due  to  uncertainty  and  level  of  detail  required. 
Increased dissemination of project information was seen to make the Programme more efficient 
as  a  whole.  Communication  within  project  teams  was  also  highlighted  as  key  to  having  an 
efficient  monitoring  process.166  Some  stakeholders  suggested  different,  more  efficient  methods 
of monitoring the Programme.167  

Reporting  systems  were  seen  as  reasonably-priced  and  beneficial.  In  the  survey,  respondents 
involved  in  the  management  and  administration  of  an  action  reported  benefits  of  the  reporting 
system,  including  allowing  the  tracking  of  progress  on  actions  against  their  original  plan  (11 
responses, 55%), increasing visibility of the Programme and its actions (6 responses, 30%) and 
allowing  Programme  participants  to  manage  budgets  effectively  (5  responses,  25%).  Eight 
respondents  (40%)  said  that  the  costs  of  the  reporting  system  were  reasonable  and  kept  to  the 
minimum  necessary  to  achieve  expected  results,  at  least  to  a  moderate  extent.  Interviews  with 
stakeholders revealed perceived benefits of the reporting system.168  

166 reported by stakeholder from an academic/research organisation 
167 Several government and policy makers in the Procurement Mechanism focus group highlighted the difficulty of 
measuring/monitoring  impact  of  funding  as  there  is  no  specific  framework  for  measuring  results  of  activities 
and quantifying progress is challenging. One stakeholder from the group suggested that operational units should 
put emphasis on what is the best that can be achieved with the available budget at the beginning as a better way 
of  monitoring.  Another  stakeholder  (from  an  NGO)  suggested  monitoring  progress  through  ‘looking  at  how 
actions affect communities – assessing to what extent is everyone at the table’. One government official/policy 
maker  stakeholder  reported  that  there  is  a  need  for  a  dedicated  data  collection  system  to  perform  monitoring 
activities  per  objective  and  per  priorities,  as  there  is  currently  a  missing  link  between  individual  projects  and 
specific objectives and thematic priorities. 

168  Such  as:  the  portal,  which  made  reporting  more  efficient  (according  to  a  government  official/policy  maker); 
Compass and SYGMA reporting systems, which enabled beneficiaries to report back to the Commission with 
less administrative burden and to track projects from start to end (according to a government/policy maker); and 

132 

 
 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

However, there were some limitations of the reporting system identified. 65% of respondents 
felt  that  the  reporting  system  could  be  improved  by  'simplifying  the  reporting  procedure’ 
(through reducing administrative burden, time and effort required). In line with survey findings, 
a  few  stakeholders  consulted  in  focus  groups  and  interviews  indicated  that  reporting  systems 
could be more effectively implemented in the Programme. Suggested improvements were related 
to reducing administrative burden on applicants; stakeholders in the Project Grants focus group 
highlighted  the  need  to  reduce  the  level  of  detail  required  for  financial  reports,  and  two 
stakeholders  from  NGO  and  organisations  representing  patients  and  service  users  groups 
mentioned  the  administrative  burden  of  submitting  Operating  Grant  reports  specifically.  For 
smaller organisations without technical capacity and knowledge, the administration involved in 
Operating  Grant  reports  was  discouraging  according  to  a  stakeholder  from  organisations 
representing patients and services users. The stakeholder from an NGO also felt that submitting a 
few smaller Operating Grant reports throughout the year as opposed to one big report annually 
may be more efficient. 

EU added value 

The  Programme  achieved  more  added  value  than  what  Member  States  could  have  achieved 
acting individually. Most respondents to the PC169 considered that Programme provided added 
value  beyond  what  could  have  been  achieved  by  Member  States  acting  alone.  The  additional 
value of having  an EU health programme was also validated by stakeholders who attended the 
focus  group  on  Joint  Actions.  Stakeholders  representing  governmental  public  health 
organisations mentioned that the Programme enabled partners to contact other EU organisations 
and use that support to have a greater impact at national level. Furthermore, these stakeholders 
considered that it was beneficial to have actions at regional, national and EU level depending on 
the level of devolution within a country. 

Coherence 

The  activities  carried  out  under  Programme  were  aligned  with  the  thematic  priorities  of  the 
Programme.  The  majority  of  survey  respondents  mentioned  that  there  were  consistent  and 
coherent  funding  decisions  across  specific  objectives  during  the  Programme  period.  Very  few 
respondents to the survey said that funding decisions were not at all coherent with the specific 
objectives.  For  example,  just  2  respondents  to  the  survey  (7%)  said  this  was  the  case  for 
objective  2,170  and  another  2  respondents  for  objective  3.171  There  were  very  few  stakeholders 
reporting inconsistencies between actions, or gaps, duplications or contradictions, which lead to 
inefficiencies) (2 respondents to the survey, 4%).  

the  role  of  FPA  and  SGA  in  reducing  administrative  burden  for  applicants  and  the  European  Commission 
(according to a government/policy maker). 

169 Almost four in ten respondents to the OPC said the Programme provided high added-value (26 responses, 39%) 
and an additional third said that it provided moderate added-value (23 responses, 34%). 
170 Objective 2: Protect Union citizens from serious cross border health threats" (2 respondents 
171 Objective 3: Contribute to innovative, efficient, and sustainable health systems 

133 

 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

Reasons given by stakeholders for a inconsistencies, gaps, duplications or contradictions within 
the  Programme  detailed  that  these  were  mainly  due  to:  issues  linked  with  the  relationships 
between  different  actors/beneficiaries;  programme  management  and  communication  with  core 
stakeholders; and the lack of national political uptake or capitalisation of findings arising from 
the Programme funding actions.  

The Programme complemented and created synergies with other EU Programmes. A majority 
of  PC  respondents  believed  that  the  Programme  complemented  and/or  created  synergies  with 
other  EU  programmes  or  with  wider  EU  policies,  to  at  least  a  moderate  extent  (37  out  of  67, 
responses,  55%).  These  respondents  explained  that  the  Programme  was  coherent  with 
contributions  of  the  European  Structural  and  Investment  Funds  (ESIF),  the  Horizon  2020 
Programme  and  the  European  Social  Fund.  They  added  that  complementarities  between  the 
Programme and these other EU policies made it possible to investigate some topics (e.g. chronic 
diseases, non-communicable diseases, rare diseases) in-depth. 

The  Programme  was  coherent  with  other  EU  policies  in  the  field  of  health.  Several 
stakeholders  representing  government  and  policymakers,  academic  and  research  organisations 
and  governmental  public  health  organisations  agreed  that  the  Programme  was  aligned  and 
coherent  with  other  EU  policies  in  the  field  of  health.  Furthermore,  some  stakeholders 
representing  government  and  policymakers  and  governmental  public  health  organisations 
highlighted the alignment with other EU funded actions and policies such as Horizon 2020, and 
the Farm to Fork Strategy. 

However,  some  respondents  felt  the  Programme  was  not  as  coherent  with  other  EU 
programmes. Some PC respondents (13 out of 67, 19%) said the Programme was not coherent 
with other EU programmes or with wider EU policies, with one NGO noting very few synergies 
for  instance  between  the  Programme  and  the  Horizon  2020  programme  for  R&D  and  a  public 
authority explaining that programmes were not interlinked with no joint funding possible. This 
public authority added that priorities as well as grants and tenders from other EU Programmes 
were  often  not  known  to  delegates  of  the  Programme.  Furthermore,  some  stakeholders 
representing government and policymakers, and organisations that represent patients and service 
users mentioned that coherence and synergies between the Programme and Horizon 2020 could 
have  been  improved.172  Furthermore,  looking  at  the  relation  between  the  Programme  and  EU 
financial  instruments,  some  stakeholders  representing  government  and  policy  makers  and 
organisations  representing  patient  and  service  users  were  asked  if  the  health  programme 

172  They  reported  that  synergies  between  Joint  Actions  and  Horizon  were  difficult  to  unlock  because  the  latter 
programme  is  more  research  oriented.  Further,  there  could  have  been  more  coherence  between  the  funded 
actions  and  the  research  programme  (Horizon  2020).  The  stakeholder  did  not  see  any  methods  to 
motivate/promote synergies with the actions funded under the programme to feed into others. She pointed that 
some recipients of these programme acknowledge these synergies and made the best use of them.  But  in her 
view, there was not a consistent effort to connect projects and/or programmes. 

134 

 
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

encouraged  cooperation  with  the  European  Structural  and  Investment  Funds  (ESIF)  and  other 
EU financial instruments. Stakeholders agreed there was a room for further cooperation between 
the Programme and the ERDF, the European Social Fund Plus (ESF) and ESIF.  

Funded  actions  within  the  Programme  contributed  to,  and  were  aligned  with,  wider  EU 
policies.  This  was  confirmed  by  some  stakeholders  representing  NGOs, 
international 
organisations  and  organisations  representing  patients  and  service  users.  For  example,  one 
stakeholder  representing  NGOs  highlighted  that  there  was  alignment  between  the  Programme 
and  other  EU  funding  mechanisms  in  relation  to  migrant  health.  Furthermore,  a  government 
policy maker that attended the focus group on Procurement Contracts mentioned that the work of 
the Programme in the migration crisis was linked to the wider EU policy on the migration crisis. 
This work therefore contributed to achieving a specific objective of the Programme as well as a 
wider EU priority.  

The Programme was aligned with national priorities. In the PC, a majority of public authorities 
said  that  the  Programme  was  aligned  with  and  addressed  national  health  priorities  during  the 
Programme period to at least a moderate extent (14 out of 20 responses, 70%). Additionally, half 
of  respondents  to  the  PC  believed  that  the Programme  complemented  and/or  created  synergies 
with  national  initiatives  and/or  programmes,  to  at  least  a  moderate  extent  (33  out  of  67 
responses,  49%).  Several  stakeholders173  agreed  that  the  Programme  priorities  and  objectives 
were aligned with Member State initiatives in the field of health. Among the national initiatives 
that  were  aligned  with  the  Programme,  stakeholders  mentioned  actions  on  tobacco  use  and 
alcohol abuse by young people, obesity, and the prevention of frailty. Similarly, three out of five 
stakeholders  that  attended  the  focus  group  on  Joint  Actions  indicated  that  the  Programme’s 
interventions were complementary to other EU or Member State initiatives in the field of health. 
However, a few other respondents to the  PC (13 responses, 19%) said the Programme was not 
coherent with national initiatives and/or programmes. 

5 

FEEDBACK ON THE CONSULTATION PROCESS  

During the consultation period, a few stakeholders noted that they were unable to participate in 
the interviews and focus groups due to competing priorities including responding to the effects 
of the Covid-19 pandemic and the conflict in Ukraine. Lastly, due to technical issues on the last 
day of the  PC, one stakeholder  was initially unable to  submit their  response to  the survey and 
PC, however this was addressed by extending the deadline for these consultation activities.  

173 Stakeholders from the following groups mentioned this: stakeholders representing government policy makers, 
governmental public health organisations, NGOs, academic and research organisations, and organisations 
representing patients and service users 

135 

 
 
  
 
 
Ex-post evaluation of the European Commission’s 3rd Health Programme 2014-2020 – Synopsis Report 

6 

USE OF THE INFORMATION GATHERED 

All of the information gathered as part of the stakeholder consultations was firstly converted into 
useable units of analysis. For example, interview audio recordings were used to write notes for 
each interview to summarise key points and quotes. A summary of key findings per evaluation 
question was written for each focus group. Summary tables and graphs were created per question 
for the PC, and stakeholder survey and open-text responses were collated into a file. 

Then, these data sources were analysed to identify patterns and trends across stakeholder groups. 
These data sources were used to examine each evaluation question alongside the desk research 
conducted  for  this  study,  and  to  reach  the  conclusions  and  recommendations  contained  in  the 
final study.  

136 

 
 
 
 
ANNEX VI. THEMATIC PRIORITIES OF THE 3RD HEALTH PROGRAMME 

Promote health, prevent diseases and foster supportive environments for healthy lifestyles 

1. 
1.1.  Risk factors such as use of tobacco and passive smoking, harmful use of alcohol, unhealthy 
dietary habits and physical inactivity 
1.2.  Drugs-related health damage, including information and prevention 
1.3.  HIV/AIDS, tuberculosis and hepatitis 
1.4.  Chronic diseases including cancer, age-related diseases and neurodegenerative diseases 
1.5.  Tobacco legislation 
1.6.  Health information and knowledge system to contribute to evidence-based decision-making 

2.  Protect Union citizens from serious cross-border health threats 
2.1.  Additional capacities of scientific expertise for risk assessment 
2.2.  Capacity-building against health threats in Member States, including, where appropriate, 
cooperation with neighbouring countries 
2.3. Implementation of Union legislation on communicable diseases and other health threats, 
including those caused by biological and chemical incidents, environment and climate change 
2.4 Health information and knowledge system to contribute to evidence-based decision-making 

3.  Contribute to innovative, efficient and sustainable health systems 
3.1.  Health Technology Assessment 
3.2.  Innovation and e-health 
3.3.  Health workforce forecasting and planning 
3.4.  Setting up a mechanism for pooling expertise at Union level 
3.5.  European Innovation Partnership on Active and Healthy Ageing 
3.6.  Implementation of Union legislation in the field of medical devices, medicinal products and 
cross border healthcare 
3.7.    Health information and knowledge system including support to the Scientific Committees 

set up in accordance with Commission Decision 2008/721/EC 

4.  Facilitate access to better and safer healthcare for Union citizens 
4.1.  European Reference Networks 
4.2.  Rare diseases 
4.3.  Patient safety and quality of healthcare 
4.4.  Measures to prevent antimicrobial resistance and control healthcare-associated infections 
4.5.  Implementation of Union legislation in the fields of tissues and cells, blood, organs 
4.6.  Health information and knowledge system to contribute to evidence-based decision-making 

137 

 
 
 
 
ANNEX VII. KEY MONITORING INDICATORS OF THE 3RD HEALTH PROGRAMME 

General Objective 1: to complement,  support and add value to the policies of the Member States to improve the 
health  of  Union  citizens  and  reduce  health  inequalities  by  promoting  health,  encouraging  innovation  in  health, 
increasing  the  sustainability  of  health  systems  and  protecting  Union  citizens  from  serious  cross-  border  health 
threats 

Indicator 1: Number of Healthy Life Years at birth 

BaseLine 
Men 

2012 
61,0 

Women 
2012 
61,7 

2014 

2015 

2016 

2017 

2018 

2019 

2020 

2021 

2022 

Target 

61,1 

62,4 

63,6 

61,4 

63,3 

64,4 

Milestones 

Actual Progress 
63,7 

63,5 

64,2 

Milestones 

Actual Progress 
64,2 

64,3 

65,1 

2020 

Final 

2020 

Final 

63,5 

64,5 

Are we on track 
Indicator type 
Unit of measurement 
Cut-Off Date 
Data source 

Moderate progress 
Input 
Healthy years 
31/05/2021 
Eurostat. Data for 2017 will be made available by Eurostat in May 2019, and for 
year 2018 in May 2020, and so on. 
Baseline: Men 2012: 61.0%, Women 2012: 61.7% 

Narrative 
Methodology 
Link MFF 14-20 / MFF 21/27 
Other methodological comments  The above indicator is taken directly from the 3rd Health Programme Regulation. 
The 2 indicators below are an operationalization. Note: the  guidelines on breast 
cancer services were published in July 2017. Plus 3 announced in 2017 but not 
approved by the end of the year. The above indicator is taken directly from the 
3rd Health Programme. 

Full  metadata  available  at  this 
adress 
Justification of the trend 

General positive trends for the  2 indicators (men and women) even if there has 
been stabilisation or slight decrease for women over the period 2016-2018 (to be 
confirmed by future revisions of these data) 

Specific Objective 1: identify, disseminate and promote the uptake of evidence-based and good practices for cost-
effective health promotion and disease prevention measures by addressing in particular the key lifestyle related risk 
factors with a focus on the Union added value 

138 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Key monitoring indicators of the 3rd Health Programme 

Indicator 1: number of Member States involved in health promotion and disease prevention, using evidence-based 
and good practices through measures and actions taken at the appropriate level in Member States 

BaseLine 
Fat 

2012 
12 

Cancer 
2012 

2014 

2015 

2016 

2017 

2018 

2019 

2020 

2021 

2022 

Target 

16 

21 

18 

21 

20 

21 

0 

0 

0 

Milestones 
24 
Actual Progress 
24 

22 

22 

25 

24 

Milestones 

10 

10 

Actual Progress 
10 

6 

28 

24 

28 

6 

2020 
28 
Final 

2020 
28 
Final 

Are we on track 
Indicator type 
Unit of measurement 

Cut-Off Date 
Data source 
Narrative 
Methodology 
Link MFF 14-20 / MFF 21/27 
Other methodological comments 

On track 

Member States having a national initiative on reduction of saturated fat & MS in 
which  the  European  accreditation  scheme  for  breast  cancer  services  is 
implemented - establishment of the scheme. 

(relevant  for  Cancer  sub-indicators):  In  2019,  Guideline  developers  and/or 
national  authorities  from  six  EU  Member  States  have  used,  implemented  or 
adapted  the  European  Guidelines,  evidence  base  or  methodology  developed  or 
applied by the European Commission Initiative on Breast Cancer, coordinated by 
the Joint Research Centre (JRC). In 2019, guidelines developers and/or national 
authorities  of  six  Member  States  have  used,  implemented  or  adapted  in  their 
national  cancer  plans  the  European  Guidelines,  evidence  base  or  methodology 
developed by the European Commission Initiative on Breast Cancer, coordinated 
by the Joint Research Centre (JRC). 

Full  metadata  available  at  this 
adress 
Justification of the trend 

Specific  Objective  2: identify  and  develop  coherent  approaches  and  promote  their  implementation  for  better 
preparedness and coordination in health emergencies 

Indicator 1: number of Member States integrating coherent approaches in the design of their preparedness plans 

BaseLine 
2013 
0 

2014 

2015 

2016 

2017 

2019 

2020 

2021 

2022 

0 

0 

4 

16 

18 

16 

20 

16 

2018 
Milestones 
22 
Actual Progress 
22 

24 

22 

28 

22 

Target 
2020 
28 
Final 

139 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Key monitoring indicators of the 3rd Health Programme 

Member States 

Moderate progress 

Are we on track 
Indicator type 
Unit of measurement 
Cut-Off Date 
Data source 
Narrative 
Methodology 
Link MFF 14-20 / MFF 21/27 
Other methodological comments  Actual results on the indicators will be available after the next reporting exercise 
by  Member  States  on  preparedness  and  response  planning  under  Article  4  of 
Decision 1082/2013/EU. The next reporting exercise took place in 2020. 

Full  metadata  available  at  this 
adress 
Justification of the trend 

Specific Objective 3: identify and develop tools and mechanisms at Union level to address shortages of resources, 
both  human  and  financial,  and  to  facilitate  the  voluntary  uptake  of  innovations  in  public  health  intervention  and 
prevention strategies 

Indicator 1: advice produced and the number of Member States using the tools and mechanisms identified in order 
to contribute to effective results in their health systems 

BaseLine 
2022 
Advice produced, in particular the number of Health Technology Assessments (HTA) produced per year 

2020 

2015 

2019 

2021 

2016 

2014 

2017 

2018 

2012 
2 

11 

6 

6 

9 

12 

0 

Milestones 
22 
Actual Progress 
22 

18 

4 

29 

27 

50 

41 

Target 

2020 
50 
Final 

Number of Member States using the tools and mechanisms identified in order to contribute to effective results in 
their health systems 

2012 

5 

5 

9 

10 

Milestones 
10 
Actual Progress 
22 

12 

23 

8 

9 

18 

23 

2020 
18 
Final 

Are we on track 
Indicator type 
Unit of measurement 

On track 
Output 
For  second  sub  indicator:  Number  of  Member  States  HTA  bodies  reporting 
uptake of joint HTA reports produced by EUnetHTA. 
01/02/2021 
EUnetHTA JA3 (2017-2021). 

Cut-Off Date 
Data source 
Narrative 
Methodology 
Link MFF 14-20 / MFF 21/27 
Other methodological comments  Relevant  for  the first  sub  indicator:  In  2020,  EUnetHTA  Joint  Action  3  carried 
out 40 joint reports This is an increase from the previous years, especially in the 
difficult context of the COVID-19 pandemic. In addition to 18 joint assessments 
and  early  dialogues  (that  are well-established  joint  HTA  activities),  EUnetHTA 
adapted  to  the  COVID-19  crisis  by  introducing  two  new  type  of  joint  reports: 
rapid collaborative reviews of SARS-CoV-2 diagnostic methods (2 reports) and 
rolling and rapid collaborative reviews of potential treatments for COVID-19 (21 

Annually 

140 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Key monitoring indicators of the 3rd Health Programme 

reports).Relevant  for  the  second  indicator:  The  uptake  of  joint  HTA  reports  by 
MS  depends  on  the  needs  of  national  HTA  bodies  (i.e.  was  affected  by  the 
COVID-19 crisis) and the market strategy of health technology developers (e.g. 
usually a health technology is not introduced simultaneously in all MS) 
www.eunethta.eu 

Full  metadata  available  at  this 
adress 
Justification of the trend 

Specific Objective 4: increase access to medical expertise and information for specific conditions beyond national 
borders,  facilitate  the  application  of  the  results  of  research  and  develop  tools  for  the  improvement  of  healthcare 
quality and patient safety 

Indicator 1: number of European reference networks established in accordance with Directive 2011/24/EU 

BaseLine 

2014 

2015 

2016 

2017 

2019 

2020 

2021 

2022 

0 

0 

0 

0 

10 

23 

24 

24 

24 

24 

30 

24 

24 

27 

2018 
Milestones 
24 
Actual Progress 
24 

Target 
2020 
30 
Final 

On track 

Are we on track 
Indicator type 
Unit of measurement 
Cut-Off Date 
Data source 
Narrative 
Methodology 
Link MFF 14-20 / MFF 21/27 
Other methodological comments  Networks  can  only  be  formally  established  as  provided  in  Commission 
Implementing  Decision  2014/287/EU  setting  out  criteria  for  establishing  and 
evaluating European Reference Networks and their Members. The implementing 
decision  was  adopted  later  than  forecasted  in  2013  (when  the  goals  where 
defined in the Health programme) and therefore the milestones and goals should 
be adapted to reality.  The establishment of ERNs is a complex procedure, which 
involves several steps and tools. The first call for ERN has been launched in the 
second half of 2016, the result was the establishment of 24 ERNs. Target reduced 
from  33  to  30  based  on  the  SANTE  strategic  plan  2016-2020.  Revised  target 
more likely to be reached by the end of the Programme. Recent discussions in the 
Board  of  Member  States  and  in  the  Coordinators  group  show  that  3  more 
networks could be expected by the end of the Programme. 

Full  metadata  available  at  this 
adress 
Justification of the trend 

Indicator 2: number of healthcare providers and centres of expertise joining European reference networks 

BaseLine 

2014 

2015 

2016 

2017 

0 

0 

120 

936 

2019 

2020 

2021 

2022 

1342 

1450 

1180 

1780 

Target 
2020 
1450 

2018 
Milestones 
1136 

141 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Key monitoring indicators of the 3rd Health Programme 

0 

0 

936 

Actual Progress 
956 

956 

953 

1185 

Final 

On track 

Are we on track 
Indicator type 
Unit of measurement 
Cut-Off Date 
Data source 
Narrative 
Methodology 
Link MFF 14-20 / MFF 21/27 
Other methodological comments  The difference with regard to 2018 is due to cleaning of  duplicate  records. The 
stabilisation of this indicator around 950, after reaching in 2017 the milestone set 
out  for  that  year,  can  be  explained  by  external  factors  linked  with  the  ERNs’ 
infrastructure  and  work procedures  –  first,  Member  States  took  much  longer  to 
endorse  Affiliated  Partners  (which  only  joined  during  2021)  and  secondly  new 
members can only be accepted after a long assessment process (which is coming 
to an end in 2021). 

Full  metadata  available  at  this 
address 
Justification of the trend 

Indicator 3: number of Member States using the tools developed 

BaseLine 

2014 

2015 

2016 

2017 

2019 

2020 

2021 

2022 

0 

0 

0 

0 

0 

0 

0 

18 

25 

2018 
Milestones 
20 
Actual Progress 
25 

24 

25 

28 

28 

Target 
2020 
28 
Final 

On track 

Are we on track 
Indicator type 
Unit of measurement 
Cut-Off Date 
Data source 
Narrative 
Methodology 
Link MFF 14-20 / MFF 21/27 
Other methodological comments  27 MS+Norway (following BREXIT) 
Full  metadata  available  at  this 
address 
Justification of the trend 

142 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ANNEX  VIII.  EXAMPLES  OF  FUNDED  ACTIONS  IMPLEMENTED  UNDER  THE  THIRD  HEALTH 
PROGRAMME 2014-2020 

This section non-exhaustively presents examples of actions funded by the Programme 
under the 4 specific objectives or addressing cross-cutting issues. 

These actions were selected through a large consultation of DG SANTE services in 
charge of health policy and by triangulation with stakeholders surveyed. 

1.  Actions  under  Specific  objective  1:  promote  health,  prevent  diseases,  and  foster 
supportive environments for healthy lifestyles: 

(i)  The  online  ‘best  practice  portal’  was  launched  in  April  2018.  Since  June  2018,  it 
counted  more  than  6  650  visitors  from  all  EU  Member  States  as  well  as  neighbouring 
countries.  Portal  visitors  can  access  good  practices  collected  by  previous  Health 
Programme actions. Stakeholders can also submit a practice for evaluation. 

As of 2020, more than 12 best  practices  selected by the Member States in  the Steering 
Group  on  Health  Promotion,  Disease  Prevention  and  Management  of  Non-
Communicable Diseases are implemented across 75% of the EU Member States, in areas 
such  as  integrated  care;  mental  health  and  the  fight  against  depression;  nutrition  and 
physical  activity;  prevention  of  alcohol  abuse  or  chronic  disease  prevention  and 
management. A joint action with EU Member States was launched on the exchange and 
implementation  of  best  practices  in  the  field  mental  health,  in  particular  best  practices 
regarding suicide prevention and the reform of mental health services. 

(ii) The Health Programme has developed the EU-Compass for Action on mental health 
and  wellbeing  which  is  a  web-based  mechanism  used  to  collect  exchange  of  best 
practices and analyse information on policy and stakeholder activities in mental health. 

In 2019, mental health good practices identified by the Steering Group were selected for 
scale-up and transfer (Suicide Prevention in Austria, Mental healthcare delivery system 
reform in Belgium, and European Alliance Against Depression). 

In 2020, the SGPP launched two joint actions with member state involvement, notably on 
mental health and on nutrition. 

(iii) On dementia, and in particular post-diagnostic support, crisis and care coordination, 
quality  of  residential  care  and  dementia  friendly  communities,  best  practices  already 
selected  have  been  piloted  under  second  Joint  Action  with  Member  States  competent 
authorities. 

143 

 
 
 
 
(iv) Actions to promote health and to prevent diseases also included the launching of a 
joint  action  with  EU  Member  States174  on  the  exchange  of  best  practices  for  the 
implementation  of  the  Tobacco  Products  Directive  (2014/40/EU)175  and  e-cigarette 
regulation. 

(v) Set up of an EU-wide tobacco tracking and tracing system to combat illicit tobacco 
products trafficking. 

(vi) Second joint action on tobacco control (JATC2) was launched in order to facilitate 
the exchange of good practices between the Member States, therefore, aiming to improve 
the  implementation  of  the  Tobacco  Products  Directive  (2014/40/EU)176  as  well  as 
implementing and delegated acts relating to e-cigarette regulation 

(vii)  In  2020  the  Joint  Action  on  Tobacco  control  (JATC)  produced  key  deliverables 
fostering  an  harmonised  implementation  of  the  Tobacco  products’  directive  across  the 
EU:  it  mapped  the  implementation’s  state  of  play  across  the  EU;  it  conducted  a  needs 
assessment identifying the areas of support for national competent authorities; it assessed 
the  laboratory  testing  capacity  of  national  regulators;  it  looked  at  emerging  trends  and 
new  challenges,  such  as  the  electronic  cigarettes;  it  reviewed  the  system  of  enhanced 
reporting  of  additives;  and  it  developed  guidelines  both  on  the  technical  and  legal 
dimensions  for  improving  data  sharing  and  exchange  from  the  EU-CEG  across  the 
Member  States.  The  Health  Programme  also  supported  the  EU  tobacco  tracking  and 
tracing system, in cooperation with WHO and is implementing the development of a set 
of automatic alerts, exploiting  the massive amount  of traceability data so as to  identify 
fraud patterns, suspicious behaviours and spot possible fraudsters177. 
These  actions  were  followed  in  2021  by  the  launching  with  EU  Member  States178  of  a 
joint  action  (JATC2)  on  the  exchange  of  best  practices  for  the  implementation  of  the 
Tobacco Products Directive (2014/40/EU)179 and e-cigarette regulation. 
Amongst  others,  JATC2  aims  to  ensure  greater  consistency  in  the  application  of  the 
Tobacco  Products  Directive  to  ensure  a  fair  internal  market  for  tobacco  and  related 
products, especially regarding market surveillance and enforcement.  In that direction, it 
will identify and assess the existing  legal  framework regarding tobacco advertising and 
advertising of emerging products. 

(viii) Actions in the field of nutrition delivered important outputs and tools, justifying the 
need for public health decision makers to invest in prevention rather than cure. A review 
of  the  available  evidence  for  cost-effective  prevention  interventions  has  identified  the 
food  reformulation  as  the  ‘best  value  for  money’  action  to  improve  nutrition  patterns. 
The Programme also supported the development of a database of nutritional information 
on processed food and drink products on the market in 16 Member States. 

174 ‘EU countries’ includes all countries participating in the Third Health Programme. 
175 EUR-Lex - 32014L0040 - EN - EUR-Lex (europa.eu). 
176 EUR-Lex - 32014L0040 - EN - EUR-Lex (europa.eu) 
177  The  current  COVID-19 related  restrictions  are  impacting progress  in  this  work:  the  contractor  cannot 

access the data room located in the Commission premises  

178 ‘EU countries’ includes all countries participating in the Third Health Programme. 
179 EUR-Lex - 32014L0040 - EN - EUR-Lex (europa.eu). 

144 

 
 
 
 
(ix)  On  cancer,  the  European  Quality  Assurance  scheme  has  been  developed  in 
harmonised,  evidence  based  and  flexible  way  to  grant  equal  and  quality-benchmarked 
treatment to patients. 
The  activities  of  the  European  Network  of  Cancer  Registries  coordinated  by  the  JRC 
delivers  important  results  providing  a  ‘data-brokering’  service  to  ensure  integrity  of  a 
single European dataset for different purposes. 128 Cancer Registries from 29 European 
countries are regularly providing data to JRC with more than 25 900 000 records so far in 
the database 

(x)  Following  an  extensive  consultation  with  Member  States  in  the  Steering  group  on 
Health  Promotion,  Disease  Prevention  and  Management  of  Non-Communicable 
Diseases, antimicrobial resistance (AMR), cancer, and access and availability of medical 
products, followed by prevention of non-communicable diseases (NCDs) were identified 
as  a  priority  area  for  population-level  health  interventions  and  best  practices  in 
September  2020  at  the  formal  meeting.  After  the  priority  assigned  to  prevention  of 
NCDs,  the  next  three  priorities  relate  to  risk  factors  on  tobacco,  environmental 
determinants of health (notably pollution) and obesity. For these four risk factors, a Best 
Practice call was published at the end of 2020 in the Best Practice Portal. In addition, a 
Best Practice call on Primary Care was opened 

(xi)  Other  projects  on  health  promotion  and  disease  prevention  launched  included 
projects  on  cardiovascular  disease  prevention;  the  prescription  of  physical  activity;  and 
on the promotion of whole grain consumption. 

(xii)  A  prize  for  non-governmental  organisations  has  been  organised  annually  by  the 
Health  Programme  for  encouraging  those  organisations  that  made  a  significant 
contribution in the field of public health 

2. Actions under specific objective 2 Specific objective 2: protect Union citizens from 
serious cross-border health threats:  

(i) Health security and Covid-19  
Although the overall EU response to COVID-19 is not in the scope of this evaluation, it 
should  be  noted  that  the  Commission  steered  running  actions  in  the  area  of  health 
security  to  contribute  to  the  fight  against  the  pandemic.  Key  examples  are:  the  Joint 
Action  Healthy  gateways  supports  the  coordination  among  EU  Member  States  to 
improve capacity for combating cross-border health threats at points of entry, including 
ports, airports and ground crossings, and the Joint Action on Strengthened International 
Health Regulations and preparedness (SHARP)180 which collaborated with the European 
Centre  for  Disease  Prevention  and  Control  (ECDC)  EVD-LabNet  (Emerging  Viral 
Diseases-Expert Laboratory Network) in ensuring quality  control  and  capacity building 
for  precise  diagnostics  at  an  early  phase  of  the  pandemic  and  demonstrated  the 
importance of using laboratory networks as a preparedness and response tool. 

180 The JA SHARP supports coordination among EU reference laboratories to prevent, detect and respond 
to biological outbreaks, chemical contamination and environmental and unknown threats to human health 

145 

 
 
 
Actions have been re-orientated as to provide coverage for specific needs regarding the 
Coronavirus threat  (such as  training for health professionals  including  practical  advice- 
isolation, waiting rooms and reception areas, cleaning, appropriate PPE etc; as well as an 
offer for real-time RT-PCR for detection including coverage of costs for shipment of the 
samples, if needed). 

(ii) During the Ebola and Zika outbreaks, part of the funds of the programme were used 
to  support  interventions  to  limit  the  spread  of  these  threats  by  strengthening  Member 
States preparedness and response in particular through the actions of the Health Security 
Committee (entry screening, medical evacuations, prevention of transmission in transport 
and hospital settings). The budget in 2014-2016 for strengthening EU response to health 
threats amounted to EUR 11 million. 

(iii)  In  2018,  the  Joint  Action  EMERGE  –  Efficient  response  to  highly  dangerous  and 
emerging  pathogens  at  EU  level  –  has  delivered  on  its  work  for  the  improvement  of 
capabilities  for  rapid  laboratory  diagnosis  of  new  or  emerging  pathogens  (e.g.  sample 
sharing).  The  Joint  Action  also  contributed  to  combating  the  outbreaks  of  ZIKA  and 
Ebola. 

(iv)  Four  projects  funded  in  2021  targeting  the  increase  vaccination  uptake.  Three  of 
them  were  focused  on  increasing  access  to  vaccination  for  disadvantaged,  isolated, 
difficult-to-reach  groups  and  newly  arrived  migrants  (taking  into  account  a  life-course 
approach).  The  fourth  project  ActToVAx4Nam  (Increased  Access  To  Vaccination  for 
Newly  Arrived  Migrants),  targets  exclusively  newly  arrived  migrants  –  in  first-line, 
transit and destination countries – and aims to make access to vaccination equitable and 
guaranteed. 

(v)  In  December  2019,  the  Health  Programme  launched  a  feasibility  study  for  the 
development of a common EU vaccination card. The proposed templates resulting from 
this study are tested in a sample of 10 000 citizens per country across 10 Member States, 
this  pilot  testing  covering  a  potential  population  of  more  than  200  million  citizens.  In 
2020,  the  Joint  Action  on  Vaccination  focused  specifically  on  vaccine  hesitancy  and 
uptake, identified by the WHO in 2019 as one of ten top health threats to global health 
and included in the Commission’s roadmap on strengthening cooperation against vaccine 
preventable diseases. 

Four  projects  were  launched  in  2021  on  improving  vaccination  access  and  uptake.  The 
IMMUNION  project  (Improving  IMMunisation  cooperation  in  the  European  UNION) 
focuses  on 
the  2018  Council 
Recommendation on vaccination while also adding value to EU and national initiatives - 
particularly the Coalition for Vaccination. 

increasing  vaccine  uptake  and  contributing 

to 

(vi)  The  Health  Programme  is  also  playing  a  crucial  role  in  addressing  Antimicrobial 
Resistance  (AMR),  by  defining  common  approaches  to  fight  AMR  and  to  control 
healthcare-associated infections in line with ongoing EU and international policies. 

Funded under the AWP 2017 (EU contribution of EUR 6.9 million) the Joint Action on 
Antimicrobial Resistance and Healthcare-Associated Infections  (EU-JAMRAI) supports 

146 

 
 
 
EU  Member  States  in  developing  and  implementing  effective  one  health  action  plans 
against AMR and healthcare associated infections (HCAI).  

3.  Actions  under  specific  objective  3:  support  public  health  capacity-building  and 
contribute to innovative, efficient and sustainable health systems: 

(i)  State  of  Health  in  the  EU  -  (1)  Country  Health  Profiles  2019  and  (2)  Health  at  a 
Glance 2020: Europe 
On 28 November 2019, the European Commission published 30 Country Health Profiles 
delivered as part of the State of Health in the EU cycle. The accompanying ‘Companion 
Report’  (a  Commission  Staff  Working  Document)  and  factsheet  flagged  five  key 
challenges faced by EU health systems: 

-  Tackling the decline in vaccination confidence across the EU, 

-  Harnessing the digital transformation of health promotion & disease prevention, 

-  Strengthening the evidence base on access to healthcare, 

-  Shifting tasks and changing the skill mix to explore new ways of providing care, 

-  Breaking down silos for safe, effective and affordable medicines. 

Several of these are closely linked to objective 3 of the Health Programme and to the key 
priorities of the 2019-2024 Commission as set out in the mission letter to Commissioner 
Kyriakides.  Several  of  these  actions  were  supported  by  the  3rd  Health  Programme  (see 
below sections on health and innovation, vaccination and access to medicines). 

In  November  2020,  the  European  Commission  and  the  OECD  published  Health  at  a 
Glance:  Europe  2020,  which  the  first  comparative  study  on  how  European  countries 
have experienced and responded to the COVID-19 pandemic. 

(ii)  Setting  up  of  the  EU  Health  Policy  Platform  (HPP),  a  collaborative  online  tool 
that makes it easy for European Commission services, health-related interest groups and 
stakeholders to communicate with each other. 

(iii) Digital innovation – paving the way to a European Health Data Space 

The Commission aims to support EU Member States in making the most of the potential 
of  digital  health  to  provide  high-quality  healthcare  and  reduce  inequalities.  Key  to 
achieving this aim is the creation of a ‘European Health Data Space’, to promote health-
data  exchange  and  support  research,  innovation  and  policy  making.  European  Health 
Data  Space  will  lead  to  better  health  outcomes  for  patients  and  public,  reduced  costs, 
increased  efficiency,  more  resilient  health  systems,  new  treatments  and  better  policy-
making.  The  Joint  Action  TEHDAS  (The  European  Health  Data  Space)  was  launched, 
with 25 participating countries, to facilitate the establishment of a European Health Data 
Space, by developing principles for the cross-border secondary use of health data. 

(iv) In 2019, the Joint Action on Market Surveillance of Medical Devices (JAMS) came 
to its end, yielding significant added value by reinforcing the market surveillance system 
for  medical  devices  and  improving  coordination  and  cooperation  among  all  Member 
States.  

147 

 
 
The  Scientific  Committee  on  health,  environmental  and  emerging  risks  (SCHEER)  is 
providing the Commission with risk assessment and scientific advice on medical devices 
(like hip implants, breast implants, phthalates used in medical devices…). 

(vv)  In  2019,  through  service  contracts,  the  first  communication  and  information 
campaign  on  Medical  Devices  was  implemented,  informing  stakeholders  about  the 
legislative changes brought by the new EU legislation on Medical Devices and in-vitro 
Diagnostic  Medical  Devices  which  will  enter  into  force  in  May  2020.  The  campaign 
mapped all relevant stakeholders, numbering more than 2000 contacts in and outside the 
EU. It produced eight informative factsheets translated in all EU languages, as well as in 
Chinese,  Japanese  and  Arabic.  The  campaign  also  comprises  the  organisation  of 
webinars  addressed  to  stakeholders  as  well  as  other  information  material  in  a  layman 
language. 

i(vi) Establishment of an Organ Database, which has helped 34 000 transplants only in 
2017. 

(vii)  A  service  contract  was  concluded  in  2021  for  the  provision  of  joint  Health 
Technology Assessment (HTA) work supporting the continuation of EU cooperation on 
HTA. The scope of this tender is to address the existing methodological issues in order to 
foster joint work on HTA, while also supporting EU cooperation on HTA beyond the end 
of  the  Joint  Action  EUnetHTA.  Therefore,  this  service  provides  relevant  input  for  a 
potential new legal framework on HTA. In particular, the tender shall lead to the further 
development  of  HTA  methodology  applicable  to  both  Joint  Clinical  Assessments  and 
Joint  Scientific  Consultations,  a  task  of  high  significance  in  areas  where  divergent 
opinions  persist.  Furthermore,  the  HTA  tender  shall  continue  and  advance  work 
undertaken  under  the  Joint  Action  EUnetHTA,  through  the  coordination  of  the  above-
mentioned joint activities, the interaction with stakeholders’ representatives (i.e. patients, 
health professionals and industry), academia and relevant EU/international organisations 
and initiatives on HTA. 

(viii) European Innovation Partnership on Active and Healthy Ageing: based on several 
projects and one joint action which focus on the topics of frailty, one overall European 
Framework  for  Frailty  Prevention  has  been  developed  as  well  as  practical  tools  e.g.  to 
screen older adults for (pre)frailty and innovative care paths. 

(ix)  The  programme  supported  work  on  health  information:  EUR  13.2  million  spent  in 
2014-2016  for  the  collection  and  analysis  of  health  information,  data  and  indicators  in 
cooperation with the Organisation for Economic Cooperation and Development (OECD), 
the  World  Health  Organization  (WHO)  and  the  Observatory  on  Health  Systems  and 
Policies,  as  well  as  ESTAT  and  Joint  Research  Centre  (JRC),  contributing  to  the 
development  of  country  specific  and  cross-country  knowledge  to  inform  policies  at 
national and EU level, through such actions as the State of Health in the EU cycle. 

(x)  The  BRIDGE  health  project  has  coordinated,  improved,  and  advanced  some  of  the 
most  influential  EU  health  indicator  development  networks  since  its  2015  inception, 
creating  synergies  between  the  efforts  of  several  earlier  projects  on  health  information. 

148 

 
These expert networks have been developing and maintaining indicators in the domains 
of  population  health  and  health  systems,  health  examination  surveys,  and  population 
injury,  and  developed  methods  to  produce  health  indicators  more  cost-effectively  in 
various  areas  using  disease  registries  and  administrative  health  data  collection  systems. 
Various options to improve the sustainability of producing indicators to underpin health 
policy  and  research  in  the  EU  have  been  developed,  and  a  blueprint  for  a  more 
sustainable organisation of these activities is being finalised. 

4. Actions under specific objective 4 facilitate access to better and safer healthcare for 
Union citizens: 

(i) ERNs are virtual networks involving healthcare providers across Europe. They aim to 
tackle  complex  or  rare  medical  diseases  or  conditions  that  require  highly  specialised 
treatment and a concentration of knowledge and resources. 

The first ERNs are up and running since March 2017. Over the five subsequent years, as 
the ERNs reach full capacity, thousands of EU patients suffering from a rare or complex 
condition have benefited from specialised treatments. 

 (ii)  Rare  diseases:  the  Joint  Research  Centre  has  developed  and  is  maintaining  the 
European  Platform  on  Rare  Diseases  Registration  (EU  RD  Platform)  receiving  specific 
financial  support  from  the  Health  Programme.  The  Platform  is  promoting  the 
interoperability  of  existing  registries  and  has  helped  in  the  creation  of  new  ones, 
including  those  developed  by  the  European  Reference  Networks.  The  migration  of  the 
two  databases  -  the  European  Surveillance  of  Congenital  Anomalies  (EUROCAT)  and 
the  Surveillance  of  Cerebral  Palsy  in  Europe  (SCPE)  –  has  been  successfully 
implemented. 

Joint  Action  on  rare  diseases  (RD-ACTION)  is  supporting  Member  States  in  the 
development and implementation of actions in the area of rare diseases. Three main goals 
of the RD-Action are to: 

1)  support the further development and sustainability of the Orphanet database; 

2)  contribute to solutions to ensure an appropriate codification of rare diseases in 
health information systems; and 

implementation  of 

in  Council 
3)  continue 
Recommendation  on  patient  safety  including  the  prevention  and  control  of 
healthcare  associated 
the  Commission 
Communication on rare diseases (COM(2008)679). 

(2009/C151/02)  and 

the  priorities 

infections 

identified 

In the area of rare diseases codification general rules for routine coding with Orphacodes 
have been established and guidelines are provided to achieve internationally standardised 
data collection. 

(iii)  The  Commission  carried  out  in  2021  an  evaluation  on  the  operation  of  Directive 
2011/24/EU on the application of patients' rights in cross-border healthcare. 

(iv) A study supporting  the impact  assessment  of the  revision of  Directive 2002/98/EC 
on  safety  and  quality  of  human  blood  and  blood  components  and  of  Directive 

149 

 
2004/23/EC on safety and quality of human tissues and cells and of their implementing 
acts was launched. The duration of the contract is nine months, and the scope of the study 
is to support the European Commission in preparing an impact assessment supporting the 
Revision of the Union legislation on blood, tissues and cells. 
The specific objectives are to:  

•  Provide a description of the baseline. Moreover,  to  summarise the key elements 
of  the  evaluation  and  complement  the  problem  definition  by  gathering  and 
analysing  evidence  on  borderline  technologies/therapies  and  the  impact  and 
lessons learned from the COVID-19 pandemic.  

•  Measure  the  likely  economic,  social  and  environmental  impacts  of  each  of  the 
options. A preliminary identification of the key  impacts  is  in  section 1.6. of the 
study. 

•  Analyse the effectiveness, efficiency and coherence of policy options in order to 

achieve the objectives of the revision. 

5. Actions addressing cross-cutting issues – health inequalities, legislation, SDGs 

 (i)  The  Joint  Action  on  Health  Equity  Europe  (JAHEE),  funded  under  the  3rd  Health 
Programme  2014-2020  has  been  an  important  opportunity  for  Member  States  to  work 
jointly to address health inequalities and achieve greater equity in health outcomes across 
all  groups  in  society,  in  all  participating  countries  and  in  Europe  at  large.  The  general 
objective  of  this  initiative  is  to  improve  health  and  well-being  of  European  citizens 
across  all  groups  in  society.  It  also  has  a  specific  focus  on  both  vulnerable  groups  and 
migrants.  

(ii)  The  programme  also  provided  the  resources  for  implementing  the  EU’s  political 
commitments  and  legal  obligations  in  health  (e.g.  implementation  of  the  tobacco  or 
health  threats  legislation,  the  EU  regulatory  framework  for  medicinal  products  and 
medical  devices,  for  substance  of  human  origin,  and  cross-border  health  care).  It  also 
supports the Member States implementation of this legislation through the development 
of common tools, such as networks, IT platforms, guidance and sharing of best practices. 
The  development  and  maintenance  of  these  tools  (e.g.  EUDAMED  database,  Euripid 
database)  is  essential  in order  to  ensure  the  smooth  operation  of  the  Internal  Market  in 
these sectors. 

(iii)  A  joint  action  was  launched  in  2021  on  Increasing  the  capacity  of  National  Focal 
Points  to  provide  guidance,  information  and  assistance  to  national  applicants  on  the 
EU4Health  programme  and  other  EU  funding  instruments,  and  another  joint  action 
addressing  the  Differences  in  national  implementation  of  the  General  Data  Protection 
Regulation  (GDPR)  in  the  health  sector-Development  of  a  code  of  conduct  for  data 
processing. 

(vi)  The  Programme  contributes  to  the  EU  work  towards  the  WHO’s  nine  voluntary 
targets  on  non-communicable  disease  and  the  UN  Sustainable  Development  Goals  3- 
Ensure  healthy  lives  and  promote  well-being  for  all  at  all  ages,  DG  SANTE  works 
through the Steering Group on Health Promotion, Disease Prevention and Management 

150 

 
 
 
of Non-Communicable Diseases (SGPP). Examples of such support included funding for 
operating grants to NGOs. Specific grant agreements were signed with 15 NGOs in 2020. 

Four  direct  grant  agreements  were  also  concluded  with  international  organisations,  as 
follows: 

-  Three  direct  grant  agreements  were  signed  with  the  OECD,  on  best  practices 
implementation;  on  patient  reported  outcomes  and  on  pharmaceutical  products’ 
accessibility. 

-  One direct grant agreement was signed with the Council of Europe to support the 

European Pharmacopeia work.  

151