EUROPEAN COMMISSION Brussels, 10.5.2016 SWD(2016) 148 final COMMISSION STAFF WORKING DOCUMENT Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS

Ex-post Evaluation of the 2nd Health Programme 2008-2013



Decision No 1350/2007/EC

establishing a second programme of Community action in the field of health (2008-13)

{COM(2016) 243 final}

{SWD(2016) 149 final}



Table of contents 1. INTRODUCTION 1.1. Purpose of the evaluation 1.2. Scope of the evaluation 2. BACKGROUND TO THE INITIATIVE 2.1. Description of the initiative and its objectives 2.2. What outputs were expected from the Programme? 2.3. What results and impacts were expected from the Programme? 2.4. Baseline 3. EVALUATION QUESTIONS 4. METHOD 4.1. Process and methods used 4.2. Limitations – robustness of findings 4.3. Quality assessment of the study 5. IMPLEMENTATION OF THE PROGRAMME 6.1. Budget distribution per financial mechanism 6.2. Budget distribution per thematic area 6.3. Participation by type of stakeholder group 6.4. Participation by geographical area

6. Findings of the Evaluation 7.1. Programme management Effectiveness and efficiency of the Programme management 7.2. The Programme's effectiveness and factors that are influencing it 7.4. Coherence and consistency with other European policies and programmes 7. CONCLUSIONS Annex I – Evaluation Matrix Annex II – Recommendations applied Annex III – JOINT ACTIONS ACHIEVEMENTS 2008 – 2013 ANNEX IV – Quality Assessment Form 1. INTRODUCTION 1.1. Purpose of the evaluation This staff working document accompanies the Commission report and the external evaluation report that the Commission is transmitting to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions under Article 13(3)(c) of Decision No 1350/2007/EC. The external and independent ex post evaluation of the 2nd Health Programme was conducted in 2014-2015. Its purpose was primarily to assess the performance of the Programme management implementation, including followup to the recommendations in past health programme evaluations. The evaluation contributes to a better understanding of the strengths and weaknesses of Programme implementation and management and provides conclusions that can be used as a basis for improving the implementation of the current 3rd Health Programme. 1.2. Scope of the evaluation This evaluation follows on from previous evaluations of the 1st Public Health Programme and the 2nd Health Programme, building on the results, in particular of the mid-term evaluation of the latter, without repeating earlier evaluation work carried out to inform the design of the 3rd Health Programme. Consequently, it focuses on specific aspects of the Programme, such as programme management, dissemination of results and synergies with other programmes, seeking to complement the previous evaluations. While addressing the functioning of the entire Programme, the contractors concentrated on issues that were insufficiently explored in past exercises and provided conclusions that can form a basis for changes to improve the effectiveness and efficiency of the 3rd Health Programme. 2. BACKGROUND TO THE INITIATIVE

2.1. Description of the initiative and its objectives According to the Treaty on the Functioning of the European Union (TFEU), a high level of health protection shall be ensured in the definition and implementation of all Union policies (Article 168 (1) TFEU). Union action, which shall complement national policies, shall be directed towards improving public health, preventing physical and mental illness and diseases, and obviating sources of danger to physical and mental health. The 2nd Health Programme was the main instrument for implementing the EU’s 2008-2013 health strategy Together for health; from 2011, it was aligned with the priorities of the Europe 2020 strategy. The Programme’s overall aim was to complement, support and add value to Member States’ policies and to contribute to increased solidarity and prosperity in the EU by protecting and promoting human health and safety and improving public health. Health is a prerequisite for economic recovery and ‘inclusive growth’, and the health sector attracts interest for innovation and ‘smart’ investment. The Programme financed pan-European actions geared to achieving three main objectives: I.improving citizens’ health security and protecting them from health threats and emergencies, such as pandemics and natural disasters; II.promoting health and reducing health inequalities across Europe, whether relating to lifestyle, such as access to opportunities for physical activity, to health care, such as access to the necessary medical intervention; and III. generating health information and health knowledge and disseminating it to relevant parties, from the general public to policymakers and health professionals. Three broad thematic areas corresponding to these objectives were identified, with priorities and subpriorities (see Figure 1). 2.2. What outputs were expected from the Programme? Under Article 168 TFEU, the Union shall encourage cooperation between the Member States, and support their action, including through the establishment of guidelines and indicators, the organisation of exchange of best practice and support for monitoring and evaluation. Member States’ responsibilities for the definition of their health policy and for the organisation and delivery of health services and medical care should be respected. The Programme was expected to support and add value to Member States’ policies and hence contribute to protecting the health and safety of citizens through actions in the field of public health. Accordingly, the Programme financed a large number of actions with a good coverage of all the priorities and sub-priorities established in Decision No 1350/2007/EC, the outputs of which can be broken down as follows: -knowledge- and evidence-building through studies and/or surveys (e.g. Eurobarometer), including evaluations and impact assessments that are beneficial on a number of levels, e.g. providing a basis for informed policymaking and reporting; -tools and/or methodologies that help to secure advantages for both the public-health communities (e.g. integrating their work processes) and citizens directly (e.g. with regard to improving diagnostic tests, improving patient care, etc.); -communication, awareness-raising and networking (e.g. co-funding pan-EU conferences and networks inter alia in the field of rare diseases); -comparable data across the EU, providing information for policymaking purposes, e.g. European core health indicators (ECHIs); -training, educational material and guidance with a positive impact on the public-health community (e.g. by providing guidelines on patient care, diagnostics, social inclusion of vulnerable groups, etc.) and on citizens who might benefit from treatment by better-educated healthcare professionals; -best practices, helping to achieve and maintain high standards in all health-related areas (research, prevention, access, care, treatment, etc.); and -capacity-building in the public-health community at different levels (e.g. increasing the capacity of healthcare systems to deal with diseases through an exchange of knowledge with healthcare institutions in other Member States). Figure 1: 2nd Health Programme – thematic areas, priorities and sub-priorities

2.3. What results and impacts were expected from the Programme? Many health-related challenges, such as cross-border health threats, cannot be addressed at country level; hence there is a clear need for EU action to complement Member States’ efforts. However, any action at EU level should demonstrate EU added value and actions cofunded by the Programme were expected to result in one or more of the following: -a contribution to the development and/or implementation of EU legislation; -money saved and duplication of efforts avoided by cooperation across national health systems for the improvement of health in the EU; -identification and application of best practice in all participating countries, e.g. procedures, approaches, methods or tools that could be applied by healthcare professionals or others; -evidence-based decision-making facilitated, e.g. by providing scientific information, real-time data for comparison and/or indicators that can inform decision-making at a higher political/policy level; -risks reduced and consequences of cross-border health threats mitigated by the establishing of relevant structures for coordination; -increase in the movement of patients and healthcare personnel between Member States, thereby contributing to a better match between supply and demand; -sustained networking activities among stakeholders, contributing to knowledge-sharing and health capacity-building in the EU; and -support for the deployment of innovative solutions for healthcare provision, in terms of both products and services. The results overall are expected to impact on public health in Europe in order to achieve the main objective of the Programme, i.e. to complement, support and add value to Member States’ policies and to contribute to increased solidarity and prosperity in the EU by protecting and promoting human health and safety and improving public health. 2.4. Baseline The 1st Public Health Programme (2003-2007) grew out of a small number of isolated, empirically managed activities in response to calls from the Council and the European Parliament, such as action on HIV/AIDS, health information, etc. The number of priorities increased gradually around the three main objectives of health promotion, health security and health information, in order to optimise impact and meet new expectations through an integrated approach. The Programme was operated exclusively through grants for projects and a small number of tenders. The Member States that joined the Union in May 2004 became involved progressively and were underrepresented in the actions financed. The Programme was managed by the Commission, except for a small part which was transferred to the Public Health Executive Agency, which became the Executive Agency for Health and Consumers (EAHC) and later the Consumers, Health, Agriculture and Food Executive Agency (CHAFEA), after its establishment and operational launch in 2006. The evaluation of the 1st Public Health Programme recognised its strong potential contribution to preparing, developing and implementing EU public-health policies, despite the broad spectrum of health priorities it covered, and called for more focus and rationalisation. The dissemination of the results was seen as an important area for improvement: the outcomes of the actions targeting health policymaking at EU, national or regional levels were neither sufficiently known nor widely used by stakeholders and policymakers. Disseminating results was seen as essential to ensuring their sustainability and helping to monitor the impact of the actions. The design of the 2nd Health Programme was similar to that of its predecessor, but involved new financial mechanisms, in addition to grants for projects and conferences, in order to respond better to stakeholders’ needs: operating grants for nongovernmental organisations, direct grants for boosting cooperation with international health organisations, joint actions with Member States and tenders to cover specific needs related to the support of EU health policies. 3. EVALUATION QUESTIONS The evaluation is based on a set of 14 questions divided into four main areas: (a) management tools; (b) dissemination practices;

(c) Programme impact; and (d) synergies with other services and programmes. The questions do not follow the classical approach to programme evaluation (focusing on relevance, effectiveness, efficiency, coherence and utility) but, following the results and recommendations of the midterm evaluation, target specific areas of concern in programme implementation. As regards programme design and management, the mid-term evaluation resulted in recommendations to: define more tangible and focused objectives and establish progress indicators; prepare strategic multi-annual planning to determine appropriate priority actions and select the corresponding financial mechanisms; provide technical assistance to potential applicants for preparing appropriate proposals; create a nomenclature for explaining EU added value and integrate it in the application process through specific criteria; provide further explanations on the scientific evidence required in proposals and how to share it; share other information with Programme stakeholders and potential beneficiaries; develop a regular reporting system for the actions and their results; communicate/disseminate project results better and more systematically and improve communication with Programme stakeholders; and make full use of consistencies and complementarities between Programme actions and other actions at international, European and national level, including sharing of data among Commission services, Member State authorities and international organisations. These recommendations underlay the first four evaluation questions, which were designed to measure the progress made on the effectiveness of the Programme management: EQ 1: To what extent have the recommendations of the mid-term evaluation concerning the management and the design of the Programme been implemented? EQ 2: How effective have recent changes in the emphasis on and use of specific funding mechanisms (i.e. use of joint actions, balance between calls for proposals and calls for tender) been in delivering policy-related outputs and what was the impact on the geographical distribution of beneficiaries? EQ 3: To what extent did the implementation of previous recommendations influence the Programme’s other operations, including the recruitment of beneficiaries and the level of participation of all Member States in Programme actions (including the facilitation of participation from low-GNI countries)? EQ 4: What are the state-of-the-art tools in terms of monitoring project outputs that could be applied to the Programme, what are the expected benefits against costs and how could they be implemented? As regards dissemination practices, the mid-term evaluation recommended fostering the dissemination of results and organising an exchange of information on results between the Agency, Commission officials, policymakers in Member States and other stakeholders.





The evaluation questions relating to dissemination were: EQ 5: (a) To what extent have the actions/outcomes/results of the 2nd Health Programme been published? To what extent are they (made) accessible to the international scientific and health community, to health policymakers, civil society and to the wider public in the EU?

(b) Are the results published and disseminated in a sustainable way?

(c) How useful is the EAHC database in this context? How can it be improved?

(d) Which other tools would be useful in this context? EQ 6: What is the relation between the publications/activity reporting and Member State participation in the 2nd Health Programme, the number of health scientists, public-health specialists and physicians per Member State? Are patterns identifiable? Have dissemination activities been undertaken in a way to overcome possible geographical imbalances in certain actions? EQ 7: To what extent do stakeholders other than Member State governments (sub-national regional organisations, civil society, social partners, etc.) promote Programme outcomes and results, and via which channels? This should consider both organisations funded by the Programme, and others. EQ 8: How could the current dissemination practices be improved to increase return on investment? Since the negotiations with the Council and European Parliament on the 3rd Health Programme confirmed the important role dissemination plays in maximising programme impact, Article 13(4) of Regulation (EU) No 282/2014 explicitly requires wide dissemination of the results. Accordingly, these questions and the subsequent answers were intended to contribute to improving dissemination further. Given the difficulty of assessing the impact of a small programme against the scale of health needs in Europe, the relevant evaluation questions focused first on the relevance of the Programme actions vis-à-vis the Union mandate on health and secondly on the short- and medium-term progress achieved in specific areas. With a view to the next programming period, the questions also seek to elicit elements for a better understanding of how the Programme could impact on health policies in the Member States. EQ 9: How and to what extent has the 2nd Health Programme supported Member States’ health policy and actions (in relation to the provisions on support, cooperation and coordination in Article 168 of the Treaty)? EQ 10: Which are the main health policy areas in which progress has been achieved due to the support of the Health Programme, and what constitutes this progress? EQ 11: What are reasonable assumptions on the way to measure the impact of the programme in terms of (a) short-term, (b) middle-term, (c) long-term timelines and (d) in relation to average project trajectories? EQ 12: Which factors/reasons may intervene and positively or negatively influence the impact of the Programme? EQ 13: What are the main lessons than can be drawn to ensure an overall successful transition from the 2nd to the 3rd Health Programme? The success of the Programme also depends on synergies with other programmes in the area of health. Thus, the last evaluation question refers to coherence and consistency and focuses on the two other major programmes (under the FP7 research programme and the Structural Funds) with substantial EU funding and interest for Member States. However, other synergies with smaller programmes are also covered, since the question concerns the Commission’s general objectives for economic growth and social inclusion. EQ 14: What synergies are there with other policies and programmes of the Commission such as the European Structural and Cohesion Funds, the programmes managed by DG RTD and other DGs (in particular EMPL, CONNECT) and to what extent did the Health Programme underpin the Commission’s general objectives – focus on Europe 2020 and their objectives related to social policy (e.g. the renewed Social Agenda) and economic growth (research and innovation, competitiveness)? 4. METHOD 4.1. Process and methods used The external evaluation study started in May 2014 and the final report was delivered in July 2015. An interservice steering group established in December 2013 discussed and validated the evaluation mandate and agreed on the evaluation questions and the terms of reference for the specific contract. The group met four times to discuss in addition to the above, the inception, interim and draft final reports and provided comments on the methods and organisation of the evaluation. It was composed of representatives from DG SANTE, RTD, AGRI, JRC, EMPL, REGIO, EAC and CONNECT, the SecretariatGeneral and CHAFEA. In addition, the European Hospital and Healthcare Federation (HOPE), the European Health Management Association (EHMA) and the European Public Health Association (EPHA) were represented in order to feed in the views of health-policy stakeholders. In the spirit of a collaborative and transparent approach, the findings and main conclusions were presented to the national focal points on 12 January and 22 May 2015 and to the Programme Committee members on 6 March 2015 and 4 February 2016 and they were asked for their comments and opinions. The evaluation involved a variety of quantitative and qualitative data collection and review methods and analytical tools to respond to specific information needs and requirements respecting the principle of triangulation. Annex I contains a matrix showing the various tools used to make assessments on the basis of agreed judgment criteria and answer each of the evaluation questions. The contractors used desk research, direct observations, a survey of the national focal points and interviews with Commission officials, CHAFEA project officers, Programme Committee members, beneficiaries and project leaders, and the assistants of two Members of the European Parliament to generate data for analysis. Also, they carried out an analysis of 80 actions selected proportionally from across the main Programme areas, priorities and financial mechanisms to assess EU added value and to review the type of actions (research, development and implementation), the type of partner organisations, the partnerships’ geographical spread, cross-sectoral cooperation and dissemination practices. To assess how co-funded actions contributed to the Programme objectives and identify factors that could strategically maximise the potential impact of the Programme, the evaluators selected 13 case studies (five projects, five joint actions and three tenders) from the 80 actions in order to delve deeper into specific aspects, such as the design of actions, implementation, results and their dissemination, and added value. They also conducted a bibliometric analysis of the Programme’s visibility in scientific journals and, to some extent, an assessment of Member States’ public-health capacity in relation to their capacity to participate in the Programme and make use of the funding.

Finally, on the basis that it is critical for the Programme’s success to ensure that all key stakeholders are effectively engaged in and/or informed of the Programme and its results, the contractors undertook an analysis of the Programme stakeholders. This sought to explore the power, position and interests that different stakeholder groups brought to the Programme and to identify how they could be involved further. When carrying out the work, the contractors defined conditions and features on the basis of which to assess Programme actions and verified their validity, in particular through the case studies: essential conditions, common to all actions, that influence effectiveness and thus could influence the probability of the action having an impact in the longer term (Table 1); and specific key features per funding instrument, as each instrument is meant to respond to different needs and produce certain outputs/results, e.g. tenders to obtain studies and respond to specific Commission needs, joint actions to boost Member States’ cooperation on common health issues, calls for proposals for projects on health issues with a wider scope and to incentivise innovation, and operating grants to support NGOs and specific networks.

Table 1: Essential conditions for actions’ effectiveness Condition Description The actions address a relevant problem To contribute to HP objectives, actions need to address a specific problem that fits into one of the priorities and where the EU added value of action is high. The actions are based on concrete and SMART objectives. In order to implement a service efficiently, you need to develop concrete objectives, operationalised in a SMART way, i.e. your objectives should be specific, measurable, achievable, realistic and time-dependent. The actions are evidence-based. Overall and SMART objectives are more likely to be achieved when projects are designed on an evidence-based understanding of how the activities they implement are related to what they actually want to achieve. This includes building on existing knowledge. The actions have clear target groups. Efficient organisations are often characterised by a relevant and explicit definition of their target groups. The actions have developed adequate implementation strategies The goals of the HP are far-reaching and require dedicated effort over a long period. This means that the chances of achieving long-term effects improve if the project activities are sustainable and are implemented by the participating actors. The actions are characterised by a high degree of target achievement. If the above conditions are met and actions achieve what they set out to do, this is likely to generate impacts that contribute to the wider HP objectives. The actions have effective strategies for disseminating results. Dissemination of results is key to facilitating their take-up beyond the participants themselves. The contractors sought to take an innovative approach to assessing on-going efforts (in line with the recommendations of the final evaluation of the 1st Public Health Programme and the mid-term evaluation of the 2nd Health Programme) to involve lowGDP/GNI Member States. This meant measuring not only increased participation in calls, but also initiatives for the transfer of knowledge to these countries (evaluation questions 3 and 6). The contractors suggested approaching participation by countries that joined the EU from 2004 onwards not only from an economic angle, i.e. participation rates of low- versus highGPD/GNI countries, but also assessing the relationship between countries’ participation and their ‘publichealth capacity’. 4.2. Limitations – robustness of findings The evaluation is not based on a theory-change approach, as it was considered too difficult to construct a posteriori an overall intervention logic for a programme with very broad objectives and multiple priorities grounded in the EU’s supporting competence in public health, as laid down in the Treaty on the Functioning of the European Union. It was also too early to assess results as outputs of the actions were just being delivered. For this reason, the contractors based their work on explicit expectations and assumptions (see the essential conditions and specific key features above) as to what the Programme and the various financial mechanisms were to achieve. They followed a purpose-driven approach to sampling (for the in-depth review, case studies, bibliometric analysis, stakeholder analysis, etc.), focusing on those actions and facets of the Programme that promised to be of most value and interest for the analysis, given the specific evaluation purpose and questions. The Commission services gave their agreement to the choices made. It was also decided to limit the breadth of the evaluation, since the 3rd Health Programme had already been launched and certain aspects became more pertinent than others, depending on their continued relevance for the new Programme (see point 2.2 on the scope of this evaluation). As a result, while certain key features of the 2nd Programme (in particular the very broad scope and resulting lack of focus, including operational and specific objectives) would normally have been addressed in more depth in a final evaluation, they were not given much prominence here. Relevant recommendations (e.g. as regards the need for more specific objectives and indicators) had already been made in the mid-term evaluation and addressed in the design of the 3rd Health Programme. As mentioned above, the contractors went beyond distinguishing participation between high- and lowGNI Member States to explore the statistical relationship between Member States’ ‘publichealth capacity’ and their participation rates (taking as a proxy variable the amount of funding that organisations from a given country were able to obtain). This approach proved to have significant methodological limitations, mainly due to the lack of a commonly agreed definition of ‘public-health capacity’. Consequently, the analysis was limited to some indicators relating to wealth (GDP/GNI), health research spending, health expenditure, health publications, healthcare resources, health outcomes and healthcare performance. Moreover, data quality and availability for some of these were not always ideal for the correlations that the evaluators were examining. The weaknesses in the design of the Programme objectives resulting in the lack of indicators for systematic monitoring as already found in the mid-term evaluation posed another limitation to assess the effectiveness of the programme. Given these limitations, the findings and conclusions are representative only for the actions investigated and cannot necessarily be extrapolated to all actions under the Programme, as the results of individual actions cannot simply be aggregated to assess overall impact. Also, given the broad Programme objectives, limited dissemination efforts, dependence on the willingness of Member State authorities to take up the results and integrate them in national health policies, and the time taken for health indicators to change, the evaluation can at this stage assess outputs from individual actions only and not the impact of the whole Programme. 4.3. Quality assessment of the study By and large, we agree with the contractors’ findings, the answers to the evaluation questions and its conclusions within the limitations described above. In the contractors’ discussions with interservice steering group members, it became clear that, for four questions more information would have been appreciated. In relation to the effectiveness of changes concerning the specific funding mechanisms (i.e. use of joint actions, balance between calls for proposals and calls for tender) in delivering policy-related outputs the contractors’ approach is more theoretical and explains how it was expected that the Programme would support Member States’ health policies, but it was not possible to show, on the basis of outputs to date, the extent to which it has achieved its goal.

In relation to the identification of patterns between Member States participation in the 2nd Health Programme and its public health capacity the limited quality and availability of data on Member States’ public-health capacity did not allow for a robust analysis. Suggestions for improvement on dissemination practices were limited to better targeting audiences but did not look into return on investment as required in the Terms of Reference. We accepted the contractor's explanation that it will not be feasible to measure the ‘return on dissemination investment’ in terms of health outcomes, since this would require complex models to assess evidence-based policy-making, which are outside of the scope of the assignment. In the absence of explicit specific objectives and indicators already from the design of the 2nd Health Programme, it was not possible for the contractor to clearly indicate the extent to which the Programme supported Member States’ health policies. The Programme is a series of successful individual actions but it is impossible for numerous reasons to draw concrete links from individual actions or the Programme as a whole to the high-level indicators ( i.e. Healthy Live Years) relating to health outcomes, when these are sometimes available. Also the actions’ desired outcomes, even in the best circumstances, take years to materialise and are largely highly specific to the actions in question. 5. IMPLEMENTATION OF THE PROGRAMME The Commission prepares every year, in close consultation with Member States health authorities serving on the Programme Committee the Annual Work Programme and adopts it through "Comitology" procedure. The Work Programme defines the most relevant actions to address Member States health needs and create added value at EU level. These actions should have high public health relevance and pertinent geographical coverage. The CHAFEA was entrusted with implementing most of the Programme through competitive calls for grants and tenders. The Commission implemented only specific, highly policy-relevant service contracts and crosscutting actions, such as IT services, itself. 6.1. Budget distribution per financial mechanism Various financial mechanisms were used to implement the Programme: Projects are used to explore a wide range of subject areas and delivery mechanisms, and take forward health policy initiatives in an innovative way, almost as ‘pilots’. They absorbed most of the available budget and provided significant scope for innovation. Their use declined in the second half of the Programme, mainly in favour of joint actions and tenders, in an effort to concentrate the Programme on a series of a few major actions aligned with the Europe 2020 strategy; Tenders: calls for service provision are used to cover specific Commission needs with regard to studies, evaluations, surveys and technical assistance. This includes IT- and communication-related services required to develop and update EU health legislation or to fulfil the Commission’s obligations under EU health legislation. A good example is the development of reference tools for the design and use of a single European coding system for tissues and cells. Service contracts are also used where the scope and objectives are very concrete and under Commission control, e.g. in the development and conduct of training courses and exercises with Member States to build capacity to deal with emergency situations; Joint actions are a new financial mechanism introduced in the 2nd Health Programme to cover specific health-policy needs and aimed at supporting EU cooperation with as many partners as possible from all countries participating in the Programme, to generate momentum for wider impact. The number of joint actions called for increased from 2011 onwards in order to enhance the Programme’s policy relevance and make it more compatible with the Europe 2020 objectives of smart and inclusive growth. Joint actions are often started, after several years of cooperation between relevant stakeholders from (or designated by) Member State authorities, in a bid to secure political endorsement and optimise policy coordination. They typically develop/share/refine/test tools, methods and approaches for specific issues or activities and involve a degree of capacitybuilding. . The gain for the Member States involved is expected to be substantial in terms of knowledge and experience exchanged and should also lead to tangible cost savings. For this reason, the Programme sought to ensure that joint actions attract the widest possible participation from all Member States; Operating grants are also a new instrument in the 2nd Health Programme. They support the running costs of pan-European NGOs and specific networks that focus on priority health issues and contribute to furthering health policy in the EU; Conference grants: while support for pan-EU conferences on important health topics was not really new as a type of action, their selection through an annual competitive call, separate from the call for projects, was introduced under the 2nd Health Programme to avoid competition with proposals for larger health projects. Grants for (twice-yearly) central conferences on health were awarded directly to the Member State holding the Presidency, which also selected the conference topic and took care of the organisation; and Direct grants to international organisations, such as the World Health Organisation (WHO) and the Organisation for Economic Cooperation and Development (OECD), were provided to continue international cooperation on major health issues (mainly the collection and analysis of health data). In total, 788 actions were financed: 147 projects, 30 joint actions, 420 service contracts, 84 operating grants, 36 direct grants with international health organisations and 71 conferences. The overall budget distribution per funding mechanism is shown in Table 2. Table 2: Programme spending by funding mechanism Funding mechanism Total % Projects €106 293 671.24 36 % Service contracts (tenders) €72 053 873.45 25 % Joint actions €63 962 704.38 22 % Operating grants €20 825 185.85 7 % Direct grant agreements €13 805 987.00 5 % Grants for conferences €5 268 308.14 2 % Other €11 693 227.81 4 % Total €293 902 957.87 100 % Source: CHAFEA database and DG SANTE

The financial instruments used most were projects, service contracts and joint actions; together, these accounted for more than 80 % of the budget. 6.2. Budget distribution per thematic area In pursuit of its objectives, the Programme supported actions in three thematic areas: health security, health promotion and health information. Actions supporting the objective of health promotion were at the heart of the Programme, accounting for 57 % of total funds allocated, while the areas of health security and health information received 23 % and 21 % respectively.

Figure 2: 1st Public Health Programme and 2nd Health Programme spending by thematic area Source: CHAFEA database and DG SANTE Health promotion gained in prominence as compared with the 1st Public Health Programme (see Figure 2), underlining the importance the 2nd Health Programme placed on addressing health determinants and tackling health inequalities. Meanwhile, actions focusing on the generation and dissemination of health information declined. The relative importance ascribed to the health security objective remained virtually unchanged, although the epidemiological surveillance networks were transferred at the end of the 1st Public Health Programme to the European Centre for Disease Prevention and Control (ECDC). Overall, the contractors found that five priorities and sub-priorities accounted collectively for about three quarters of Programme spending: health determinants and healthy lifestyles: 24 % of the overall Programme budget was aimed at tackling key health determinants such as nutrition, alcohol, tobacco and drugs, and other determinants more related to social and environmental factors; prevention of major and rare diseases: 16 % of the overall spending related to major diseases (e.g. cardiovascular disease, cancer and HIV/AIDS prevention) and rare diseases (including support for developing recognised expert reference groups, assistance to Member States in developing and taking forward rare-disease strategies, and contributing to WHO international classifications of rare diseases); health monitoring and data: 11 % of the budget was spent on forming an effective and sustainable network for health technology assessment (HTA) across Europe to help develop reliable, timely, transparent and transferable information to contribute to HTAs in European countries. It also supported work relating to ECHIs to facilitate monitoring and comparison between EU countries, thereby serving as a basis for policymaking; health threats: 13 % for actions inter alia to facilitate collaboration between laboratories and develop common testing methods, with the aim of developing strategies and mechanisms to respond to health threats and emergencies; safety: 10 % to fund a variety of actions relating to issues such as organ donation and transplantation, and patient safety, some of which facilitated the exchange of organs donated in Member States; assessing data on manufactured nanomaterials and seeking to establish a European framework for the evaluation of organ transplant results. The reports and deliverables produced by the actions co-funded under the Programme are available on CHAFEA’s website. A database provides open access to the results, with the exception of service contracts and direct agreements with international organisations. 6.3. Participation by type of stakeholder group A wide and diverse range of stakeholders participated in and benefited from the grants provided by the Programme. Figure 3 provides a breakdown by group of grant beneficiaries. More specifically, the groups are as follows: government organisations: these represent the largest group (37.8 %) of stakeholders that participated in the Programme. They include health policymakers, regulators, general or specialised governmental publichealth organisations and institutions, and healthcare providers. They were chiefly interested in participating in joint actions, particularly those relating to health security and health information; non-profit and non-governmental organisations: these make up a second rank of participating organisations (30.7 %). They mainly received operating grants, but also participated in projects and joint actions; academic and research organisations: with a share of 26.5 %, these were involved mainly in projects (to the same approximate extent across all three thematic areas);

commercial organisations: their participation accounts for 3.2 % (significantly more if we take into consideration their participation in tenders which are in the most of the cases addressed to them); and international organisations: their cooperation with the Commission, mainly under the health information objective (collecting and analysing health data), accounted for 1.7 % of the total. Figure 3: Participation of stakeholder groups receiving grants under the 2nd Health Programme Source: CHAFEA database and DG SANTE 6.4. Participation by geographical area All Member States and the three EEA EFTA countries (Iceland, Liechtenstein and Norway) participated in the Programme. As the Programme was open to candidate countries, Croatia was involved for the entire period (as a Member State from 1 July 2014). In line with recommendations in the final evaluation of the 1st Public Health Programme, efforts were made, especially through the joint actions, to involve more actors from the ‘EU-12’ Member States that joined the Union in and after 2004. In terms of number of beneficiaries, participation in the Programme reflected relative population sizes. Just over three quarters (76.1 %) of beneficiaries were based in the ‘EU-15’, while the EU-12 accounted for 20.4 %. There is a disparity in terms of the allocation of funding, with 88.3 % going to organisations from the EU-15 and 9.4 % to those based in the EU-12. This is probably due in part to differences in wages and labour costs. However, the difference between EU15 and EU12 is far more pronounced when one considers the spread of lead beneficiaries, of which an overwhelming 95 % were based in the EU15, with only 4 % based in the EU12. This was especially visible in service contracts, projects and operating grants, for which nearly all lead partners were based in the EU15. However, 15 % of lead partners for grants for conferences (11 of 71) and 10 % for joint actions (3 of 30) were based in the EU12. This breakdown is presented in Figure 4. Figure 4: Proportion of (total and lead) beneficiaries and funding received (EU15, EU12 and other participating countries) Source: CHAFEA database and DG SANTE The mid-term evaluation mentioned that administrative and cultural barriers, in addition to financial constraints, could act as obstacles to the participation of low-GDP countries. The final evaluation (following a survey of national focal points) reported that the EU-15 faced more (or at least cited more often) a lack of human and financial resources and administrative burden (see Figure 5). Figure 5: Main barriers to the participation of EU12 and EU15 organisations Source: Survey of national focal points in the framework of the ex post evaluation 6. Findings of the Evaluation 7.1. Programme management The relevance of the Programme was extensively assessed by the mid-term evaluation, but the ex post evaluation provided a second opportunity to examine the relevance of a series of co-funded actions. This was done on the basis of the 13 case studies, which led to positive conclusions.

The mid-term evaluation concluded that the Health Programme is focusing on relevant priority areas addressing the main public health issues in Europe; however the Programme's broad objectives were not helpful to prioritise actions as most health-related issues could fit under them under any circumstances. However, the ex-post evaluation also found that during its second half, the Health Programme increased the policy relevance of funded actions. Through an increased involvement of DG SANTE's management in the annual planning a greater level of coherence with the Europe 2020 targets was achieved. Through making use of joint actions to a greater extent buy-in from national governments and participation of key stakeholders from nearly all Member States was secured. However, the lack of appropriate indicators at Programme level and the absence of a systematic monitoring to link the available data at action level with higher level health indicators made it difficult to fully understand whether and how the Programme impacts on national health policies. Selecting actions on the basis of their EU added value The 2nd Health Programme aims at complementing, supporting and adding value to Member States’ policies thus providing ‘EU added value’. As Decision No 1350/2007/EC establishing the 2nd Health Programme did not define ‘EU added value’, the EAHC/CHAFEA developed seven criteria to determine whether proposed actions have the potential to generate EU added value. These criteria were tested and validated in the course of the mid-term evaluation in 2011. The Commission added an eighth criterion concerning potential for innovation in the area of health and integrated all eight criteria in Regulation (EU) No 282/2014 (recital 6) establishing the 3rd Health Programme and in the subsequent awarding procedures for actions to be co-funded. The eight criteria used are the following: 1)implementing EU legislation: to ensure that actions contribute to the development and/or implementation of EU legislation; 2)economies of scale: to save money and provide a better service to citizens by avoiding duplication of effort and cooperating across national health systems; 3)promotion of best practice: to apply best practice in all participating Member States, e.g. by identifying procedures, approaches, methods or tools that could be applied by healthcare professionals or others; 4)benchmarking for decision-making: to facilitate evidence-based decision-making, e.g. by providing scientific information, realtime data for comparison and/or indicators that can impact decisionmaking at a higher political/policy level; 5)cross-border threats: to reduce risks and mitigate the consequences of cross-border health threats by establishing relevant coordination structures; 6)free movement of persons: to increase the movement of patients and healthcare personnel between Member States, thereby contributing to a better match between supply and demand; 7)networking: to ensure that networking activities among stakeholders, which contribute to knowledgesharing and building health capacity in the EU, are supported and sustained; and 8)unlocking the potential of innovation: to support the deployment of innovative solutions for healthcare provision, in terms of both products and services. In the ex-post evaluation 80 actions were scored by an expert panel for potential to deliver EU added value and under which added value criteria (validation of the selection process). For 13 actions outputs and results were assessed for delivering EU added value. This served as a basis for analysing which type of actions have the greatest potential to deliver EU added value and on which criteria added value is being delivered. The evaluation found that use of the criteria is effective and that actions co-funded through the Programme, particularly the joint actions, scored high on the EU added value. Much of the demonstrable EU added value relates to the identification of best practices, the scientific evidence to be used for benchmarking for decisionmaking and networking activities (see figure 6 and 7). However, these three criteria are not sufficiently linked to tangible and concrete benefits. Further guidance on these criteria would be necessary to enable applicants to propose more suitable actions that not only identify good practices, for instance, but also address barriers to implementing them across the EU. Actions received medium scoring for innovation, EU health legislation and economics of scale while the criteria of cross-border health threats and free movement of persons were under -represented. Figure 6: Average scores by EU added value criteria, all actions Figure 7: Proportion of actions averaging scores of 2.0 or more and 1.0 or less, by EU added value criteria This way, the evaluation made clear that the robustness and completeness of the three first highly scored criteria is not optimal and lack the necessary discrimination power to avoid that the large majority of the actions fit broadly under these ones. Effectiveness and efficiency of the Programme management

The effectiveness of the Programme management was assessed by the contractors, including the programme's increased focus, on priority areas, while addressing Member States’ needs and encouraging their participation. Consideration was given to how implementation was monitored and how results were disseminated. Relevant findings will be used to inform implementation of the current 3rd Health Programme on the basis of the findings of the contractor, the following conclusions can be drawn. The Programme management improved significantly in the second half of the period. Substantial efforts were made to implement recommendations from the mid-term evaluation, relating inter alia to more strategic programming, the systematic use of EU-added-value criteria in grant applications and evaluation, providing clearer guidance to applicants and having better contact with applicants and beneficiaries. Annex II presents the measures taken to implement these recommendations and the remaining issues looked into in the evaluation with a view to suggesting further improvements. The changes in the management of the Programme increased its potential to serve Member States’ needs and to complement and support their health policies. More directive planning methods and increased use of joint actions and tenders resulted in a greater focus on specific health policies in order to meet specific needs. While the 1st Public Health Programme was implemented mainly via projects and a small number of tenders, the 2nd Health Programme relied increasingly on joint actions. These aim to involve a significant number of Member States working jointly on key health policies in the expectation that the outputs will be more policyrelated (as compared with outputs from projects). A total of 30 joint actions were co-funded during the 2nd Health Programme, for almost €64 million. A full list of joint actions and overall achievements is provided in Annex III. Although joint actions attracted participation from all Member States and other participating countries, the Programme does not seem geographically balanced in terms of budget distribution and Member States’ participation (total number of beneficiaries and beneficiaries in leading positions), as shown in Figure 4. While joint actions were a financing mechanism used increasingly in the second half of the Programme and attracted relatively more participants from the EU-12 countries (which accounted for 13 % of the overall budget spent on joint actions), the intensive use of calls for tender over the Programme period attracted interest chiefly from entities in a limited number of EU-15 Member States, with Belgium in the lead. This offsets the relatively higher EU-12 participation in joint actions. This evaluation takes a significant step forward in starting to reflect on Member States’ public health capacity and how this affected their participation in the Programme. The previous evaluations have only mentioned the financial barriers some Member States could face for their participation in the Health Programme, and the distinction was made between high and low GDP/GNI Member States. The ex-post evaluation even under the methodological limitations imposed by the fact that there is no common agreed definition at EU level for "public-health capacity", gave a relatively interesting insight using only some indicators relating to wealth (GDP/GNI), health research spending, health expenditure, health publications, healthcare resources, health outcomes and healthcare performance. It permitted the linkage of the low participation of Member States (in the most of the cases these are low GNI countries) with what public-health capacity meant for the Health Programme. Building public health capacity was not an objective under the Programme, but findings suggest a posteriori that it may have been a means or even a prerequisite for successful participation: the Member States that participated more actively were those with solid public-health capacity, while the ‘weaker’ countries had lower participation rates and received less funding. The evaluations repeatedly point to the challenge of monitoring Programme implementation. Monitoring output and outcome from a programme with so many actions, diverse in terms of form and content, is not always easy, especially when it comes to making links to higher-level public-health indicators. However, CHAFEA collects comprehensive monitoring data at input and activity levels. However the data are not organised systematically; which hampers their aggregation and use in real time to inform strategic planning so as alignment with priorities and objectives cannot sufficiently be ensured and deficiencies persist in the monitoring of the Programme performance. Given the architecture of the Programme, the evaluation was unable to shed more light on this issue or provide a common set of indicators for all actions or objectives. It does, however, underline the importance of suitable indicators for good monitoring and reporting for the improved dissemination of action outputs. Dissemination activities The increased dissemination efforts are recognised as contributing to the success of the Programme. Failure to share outputs and results with those who need them to build health policies and other initiatives based on scientific knowledge tested in real settings constitutes a real obstacle to assessing the Programme’s impact. The contractors carried out an extensive analysis of the means used for dissemination, either by the beneficiaries in the framework of the co-funded actions or by CHAFEA, on a more aggregated level. Previous evaluations showed that Programme outputs and results should be disseminated at three levels: 1) at the level of the co-funded actions; every action has its own dissemination strategy and plan, which in some instances is very effective and in others less so, mainly due to a lack of clarity and focus as to the most relevant target audiences and how best to reach them; 2) dissemination activities organised and means produced by CHAFEA, such as brochures, ‘cluster meetings’, project database, etc.; and 3) dissemination by the Commission, e.g. a highlevel conference on EU health programmes organised in Brussels on 3 May 2012, DG SANTE’s bi-monthly electronic newsletter and information for policymakers, Programme Committee members, the European Parliament and the Council through annual programme implementation reports. The contractors assessed these levels and the means applied in more detail. For example, beneficiaries are encouraged to publish their results in scientific journals. The bibliometric analysis showed that numerous published articles (more than expected) referred to 2nd Health Programme actions, but the visibility of the Programme is not always sufficient, as it was not always acknowledged as the source of the funding, even though this is required under the grant agreements. As the most appropriate audiences for the dissemination of results vary, so do the most effective tools and channels for reaching these audiences. Some actions and their results are relevant for specialists only; others have wider relevance also for patients and healthcare service users. Overall, however, the evaluation research suggests that the most frequently targeted audiences are governmental organisations, healthcare professionals, and academia and researchers (in this order). These can sometimes be reached via publications in scientific journals (which result from some actions funded under the Health Programme), but it is important to note that research is not the main focus of the Health Programme, and scientific publications, although they present an interesting channel for disseminating information, are not always the most effective way of reaching directly those stakeholders responsible for implementing changes in the area of health. The contractors also assessed the utility of CHAFEA’s project database, which provides public access to the abstracts and deliverables of co-funded actions (with the exception of tenders and direct grants to international organisations). The database is quite static and not always up-to-date. The deliverables, e.g. extensive final project reports, are not always user-friendly and additional interactive functions are lacking that could make the database a useful tool providing a real service to stakeholders. The ‘cluster meetings’ organised by CHAFEA in cooperation with competent Member State authorities are assessed positively as attracting good attendance and decent press coverage. Three cluster meetings were held under the 2nd Health Programme, on rare diseases, organ transplantation and vaccination. The 60 – 120 participants were experts in the relevant field as well as journalists from different EU countries, invited by Chafea. The number of journalists was about 20, covering an equal number of Member States. The average number of articles that appeared after the meeting in the general and specialized press is about 25 covering about half of the Member States, Efforts are needed to promote that articles triggered by such meetings mention the Programme and the EU explicitly and to widen press coverage beyond the Member State in which the meeting takes place. While the contractors could not assess the extent to which the various dissemination actions reached the various stakeholders, they recognise that dissemination had improved over the course of the Programme and that this contributed to its efficiency. They point out that no dissemination activities were undertaken specifically to overcome geographical imbalances. They also remark that most publications, guidance documents, etc. are available only in English. The smooth functioning of programme management and the growing responsibility of CHAFEA across all administrative functions meant that certain tasks could be streamlined and made more efficient. 7.2. The Programme's effectiveness and factors that are influencing it

Previous evaluations sought to measure the impact of the Programme, but this proved difficult for reasons inherent in its design, the multiplicity of its actions and its broad objectives, which interact with many other external factors, such as the long timescale over which effects on health materialise. The majority of actions in the framework of this evaluation were assessed as successful in terms of their implementation, but it is not possible to ‘add up’ their outputs or to follow through on their individual impacts to produce a composite Programme impact (see limitations referred to in section 5.2). The merit of the evaluation is that it highlights numerous factors on which the Programme’s impact depends and which influence it positively (or negatively if absent). The case studies, which assessed the outcome of 13 actions, showed that it is of vital importance that actions: have clear links to existing policy initiatives (to demonstrate how they further existing policy initiatives and policies; this corresponds to the ‘policy relevance’ award criterion for selecting the most relevant actions for co-funding); have prepared plans for sustained follow-up efforts (in order to avoid co-funding actions that will not continue once the EU co-funding is stopped); work to propose feasible policy changes (considering the context) in the medium term (this will help beneficiaries to concentrate their work on actions that can bring tangible and pragmatic results by addressing not only what has to be done, but also the challenges to be overcome and prepare the field for changes in the health sector); have a well-delineated scope and clearly defined objectives (the absence of which may result in partners taking disparate action without working towards a common goal); have a plausible ‘intervention logic’ (to guide the partners throughout implementation); involve all relevant partners (the absence of a strong and complementary partnership can hamper implementation); have strong project management (leadership is important if actions are to be implemented according to the plan and achieve high-quality results); involve constructive engagement from DG SANTE/CHAFEA; and are implemented through the most suitable financial mechanism. Since ultimate responsibility for public health is (mostly) left to other organisations (in particular national health authorities), the success of the Programme derives from its ability to help make those other organisations (which range from international organisations and national health ministries to universities and NGOs) do their jobs better and more effectively. Actions were more successful when they addressed identifiable policy needs, had a well-delineated scope and produced results that could be readily applied in practice. While joint actions and service contracts met these criteria to a greater extent than projects, there were a few examples among all action types where this was not the case. With the increased use of joint actions in the second half of the period, the Programme selected specific areas in which it especially sought to directly involve Member States authorities and the relevant bodies responsible for implementing health policies who have an interest in applying the Programme's outcomes, and thus maximised the chances of impacts materialising in the years to come. Significant achievements resulted from the majority of Member State health authorities being involved and cooperating very closely at the appropriate level on major health issues of common interest (see Annex III). Joint actions often are the culmination of long years of cooperation and build on previous achievements made possible through project grants started sometimes 10 or more years ago. Figure 8 illustrates by way of example the impact trajectory of the EUnetTHA joint action on HTA. Figure 8: Impact trajectory of the EUnetTHA joint action The challenge for the 2nd Health Programme, given its modest budget, was guaranteeing the sustainability of actions and results of which the impact is demonstrated only if they are taken up and used by Member State authorities and/or other actors. This is why ‘reiterations’ of actions (possibly leading from a project to a joint action) are observed for some priority health issues and no followup for negative priorities that were not supported further.

While funding of recurrent actions was included as an option under the Programme in order to meet this challenge, there are two risks: if funding for priorities is suspended after a certain period of time, the achievements could be lost; and if the Programme spends too much on multiple iterations of a few priority actions, it could fail to identify meaningful new initiatives and miss opportunities to invest in new areas in rapidly changing contexts. Each funding mechanism has its strengths and weaknesses. Joint actions, projects and service contracts were all shown to be highly appropriate conditions that played to the relative strengths described in the table below. By contrast, opting for the wrong funding mechanism in given circumstances (e.g. using a project when DG SANTE’s needs and desired product are well defined, which is better suited to service contracts) severely undermined actions’ potential effectiveness (and cost-effectiveness). Table 3: Conditions of success for given funding mechanisms Funding mechanism Ideal circumstances Risks / challenges Joint actions Clearly established case for pan-European collaboration at a technical (and not only political) level Buy-in from key stakeholders in (nearly) all Member States Feasibility of desired results already confirmed from previous work Political momentum sufficient for results to be applied in practice Due to their size and the number of partners typically involved, joint actions are costly to implement and can be difficult to manage If established prematurely, joint actions can be too unwieldy to provide a forum for exploring new ideas and experimenting The chances of results being taken up is reduced if a critical mass of Member States is not secured Projects Highly relevant topic but case for pan-European collaboration not fully established, particularly regarding practical solutions Need for a ‘pilot’ to ascertain level of interest and feasibility of changing status quo Availability of strong leadership and established interest from a smaller group of committed partners to pursue a focused set of objectives Value of collaboration beyond the level of the partners themselves needs to be established If the primary focus is on networking and sharing best practices, the need to create more tangible results can be lost Projects often struggle with national differences in data availability / comparability Overly ambitious / diverse objectives can reduce effectiveness If policy links are absent, it is difficult to overcome barriers for EU-wide implementation of results Service contracts Existence of specific and clearly defined DG SANTE needs / ideas Narrow set of objectives and limited scope Clear link to specific policy process or initiative Level of ambition needs to be aligned with typical budgets (€100-250k). Clear need for action should be established beyond interest of specific DG SANTE units. Excessive reliance on service contracts would be detrimental to HP inclusiveness (in terms of types and geographic spread of beneficiaries) 7.4. Coherence and consistency with other European policies and programmes There are important synergies between the 2nd Health Programme and the Seventh Framework Programme (FP7): actions under the former build on and use FP7-funded research (e.g. on health threats from nanomaterials) and the latter is a vehicle for the further investigation of issues and knowledge gaps that arise as a result of Programme actions (e.g. on specific HTA methodologies and application areas). Synergy effects with the Structural Funds are less obvious, as the main results produced by the 2nd Health Programme actions such as networking or joint solutions (good practice methods or approaches) do not lend themselves to implementation using co-funding from the European Regional Development Fund, the Cohesion Fund or the European Social Fund. However, almost €5 million from the Programme budget was spent to promote the use of Structural Funds for health. These actions provided guidance and awareness-raising that should enable those responsible for operational programmes to address health-related issues more effectively. From 2011, the Programme placed more emphasis on the Europe 2020 goals for smart and inclusive growth, by prioritising: oactions relating to the European Innovation Partnership in the field of active and healthy ageing, which was set up as an Innovation Union flagship initiative; oactions to address health determinants such as nutrition, smoking and alcohol abuse, which underlie many age-related chronic diseases; owork on cancer and rare diseases; oEU cooperation on HTA; owork on the safety of blood, tissues, cells and organs (which contributes to improving health across the lifecycle, thereby contributing to healthy ageing);

omeasures that apply information and communication technologies in the area of health; and oactions aimed at bridging health inequalities to ensure better health for all and better access to healthcare systems. Comparing spending on such actions under the most relevant priorities in the first and second halves of the Programme (i.e. before and after Europe 2020 was adopted), the budget for actions on active and health ageing increased by 485 % and for those on health inequalities by 307 %, while for those on smart growthrelated priorities it saw a modest increase or even slight decrease. In contrast, funding for actions addressing health determinants and promoting healthy lifestyles and those aiming to develop a health monitoring system/collect comparable data decreased by 17 % and 21 % respectively (Table 3). Table 3: Funding for key priorities relating to Europe 2020 objectives Europe 2020 objective Priority HP funding 2008-2010 (€) HP funding 2011-2013 (€) Change Smart growth Organs and substances of human origin, blood and blood derivatives (1.2.2) 4 213 499 5 239 964 +24 % Increase healthy life years and promote healthy ageing (2.1.1) 2 887 184 16 893 162 +485 % Address health determinants and promote healthy lifestyles (2.2.1) 32 897 669 27 221 835 -17 % Prevention of major diseases of particular significance, and rare diseases (2.2.2) 19 103 140 19 920 192 +4 % Develop a sustainable health monitoring system and collect comparable data (3.2.1) 15 719 845 12 490 561 -21 % Inclusive growth Identify the causes of, address and reduce health inequalities (2.1.2) 3 552 153 14 440 968 +307 % Source: CHAFEA database and DG SANTE 7. CONCLUSIONS The main conclusions that can be supported based on the evaluation can be summarised as follows: The management of the 2nd Health Programme improved compared with the 1st Public Health Programme as a result of being entrusted to the Executive Agency and of the introduction of different financial mechanisms, such as joint actions, operating grants and direct grants with international organisations, which better serve stakeholders’ needs and respond to their expectations. By following up many recommendations from the mid-term evaluation, implementation improved further in the second half of the Programme, through the use of more strategic programming, the systematic use of EU-added-value criteria in grant applications and selection, clearer guidance for applicants and better contact with applicants and beneficiaries. However, there are still concerns around monitoring, dissemination of results and administrative burden for applicants and beneficiaries. Good monitoring is crucial, also for communicating Programme results, and increased systematic efforts are needed to ensure dissemination with clear strategic objectives, targeting stakeholders with the most influence and power. The Programme was highly coherent with the Europe 2020 strategy for smart and inclusive growth. Since 2011, its policy relevance was enhanced by the use of more joint actions and tenders, and more funding for actions that focus on promoting healthy and active ageing and reducing health inequalities. However, every case of relevant need does not necessarily imply a strong case for EU involvement. To maximise the impact of the Programme, the relevant actions should demonstrate clearly their EU added value. This is why criteria for EU added value were established in the course of the Programme and applied in the assessment of proposals. For the 3rd Health Programme, the Commission went a step further by proposing that Regulation (EU) No 282/2014 contained an explicit reference to these criteria, which continue to be integrated in the evaluation process for awarding funding to ensure that all co-funded actions deliver EU added value. In comparison with other financial mechanisms, joint actions often generate substantial added value, in some cases leading to tangible cost savings in addition to providing useful lessons for the Member States involved. They deal with priorities determined through a comprehensive appraisal of public-health needs in Europe, with input from Member States and other interested parties, e.g. expert committees. They often support a policy process in a specific field of health and encourage Member States to cooperate in the implementation of an existing legal framework. Some joint actions address emerging health problems in the global health environment; pathogens do not respect borders and can affect several Member States, so common action is often required. Joint actions can also arise in response to ‘horizon-scanning’ work by EU expert committees to identify emerging health problems in Europe which could become a priority for action at European level. The ‘project’ funding mechanism, as used for the majority of actions aimed at health promotion and accounting for 57 % of the total funding awarded, responds to EU-added-value criteria relating mainly to best practices, benchmarking for better decision-making and networking. These criteria have weak links to tangible policy benefits and need to be more clearly defined and communicated so as to help applicants to submit proposals and ensure that actions deliver more tangible and concrete benefits. Therefore, the evaluators’ suggestion for re-working these three criteria seems to make sense. This could be very useful for credibly demonstrating how such actions lead to more concrete benefits over the longer term. The Programme finances actions with the potential to influence health policies positively at national and EU level, taking into account certain conditions, including long-term financing for secured sustainability of activities and outcomes. Nevertheless, the contractors found it difficult to assess the impact of the Programme as a whole, given its broad objectives, the multiplicity of priorities and the absence of indicators for measuring progress (an issue raised in earlier evaluations). Recommendations from the mid-term evaluation calling for a focus on a restricted number of actions with defined progress indicators linked to the corresponding Programme objectives have been implemented in the design of the 3rd Health Programme and will be assessed in its mid-term evaluation in 2016-2017. Regarding the low participation of EU 12 Member States in the Health Programme, the evaluation findings confirm that efforts should be continued to encourage greater participation from underrepresented Member States, inter alia by targeting key governmental institutions and drawing attention to the opportunities on offer. If it continues, the low participation of some Member States may hamper the success of the 3rd Health Programme. Going forward, and given that public-health capacity-building is explicitly included in the legal basis of the 3rd Health Programme, it would seem necessary for the concept to be clearly defined, agreed, factored into clear Programme priority-setting and the conducive design of individual actions (including possible specific mechanisms to support ‘weaker’ Member States), and further elaborated in monitoring arrangements and future evaluations. Also, wider dissemination of outputs and results can provide good examples and convincing arguments for more involvement and leadership for all Member States/participating countries interested in the Programme.



Annex I – Evaluation Matrix

Evaluation questions Judgment criteria Preliminary indicators Data sources Evaluation bloc (a) Programme management tools i. To what extent have the recommendations of the mid-term evaluation concerning the management and the design of the Programme been implemented? Extent to which a strategic multi-annual planning with long-term targets was developed. More tangible and focussed objectives (SMART) developed Evidence of long-term planning Targets set in consultation with national health experts Indicators developed Desk research (including statistical review and analysis) NFP survey Interviews with: -DG SANTE and CHAFEA officials -Programme Committee members and National Focal Points -External stakeholders Extent to which the Programme continues to employ the current system of different and diversified financial mechanisms. Number of actions funded (2008-13) and budget allocated by financial mechanisms Officials’ and stakeholders’ views on the balance of funding mechanisms  Extent to which clear guidelines are provided to potential applicants and applicants. Evidence of clear guidelines provided to potential applicants and applicants (e.g. on intervention logics, ToCs, SMART objectives, indicators, definitions of certain terms etc.)_ Evidence of retention of requirement for proposals to outline compliance with AWP priority areas and HP objectives Stakeholders’ perceptions on whether clear / simple guidelines are provided  Extent to which EU added value is being emphasised, particularly at the proposal stage. Existence and level of dissemination of a nomenclature for explaining EU added value etc. Level of utilisation of this nomenclature in proposals and actions  Extent to which Programme application procedures have been simplified and rationalised. Evidence of simplified and rationalised Programme application procedures (including “lighter” application process, use of framework contracts to reduce repetitive and burdensome tendering procedures) Officials’’ and stakeholders’ perceptions on whether Programme application procedures have been simplified and rationalised and what more could be done   Extent to which monitoring has been enhanced Evidence that applicant satisfaction surveys are still being undertaken Evidence of the monitoring of types of organisations applying for funding Evidence of more in depth assessments of samples of actions  Extent to which actions and their results have been better disseminated Evidence of dedicated budgets for dissemination at action level Evidence of clearer guidelines on target groups, dissemination plans Evidence of increased dissemination activities from DG SANCO / action leaders to stakeholders (e.g. Programme Committee, EP, Council, CoR, wider audiences)  Extent to which better use has been made of synergies at national, European and international levels Evidence of (increased) collaboration with relevant interested parties/organisations, e.g. through meetings and conferences Evidence of increased data sharing between HP actions and similar activities at national, European, international levels  ii. How effective have recent changes in the emphasis on and use of specific funding mechanisms (i.e. use of Joint Actions, balance between calls for proposals and calls for tender) been in delivering policy-related outputs, and what was the impact on the geographical distribution of beneficiaries? Extent to which changes in the emphasis on and use of specific funding mechanisms have contributed to policy-related outputs Evidence and examples of policy-related outputs Evidence demonstrating links between the emphasis / use of specific funding mechanisms and policy-related outputs Officials’ and stakeholders’ perceptions on whether changes in the emphasis on and use of specific funding mechanisms have contributed to policy-related outputs Desk research (including statistical review and analysis) Interviews with: -DG SANTE and CHAFEA officials -Programme Committee members and National Focal Points -External stakeholders Extent to which changes in the emphasis on and use of specific funding mechanisms has had any impact on the geographical distribution of beneficiaries Evidence and examples of changes in the geographical distribution of beneficiaries Evidence / examples demonstrating links between the emphasis / use of specific funding mechanisms and changes in the geographical distribution of beneficiaries Officials’ and stakeholders’ perceptions on whether changes in the emphasis on and use of specific funding mechanisms have contributed to changes in the geographical distribution of beneficiaries  iii. To what extent did the implementation of previous recommendations influence the Programme’s other operations, including the recruitment of beneficiaries and the level of participation of all Member States in Programme actions (including the facilitation of participation from low GNI countries)? Extent to which the implementation of previous recommendations has impacted on the recruitment of beneficiaries Quantity of proposals received per year under the different funding mechanisms Evolution of the profile of applicants (by Member State and by partner-type) Officials’ and stakeholders’ perceptions on whether and in which ways the implementation of previous recommendations has impacted on the recruitment of beneficiaries Desk research (including statistical review and analysis) Interviews with: -DG SANTE and CHAFEA officials -Programme Committee members and National Focal Points Extent to which the implementation of previous recommendations has impacted on the level of participation of all Member States, and in particular has facilitated participation from low GNI countries Evolution of the level of participation of Member States Officials’ and stakeholders’ perceptions on whether and in which ways the implementation of previous recommendations has impacted on the level of participation of all Member States  iv. What are the state of the art tools in terms of monitoring project outputs that could be applied to the Programme, what are the expected benefits against costs and how could they be implemented? Extent to which there are relevant state of the art tools for monitoring project outputs that could be applied to the Programme Inventory of state of the art tools for monitoring project outputs Examples from national and EU Programmes Desk research NFP survey Interviews with: -DG SANTE and CHAFEA officials -Programme Committee members and National Focal Points -Representatives of other programmes and initiatives (e.g. National Institutes of Health, National Institute for Health Research) Benefits against costs of applying tools for monitoring project outputs SWOT analysis of identified tools / monitoring systems Officials’ and stakeholders’ perceptions on benefits and costs of applying identified tools for monitoring project outputs  Ways in which these tools could be implemented to the Programme, including associated risks Officials’ and stakeholders’ perceptions on ways in which these tools could be implemented to the Programme, and associated risks  Evaluation bloc (b) Programme dissemination practices v. a) To what extent have the actions/ outcomes/ results of the Second Health Programme been published? To what extent are they (made) accessible to the international scientific and health community, to health policy makers, civil society, and to the wider public in the EU? b) Are the results published and disseminated in a sustainable way? c) How useful is the EAHC database in this context? How can it be improved? d) Which other tools would be useful in this context? (for part d) of the question, see question viii. below) Extent to which individual HP actions develop dissemination strategies Extent to which relevant stakeholder groups are targeted (as recipients or multipliers of information) Extent to which dissemination strategies for action results are implemented effectively % of actions that developed explicit dissemination plans Breakdown of stakeholders targeted Funds budgeted / spent on dissemination (as a share of total action budget) Access (no. of hits) to selected actions’ websites No. of copies of final report sent / distributed (electronic and hard copies) No. of dissemination events organised (conferences etc.) to present results Use of other dissemination tools (e.g. newsletters) Evidence of uptake / implementation of results, depending on dissemination strategies pursued Effectiveness of dissemination strategies / tools / activities / target audiences as perceived by stakeholders Review of database / documentation on 90 actions Case studies Stakeholder analysis Stakeholder interviews Quantity of scientific output stemming from Programme actions Quality of scientific output stemming from Programme actions Number of articles published in peer-reviewed journals Number of citations of such articles Journal impact factor of HP2-related publications H-index (citation index ) of HP2-related publications Bibliometric analysis of selected actions Sustainability of dissemination efforts Extent to which further dissemination is foreseen / carried out after the end of HP2 funding Extent to which relevant websites remain live / updated following action completion Extent to which scientific output remains accessible / is cited (see above) Review of database / documentation on 90 actions Case studies Usefulness of EU (DG SANTE / CHAFEA) dissemination activities and tools, in particular the database No. of visits to CHAFEA database Usefulness of database as perceived by stakeholders Stakeholder views and perceptions of other tools (events, publications, etc.) by DG SANTE / CHAFEA Review of database and other tools / activities Stakeholder interviews NFP survey vi. What is the relation between the publications/activity reporting and the Member State participation in the Second Health Programme, the number of health scientists, public health specialists and physicians per Member States? Are patterns identifiable? Have dissemination activities been undertaken in way to overcome possible geographical imbalances in certain actions? Extent to which participation / publication rates by MS reflect the relative public health capacity of MS Participation of organisations from different MS in actions (as coordinator or collaborator), per year No. of scientific publications related to HP actions, by MS Measures of public health capacity: (1) health resources (financial / human), (2) health research (GERD, publications, clinical trials) Relationship between the above, patterns and trends over time HP action ‘portfolio’ review Data review from other sources (ECHI, other sources) Extent to which dissemination has contributed to reducing imbalances No. and type of measures adopted to overcome imbalances Extent of change that can be attributed to such measures No. of dissemination activities undertaken to redress geographical imbalances in selected actions Geographical differences in the level of satisfaction of stakeholders of selected projects with the dissemination efforts made Stakeholder interviews NFP survey Case studies vii. To what extent do stakeholders other than Member State governments (subnational regional organisations, civil society, social partners etc.) promote Programme outcomes and results, and via which channels? This should consider both organisations funded by the programme, and others. Extent and outreach of promotion activities carried out by the various non-State actors Extent of recourse to the different communication channels (web, other media, conference /meetings etc.) Tangible effects on health policy / strategy at the various level related to dissemination % of actions with non-State actors promoting outcomes/results spontaneously (broken down by type) Frequency of utilisation of the various communication channels for the above activities Level of outreach of the above dissemination actions (for selected projects) No. of reported cases of tangible effects (and type) on policy / strategy from the above activities (selected actions) Review of database / documentation on 90 actions Online questionnaire with National Focal Points Interviews with non-governmental promoters / disseminators Case studies (focused on dissemination) viii. How could the current dissemination practices be improved to increase return on investment? Room for improvement of current dissemination practices Cost-effectiveness of other dissemination tools / approaches that could be used Overview of other relevant possible dissemination tools (to be identified) Estimated costs of alternative approaches Likely benefits Ideas gleaned from interviews / surveys / other sources Cost-benefit / cost-effectiveness analysis Evaluation bloc (c) Impact of the programme ix: How and to what extent has the Second Health Programme supported Member States’ health policy and actions (in relation to the provisions on support, cooperation and coordination in Article 168 of the Treaty)? Extent to which the Programme has supported Member States’ health policy and actions Ways in which the Programme has supported Member States’ health policy and actions Evidence and examples of ways in which the Programme has supported Member States’ health policy and actions Extent to which the Programme meets the essential criteria for maximising its potential impact Stakeholders’ perceptions on ways in which the Programme has supported Member States’ health policy and actions Stakeholders’ perceptions on (i) whether Programme activities are additional to what MS would have done anyway, and (ii) what might have happened in the absence of the Programme Desk research - screening exercise and in-depth review Case studies NFP survey Interviews with: -DG SANTE and CHAFEA officials -Programme Committee members and National Focal Points -External stakeholders x: Which are the main health policy areas in which progress has been achieved due to the support of the Health Programme, and what constitutes this progress? Identification of the main health policy areas in which progress has been achieved as a result of Programme support Evolution over time in the main health policy areas where actions have been funded Evidence and examples of health policy areas where significant progress has been achieved due to the support of the Programme based on an assessment in relation to the 7 added value criteria Stakeholders’ views on the health policy areas where the Programme has had the greatest impact and where it should be providing more support Stakeholders’ perceptions on (i) whether Programme activities are additional to what MS would have done anyway, and (ii) what might have happened in the absence of the Programme Desk research - screening exercise, in-depth review and added value analysis Case studies NFP survey Interviews with: -DG SANTE and CHAFEA officials -Programme Committee members and National Focal Points -External stakeholders Identification of actions / results illustrating the progress achieved Degree to which given actions are sustainable, i.e. would continue with a reduction in / in the absence of EU funding Extent to which given actions meet the pre-defined essential conditions for actions (i.e. the action has defined SMART objectives and targets; the action has defined a clear target group; the action has developed clear implementation and dissemination strategies; the action has delivered its main deliverables to a sufficient standard; the action has covered its stated target group; the action has met its stated targets; the action has disseminated its outputs / results to plan) Stakeholders’ perceptions on the actions / results illustrating the progress achieved  xi: What are reasonable assumptions on the way to measure the impact of the programme in terms of timelines a) short-term, b) middle-term, c) long-term and d) in relation to average project trajectories? Identification of assumptions to measure the impact of the programme in terms of timelines a) short-term, b) middle-term, c) long-term and d) in relation to average project trajectories In-depth review of Programme impact for a selection of actions and analysis of when this can be measured Stakeholders’ perceptions on time-adapted ways in which to measure the Programme’s impact Case studies NFP survey Interviews with: -DG SANTE and CHAFEA officials -Programme Committee members and National Focal Points - xii: Which factors/reasons may intervene and influence positively or negatively the impact of the Programme? Identification of factors/ reasons that can positively or negatively influence the impact of the Programme Identification of examples of factors / reasons that can positively or negatively influence the impact of the Programme Stakeholders’ perceptions on factors / reasons that can positively or negatively influence the Programme Case studies NFP survey Interviews with: -DG SANTE and CHAFEA officials -Programme Committee members and National Focal Points -External stakeholders xiii: What are the main lessons than can be drawn to ensure an overall successful transition from the second to the third Health Programme? Identification of lessons to ensure an overall successful transition from the second to the third Health Programme Evidence and examples of lessons to ensure an overall successful transition from the second to the third Health Programme Stakeholders’ perceptions on lessons to ensure an overall successful transition from the second to the third Health Programme Desk research - Desk review and analysis of relevant documentation Desk research - screening exercise, in-depth review and added value analysis Case studies NFP survey Interviews with: -DG SANTE and CHAFEA officials -Programme Committee members and National Focal Points -External stakeholders Evaluation bloc (d) Synergies with other policies and programmes xiv. What synergies are there with other policies and programmes of the Commission such as the European Structural and Cohesion Funds, the programmes managed by DG RTD, other DGs (in particular EMPL, CONNECT) and to what extent did the Health Programme underpin the Commission’s general objectives -focus on Europe 2020 and their objectives related to social policy (e.g. the renewed Social Agenda) and economic growth (research and innovation, competitiveness)? Extent to which health interventions funded under the HP and key other programmes have potential synergies (theoretical analysis) Extent of actual synergies (illustrated by examples) Level of coordination at planning and implementation level (delivery) Comparison of health-related funding (HP, FP7, ERDF, ESF) Specific examples of synergy effects (if any) Existence of EU-level mechanisms for coordination of priorities, exchange of results and/or cross-fertilisation Satisfaction of key staff with the degree of synergies / complementarity Desk research (including value chain analysis) Interviews with officials representing other services and programmes Coherence of HP with main EU policy objectives (Europe 2020) No. of HP2 projects consistent with Europe 2020 objectives Effects of HP2 actions on smart / inclusive growth Desk research Annex II – Recommendations applied





Recommendation(s) of the mid-term evaluation of the HP 2008-2013 Intended effect(s) Preliminary assessment of implementation Issues tested further during the ex-post final evaluation of the HP 2008-2013 CONCEPTION More tangible and focussed objectives (SMART); better defined strategic framework for the HP and the development of indicators To ensure the programme is focussed on certain public health issues (especially those that are difficult for MS to reach individually) and for targets / indicators to be developed to measure the extent to which these objectives / priority areas are achieved as well as greater transparency in how priorities are reached in Annual Work Programmes (also see “Design” Recommendation – to give EU added value greater prominence in proposal stage) 2nd HP: 2012 and 2013 AWPs focused on Europe 2020 relevance Evidence of shift from ‘bottom up’ development process to ‘strategic’ prioritisation (based on increased input from DG SANTE senior leaders) To what extent were indicators developed at a strategic level (rather than action level) under the 2nd HP to show progress / achievements relative to the stated objectives? 3rd HP: Objectives re-defined for the 3rd HP with the result that they are much more focused on supporting EU priorities and therefore the pursuit of EU added value Progress indicators have been introduced for 3rd HP (however there is no reference to them in official documents consulted to date, e.g. the AWP 2014) Develop long-term planning and targets in consultation with national health experts To facilitate: Setting appropriate priority actions vis-à-vis health needs and objectives in individual Member States Selecting financing mechanisms To ensure an appropriate spread among the objectives and priorities. Greater share of funding allocated to joint actions, service contracts and operating grants (cf. smaller share to presidential conferences, projects) from 2011 – 2013 cf. 2008-2010. More emphasis (both in terms of number of actions and budget spent) on “Health Promotion” cf. “Health Security” and “Health Information”. According to our preliminary assessment, the Annual Work Plans do not provide a clear rationale / justification behind varying levels of funding for each objective All countries (with the exception of Luxembourg) received more funding for Health Promotion than Health Security or Health Information, although there were some exceptions Assess in greater detail whether DG SANTE explain and document this process clearly and whether the AWPs provide sufficient rationale / justification behind varying levels of funding for each objective Has value for money improved through greater use of JAs? It remains to be seen (through survey analysis and interviews) whether divergences in the focus of funding by strand can be attributed to differing health needs in each country DESIGN Retain current funding mechanisms and consultation of action leaders on their experience with new funding mechanisms To continue to provide insight into the pros and cons of each funding mechanism, what aspects they would change / improve at the end of each action All funding mechanisms continue to be used but as described above, the relative proportion of budget allocated to each funding mechanism shifted following on from the Mid-term evaluation Look into degree of consultation with action leaders once an action has come to a close Retain requirement for proposals to outline compliance with AWP priority areas and HP objectives DG SANTE continue assessing proposals according to their policy relevance External evaluators continue rating proposals according to their evidence base Guides include information on awarding criteria which make reference to policy and contextual relevance of action as well as reference to evidence base Emphasise and clarify EU added value particularly in the proposal/application process stage The promotion of clearer EU AV allows the programme to better achieve its objective All EU AV criteria are still not altogether clear Poster on EU added value produced for High Level Conference on Health Programmes EU Added Value is included in the application guides but the information has not evolved since the mid-term evaluation / does not make reference to clear guidelines on EU added value Meaning of EU added value is outlined in FAQ for Health Programme calls for proposals 2013 on the CHAFEA website (not clear when this was first included) 3rd HP: EU AV criteria are detailed in Programme Regulation 282/2014/EC (6) Reference is made to EU AV in the (annexes to) AWP 2014 Calls for proposals for actions make reference to EU AV and factsheet on EU AV to be distributed along with calls for proposal from 2014 onwards as well as a reference included in the guides for applicants Info days for potential applicants organised on 11 April and 11 June 2014 Clarity /applicability of EU AV (in general and for applicants)? MANAGEMENT Monitor the organisations applying for funding and carry out a more in depth assessment of sample of actions To ensure there is equal access for all applicants to receive funding To provide insight into the actions funded but also to have data available to disseminate to interested parties Annual c