COMMISSION STAFF WORKING DOCUMENT Evaluation of EU Agencies: Eurofound, Cedefop, ETF and EU-OSHA Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL AND THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE Evaluation of EU Agencies: Eurofound, Cedefop, ETF and EU-OSHA
Date: 2024-09-27
Institutions: Directorate-General for Employment, Social Affairs and Inclusion, European Commission
Subjects: Cedefop, Eurofound, European Agency for Safety and Health at Work, European Training Foundation, operation of the Institutions, vocational education, working conditions

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 27.9.2024  SWD(2024) 222 final  COMMISSION STAFF WORKING DOCUMENT Evaluation of EU Agencies: Eurofound, Cedefop, ETF and EU-OSHA              Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL AND THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE Evaluation of EU Agencies: Eurofound, Cedefop, ETF and EU-OSHA  {COM(2024) 414 final}  
 
 
Table of contents 

1. 

INTRODUCTION ................................................................................................................. 5 

1.1. 

PURPOSE AND SCOPE OF THE EVALUATION ............................................................................... 5 

1.2. 

EVALUATION METHODOLOGY AND LIMITATIONS ........................................................................ 6 

2.  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION .................... 9 

2.1. 

DESCRIPTION OF THE INTERVENTION AND ITS OBJECTIVES ............................................................ 9 

2.2. 

POINT(S) OF COMPARISON .................................................................................................. 11 

3.  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? .. 12 

3.1. 

FINANCIAL AND HUMAN RESOURCES ..................................................................................... 12 

3.2. 

DELIVERING OUTPUTS AND ACTIVITIES ................................................................................... 14 

3.3. 

FACTORS FACILITATING/HINDERING EFFECTIVENESS ................................................................. 15 

4. 

EVALUATION FINDINGS ............................................................................................... 16 

4.1. 

TO WHAT EXTENT WAS THE INTERVENTION SUCCESSFUL AND WHY? ............................................ 16 

4.2. 

HOW DID THE EU INTERVENTION MAKE A DIFFERENCE AND TO WHOM? ...................................... 57 

4.3. 

IS THE INTERVENTION STILL RELEVANT? .................................................................................. 62 

5.  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? .............................. 65 

5.1. 

CONCLUSIONS ................................................................................................................... 65 

5.2. 

LESSONS-LEARNED ............................................................................................................. 70 

ANNEX I:   PROCEDURAL INFORMATION ......................................................................... 76 

ANNEX II:  METHODOLOGY AND ANALYTICAL MODELS USED ............................... 81 

ANNEX III:  EVALUATION MATRIX ..................................................................................... 93 

ANNEX IV:  OVERVIEW OF BENEFITS AND COSTS ...................................................... 101 

ANNEX V:  STAKEHOLDER CONSULTATION – SYNOPSIS REPORT ........................ 108 

ANNEX VI: AGENCIES INTERVENTION LOGICS ........................................................... 126 

ANNEX VII: FIGURES AND TABLES ................................................................................... 133 

ANNEX VIII: TECHNICAL ANNEX ...................................................................................... 148 

ANNEX IX:  TECHNICAL ANNEX (II) ................................................................................. 158 

ANNEX X: LIST OF INDICATORS PER AGENCY ............................................................. 175 

ANNEX XI: COST-EFFECTIVENESS OF THE AGENCIES’ KEY ACTIVITIES ........... 183 

ANNEX XII: RESULTS ACHIEVEMENT ............................................................................. 187 

 
 
 
 
 
 
 
 
 
 
Term or acronym 

Meaning or definition 

Glossary 

AAR 

ACQF  

ACVT  

CAAR  

CATI  

Annual activity report 

African Continental Qualifications Framework 

Advisory Committee on Vocational Training 

Consolidated annual activity report 

Computer assisted telephone interview 

CEDEFOP 

European Centre for the Development of Vocational Training 

DG COMM  

Directorate General for Communication 

DG EAC  

Directorate General for Education, Youth, Sport and Culture 

DG EMPL  

Directorate General for Employment, Social Affairs and Inclusion 

DG HOME  

Directorate General for Migration and Home Affairs 

DG INTPA  

Directorate General for International Partnerships 

DG NEAR  

Directorate General for Neighbourhood and Enlargement Negotiations 

DG SANTE  

Directorate General for Health and Food Safety 

DGVT  

EBRD  

EC  

ECA  

ECDC  

EEA  

EFCA  

EFTA  

EIGE  

ELA  

EMAS  

EMCC  

EMU  

EP  

Directorate General for Vocational Training 

European Bank for Reconstruction and Development 

European Commission 

European Court of Auditors 

European Centre for Diseases Prevention and Control 

European Environment Agency 

European Fisheries Control Agency 

European Free Trade Association 

European Institute for Gender Equality 

European Labour Authority 

Eco-management and audit scheme 

European Monitoring Centre on Change 

Economic and Monetary Union 

European Parliament 

 
 
 
 
 
EPMS  

EPSCO  

Eurofound performance monitoring system 

Employment, Social Policy, Health and Consumer Affairs Council 

EQAVET  

European Quality Assurance in Vocational Education and Training 

EQF  

ERM  

ESCO  

European Qualifications Framework 

European Restructuring Monitor 

European Skills, Competences, Qualifications and Occupations 

ESENER  

European survey of enterprises on new and emerging risks 

ESJS  

ESOSH  

ETF  

ETUC  

EU  

EUAN  

European skills and jobs survey 

European Society of Occupational Health and Safety 

European Training Foundation 

European Trade Union Confederation 

European Union 

European Union Agencies Network 

EU-ANSA  

European Union Agencies Network on Scientific Advice 

EU-OSHA  

European Union Agency for Safety and Health at Work 

EUR  

Euro 

EUROFOUND  

European Foundation for the Improvement of Living and Working Conditions 

EWCS  

FOP  

European working conditions survey 

EU-OSHA focal point 

GRETA  

Green responses to excellence through thematic actions 

HCD  

HR  

HWC  

IA  

IAS  

ICT  

ILO   

IOSH  

IPA  

Human capital development 

Human resources 

Healthy workplace campaign 

Internal audit 

Internal audit system 

Information and communication technology 

International Labour Organisation 

Institution of Occupational Safety and Health 

Instrument for Pre-accession Assistance 

 
 
 
 
 
IT   

KPI  

MB  

MFF  

MS  

MSD  

NDICI  

NEC  

NEET  

NGO  

NQF  

OECD  

OiRA  

OPC  

OSH  

PD 

PTSD  

SAI  

Information technology 

Key performance indicator 

Management board 

Multiannual financial framework 

Member States 

Musculoskeletal disorders 

Neighbourhood, Development, and International Cooperation Instrument 

Network of European Correspondents 

Not in education, employment or training 

Non-governmental organisation 

National Quality Framework 

Organisation for Economic Cooperation and Development 

Online interactive risk assessment 

Open public consultation 

Occupational safety and health 

Programming documents 

Post-traumatic stress disorder 

Strategic areas of intervention 

SMART 

Specific, Measurable, Achievable, Realistic, and Timely 

SME  

SWOT  

TFEU  

Small and medium-sized enterprises 

Strengths, weaknesses, opportunities and threats 

Treaty on the Functioning of the European Union 

UNESCO  

United Nations Educational, Scientific and Cultural Organisation 

UNICEF  

UNIDO  

VET  

WHO  

United Nations International Children’s Emergency Fund 

United Nations Industrial Development Organisation 

Vocational education and training 

World Health Organisation 

 
 
 
 
 
1. 

INTRODUCTION 

1.1. 

PURPOSE AND SCOPE OF THE EVALUATION 

This document evaluates four of the five EU decentralised agencies that fall under the remit of 
DG EMPL: the European Foundation for the Improvement of Living and Working conditions 
(Eurofound); the European Centre for the Development of Vocational Training (Cedefop); the 
European Training Foundation (ETF); the European Agency for Safety and Health at Work 
(EU-OSHA) (‘the Agencies').  

Since it was set up in 2019, the European Labour Authority (ELA) has  also been under the 
remit of DG EMPL. However, due to its early stage of development and the on-going 2024 
evaluation as required by its Founding Regulation, ELA does not fall within the scope of this 
evaluation. Nonetheless, ELA stakeholders have been consulted throughout this study, and the 
alignment of the mandates between the Agencies and ELA has been taken into account. 

According to their Founding Regulations1, each of the Agencies should be evaluated at regular 
intervals. The previous joint evaluation of the Agencies took place in 20192. The Commission 
evaluation  report  was  published  on  9 April 20193  along  with  the  supporting  study4.  The 
Agencies also have their own evaluation programmes. 

In  line  with  the  Commission’s  ‘better  regulation’  guidelines,  this  evaluation  analyses  the 
effectiveness, efficiency, relevance, coherence and EU added value of each Agency, assessing 
to what extent they have fulfilled their mandates, specific objectives and activities. It includes 
individual and cross-cutting assessments, with an emphasis on synergies, cooperation and how 
the  Agencies  complement  one  another.  The  evaluation  seeks  to  provide  background 
information and insights to feed into future discussions on the Agencies, and if necessary, to 
contribute  to  potential  legislative  proposals  responding  to  evolving  needs  or  circumstances, 
including changes in their mandates. 

Scope of the evaluation  

The  evaluation  covers  2017-20225,  and  includes  the  entire  thematic  remit  of  the  Agencies: 
employment,  industrial  relations  and  the  improvement  of  living  and  working  conditions 
(Eurofound); health and safety at work (EU-OSHA); vocational education and training, as well 
as skills and qualification policies (Cedefop and the ETF). The evaluation also addresses the 
role  of  each  Agency  in  dealing  with  the  COVID-19  pandemic,  as  well  as  Russia’s  war  of 
aggression against Ukraine, where relevant.  

1 Regulation (EU) 2019/127 establishing the European Foundation for the improvement of living and working 
conditions  (Eurofound),  repealing  Council  Regulation  (EEC)  No  1365/75.  Regulation  (EU)  2019/128 
establishing  a  European  Centre  for  the  Development  of  Vocational  Training  (Cedefop)  repealing  Council 
Regulation (EEC) No 337/75. Regulation (EU) 2019/126 establishing the European Agency for Safety and 
Health at Work (EU-OSHA), repealing Council Regulation (EC) No 2062/94. Regulation (EC) no 1339/2008 
establishing a European Training Foundation (recast). 

2 Including an in-depth assessment of the potential for mergers between the Agencies. See the  2019 supporting 

study Annex 6 Transversal report.  

3 Commission Staff Working Document, Evaluation of the EU Commission Agencies working in the employment 
and social affairs policy field: EUROFOUND, CEDEFOP, ETF and EU-OSHA, SWD(2019)160 final. 

4 PPMI/Ecorys (2018) Evaluation of the EU Agencies under the remit of DG Employment.  
5 Previous evaluation time scope covered from 2011 to 2016. 

4 

 
 
 
In terms of geographical scope, the evaluation covers the 27 EU Member States, as well as:  

-  for  the  ETF:  activities  in  partner  countries,  including  in  Eastern  Partnership  countries, 
Central Asia, Southern and Eastern Mediterranean, Western Balkans and Türkiye6;  
- for Eurofound: surveys in non-EU Member States7;  
- for Cedefop: the contribution of Norway and Iceland to Cedefop’s budget and activities8;  
- for EU-OSHA: the participation of Iceland, Lichtenstein and Norway in EU-OSHA's work 
(European  Commission  and  EFTA  agreement);  the  projects  run  by  EU-OSHA  in  the 
Western Balkans and Türkiye (under the Instrument for Pre-accession Assistance - IPA). 

1.2.  EVALUATION METHODOLOGY AND LIMITATIONS 

1.2.1 METHODOLOGY 

The evaluation was supported by an external study (supporting study)9. The supporting study 
used a variety of primary and secondary data collection tools as well as triangulated analysis: 
mapping research based on the Agencies' management and monitoring documents, other studies 
and  reports  related  to  the  Agencies,  stakeholder  feedback  surveys  (commissioned  by  the 
Agencies), a wide range of consultation activities including interviews with and surveys of key 
stakeholders and staff working in the Agencies.  

The  supporting  study10  included  20  Agency-specific  and  five  cross-cutting  in-depth  case 
studies. It also made use of a European Commission public consultation published on ‘Have 
your  say’  to  collect  feedback  on  the  evaluation  criteria  covered  by  the  study.  A  cost-
effectiveness analysis focused on assessing Agency-level and activity-level cost-effectiveness. 
The cross-cutting analyses11 assessed the mandates and cooperation between the Agencies and 
provided significant input to answer the coherence questions. Finally, a validation focus group 
with key stakeholders took place on 18 September 2023 to test preliminary evaluation findings. 

1.2.2 LIMITATIONS  

Limitations to the evaluation research are associated with the scope, research coverage, data 
availability and data quality. Many of these limitations stem from the Agencies’ monitoring 
systems and key performance indicators (KPIs) which restrict the assessment of performance 

6  ETF  partner  countries  are:  Albania,  Algeria,  Armenia,  Azerbaijan,  Belarus,  Bosnia  and  Herzegovina,  Egypt, 
Georgia, Israel, Jordan, Kazakhstan, Kosovo (this designation is without prejudice to positions on status, and 
is in line with UNSCR 1244 and the ICJ opinion of the Kosovo declaration of independence), Kyrgyzstan, 
Lebanon, Libya, Moldova, Montenegro, Morocco, North Macedonia, Palestine (this designation should not be 
construed as recognition of a State of Palestine and is without prejudice to the individual position of the EU 
Member States on this issue), Russia, Serbia, Syria, Tajikistan, Tunisia, Türkiye , Turkmenistan, Ukraine and 
Uzbekistan. ETF stopped all its operations in Russia in February 2022. 

7 Non-EU countries covered in some of Eurofound surveys are Albania, Bosnia and Herzegovina, Iceland, Kosovo, 

Montenegro, North Macedonia, Norway, Serbia, Switzerland, Türkiye and the United Kingdom. 

8 From 2023, Lichtenstein is also involved in Cedefop’s activities. 
9 The supporting study was carried out by Ecorys/PPMI and was coordinated by the Commission’s Directorate-
General for Employment, Inclusion and Social Affairs, with the support of an Inter Services Steering Group 
(ISSG). See details in Annex I. 

10 See detailed overview of the methods used in Annex II. 
11 See Annex V of the supporting study. 

5 

 
 
  
 
and achievement of objectives. These constraints were taken into account during the study’s 
design and implementation, with mitigation measures introduced accordingly. 

1.2.2.1. Agencies monitoring systems and KPIs limitations:   

Overall,  the  Agencies  have  numerous  and  similar  mechanisms  to  ensure  accountability  and 
assess  performance,  including:  monitoring:  internal  performance  monitoring/measurement 
systems,  mechanisms to track risks; reporting: single programming documents and multiannual 
strategies, annual reports and consolidated annual activity reports, annual budgets and accounts, 
establishment  plans,  management  board  and  executive  board  meeting  minutes;  evaluation: 
reports by ECA, EP, and IAS, internal and external evaluations and controls, staff surveys and 
internal user/stakeholder surveys (except ETF).  

Despite growing cooperation between the Agencies to increase the alignment and quality of 
their monitoring systems, notable differences and limitations persist:  

Limited comparability of KPIs among the Agencies: Although each Agency collects a large 
amount of data, these data are not always comparable due to variations in methodology when 
defining and quantifying KPIs. To address this, priority was given to KPIs that are common to 
most Agencies; proxy indicators were used where possible. Qualitative information from other 
data collection methods was also used to triangulate the findings (e.g. Cedefop does not use a 
quantitative indicator for work programme delivery so alternative qualitative information was 
used, including European Parliament discharge reports).  

Result indicators and benchmarking limitations to assess success:  

(i)  The  Agencies  did  not  consistently  set  targets  for  performance  indicators,  which  made  it 
difficult to analyse whether the objectives had been achieved. While targets were set for some 
operational KPIs (e.g. budget implementation and output delivery), this was not the case for 
most result indicators (see Annex X for a list of Agencies’ KPIs with and without targets). In 
such  instances,  the  study  analysed  trends  over  time,  comparing  achievements  with  2016 
baseline figures or with data from the entire 2011-2016 period when available. 

(ii) ETF and Cedefop changed their approach to monitoring indicators during the evaluation 
period making it difficult to analyse some trends12. In 2021, the ETF changed its approach to 
monitoring  indicators  with  the  introduction  of  the  2027  ETF  strategy,  which  reduced  data 
comparability. In those cases, indicators are used to a limited extent and explanations are given 
based on triangulated evaluation sources.  

(iii) In some cases, the ETF set targets, but the ETF consolidated annual activity reports do not 
provide sufficient information for assessing the achievement of those quantitative targets. As a 
result,  there  is  a  disconnect  between  the  specified  targets  and  the  description  of  what  was 
achieved13. 

12 For example, Cedefop changed its approach to monitoring website traffic in 2019. 
13 In 2021, the Agency introduced a set of new output indicators (e.g. number of reports and knowledge products 
produced, Torino process assessment). However, these were not included in the annual report for 2022. 

6 

 
 
 
(iv) Some of the proposed indicators are no longer relevant for assessing success (e.g. number 
of press releases). 

Lack of KPIs on country-level impacts: The data on results and impacts at national level is 
limited  and  is  not  systematically  monitored  by  the  Agencies.  The  responses  to  the  public 
consultation  and  stakeholder  survey  were  insufficient  to  analyse  the  results  by  country.  To 
address  this,  qualitative  information  collected  through  interviews,  case  studies  and  mapping 
tasks was used to identify examples of uptake, use, and impact of Agencies activities at national 
level. Evaluations of specific programmes and activity areas commissioned by the Agencies 
were also used (e.g. the evaluation of the EU-OSHA OiRA tool was used to identify uptake and 
use at national level alongside information from the OSH-wiki website). 

1.2.2.2. Other limitations 

Mapping  methodology:  The  mapping  task,  based  mainly  on  the  Agencies’  monitoring  and 
evaluation  systems,  had  limited  scope  to  incorporate  outsider  views.  Interviews,  a  public 
consultation, a targeted survey and case studies were therefore needed to ensure that a variety 
of perspectives could be considered in the evaluation.  

Limitations  of  the  public  consultation  results:  The  public  consultation  received  only  101 
responses  across  all  four  Agencies,  limiting  the  scope  to  analyse  responses  by  stakeholder 
group. To address this, the supporting study  included a targeted survey of Agency staff and 
stakeholders which generated 560 responses. This was sufficient to enable staff and stakeholder 
responses  to  be  analysed  separately,  with  the  latter  further  broken  down  by  replies  from 
Management/Governing Board members and other stakeholders. This allowed a differentiation 
between  those  involved  in  the  Agencies’  operations  and  those  using  the  Agencies’  outputs. 
Public consultation and targeted survey data were also triangulated with all other data sources 
collected during the study. 

Potential bias from evaluation tools based on perceptions: The evaluation collected staff and 
stakeholder  views  through  semi-structured  interviews,  a  targeted  survey  and  a  public 
consultation. To address any potential bias, the study considered that opinions were influenced 
by the stakeholders’ relationships with the Agencies. The perception-related data were therefore 
triangulated with other data sources. For instance, in the case of complex issues such as the 
functioning  of  the  Management/Governing  Boards,  in-depth  interviews  with  both  direct 
participants and informed outsiders were used to inform the findings. 

Limitations of the cost effectiveness analysis methodology: Typically, a cost-effectiveness 
analysis assesses if the cost of the activities match the achieved outputs and results. Yet, for 
many of the Agencies’ activities, it was not possible to identify a single effect or indicator that 
could be easily quantified. Unit cost analysis at the level of outputs was difficult as each Agency 
is unique, with its own remits, objectives and activities, making it difficult to apply a set of 
comparable performance indicators. Consequently, the performance of the Agencies had to be 
considered against multiple objectives rather than the achievement of a sole output by a given 
date. To address these challenges, a mixed-method approach was used to gather information 
through  desk  research,  interviews,  surveys  and  case  studies.  The  goal  was  to  evaluate 

7 

 
 
 
production efficiency (the relationship between inputs and outputs), allocation efficiency (the 
relationship between inputs and higher-level results) and how efficiency could be improved.  

Challenges linked to specific evaluation criteria: The effectiveness criterion assesses impacts 
which are affected by multiple factors that are often outside of the control of the Agencies. In-
depth analyses of selected initiatives in case studies addressed these challenges. Moreover, the 
Agencies’ specific objectives, based on their programming documents, are not formulated in a 
SMART way, which makes it difficult to assess the extent of the success of the interventions.  
The efficiency criterion was affected by the limitations of the cost-effectiveness methodology 
presented above.  

2.  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION 

2.1.  DESCRIPTION OF THE INTERVENTION AND ITS OBJECTIVES 

This  evaluation  builds  on  the  intervention  logics  (ILs)  of  the  Agencies.  Each  IL  includes  a 
complex set of operational, specific and general objectives for each Agency. These objectives 
are mirrored, respectively, by activities/outputs, results and impacts, which are the core of the 
Agencies’ effectiveness and efficiency assessment (see Section 4.1.1.).  

The objectives and tasks of the Agencies are set out in: (i) their Founding Regulations – which 
set  the  mandates/general  objectives;  and  (ii)  their  programming  documents  for  the  relevant 
period  (midterm  perspectives  and  work  programmes)  -  which  set  specific  and  operational 
objectives. Detailed intervention logics per Agency are presented in Annex VI, along with full 
references to the Agencies’ key strategic programming documents.  

To fulfil their objectives19, the Agencies carry out several broad types of activity, depending on 
their  specific  mandate  and  intervention  logic,  namely:  (i)  research  and  monitoring;  (ii) 
communication, dissemination and raising awareness; and (iii) capacity building.   

During  the  current  evaluation  period  several  external  factors  need  to  be  considered  in  the 
Agencies’ ILs, as they influenced the achievement of results and impacts. The most significant 
factor, common to all four Agencies, is the impact of the COVID-19 pandemic. Agency-specific 
external factors include the level of engagement of national focal points (for EU-OSHA), the 
Network of National Correspondents - NEC (for Eurofound), the geopolitical context and the 
level of engagement of partner countries (for the ETF).  

The  mandates  of  the  Agencies,  which  correspond  to  the  general  objectives  of  the  ILs,  are 
presented below (in bold). Detailed specific objectives per Agency, are presented in Annex VI. 
Operational objectives are presented in Table 2. 

EUROFOUND  -  Eurofound  was  established  in  1975  by  Regulation  (EEC)  No  1365/7514. 
Together with Cedefop and EU-OSHA, it was one of the first three EU Agencies to be created. 
Located  in  Dublin,  Ireland,  its  main  mission  is  to  carry  out  research  in  the  areas  of 

14 The Founding Regulation was amended in 1993, 2003 and 2005, mainly to take account of enlargement or treaty 
changes, and it was ultimately repealed in 2019 by Regulation (EU) 2019/126 of the European Parliament and 
of the Council of 16 January 2019. 

8 

 
 
 
 
employment, industrial relations, and living and working conditions, with the objective of 
shaping and implementing policies aimed at improving living and working conditions.  

The 2021-2024 programming document of Eurofound sets out six strategic areas15 on which to 
focus: (1) working conditions and sustainable work; (2) industrial relations and social dialogue; 
(3) employment and labour markets; (4) living conditions and quality of life; (5) anticipating 
and  managing  the  impact  of  change;  and  (6)  promoting  social  cohesion  and  convergence. 
Eurofound  operates  three  observatories:  (1)  the  European  Observatory  of  Working  Life 
(EurWORK - working conditions and industrial relations); (2) the European Monitoring Centre 
of Change (EMCC - monitoring of change for employment); and (3) the European Observatory 
on Quality of Life (EurLIFE - living conditions and quality of life).  

Eurofound  conducts  regular  surveys:  the  European  working  conditions  survey  (EWCS),  the 
European company survey (ECS), the European quality of life survey (EQLS), and living and 
working in relation with COVID-19. The Agency also carries out individual research projects. 
In  its  field  of  competence,  it  contributes  to  the  analytical  and  policy  work  of  the  European 
Commission  and  Member  States,  and  of  organisations  representing  both  employers  and 
employees.  In  2023,  Eurofound  had  around  100  staff  members  (own  staff  only)  and  a  total 
annual  budget  of  EUR  23.8  million,  including  an  operational  budget  of  EUR  6.1  million 
(programming document 2021-2024).  

CEDEFOP - Cedefop was established in 1975 by Council Regulation (EEC) No 337/197516. 
Located in Thessaloniki, Greece, its main mission is to support the promotion, development 
and implementation of evidence-based, high-quality EU policies on vocational education 
and  training  (VET).  Cedefop  provides  information,  research,  analyses  and  evidence  at  the 
crossroads of VET, skills and qualifications, and the labour market. More specifically, Cedefop 
monitors and analyses developments in these three areas. It contributes to European tools in the 
area of education and training, such as the European Qualification Framework and Europass. It 
provides  skills  analysis  and  forecasting  together  with  tools  for  validating  non-formal  and 
informal  learning.  As  a  platform  for  policymakers,  social  partners,  researchers,  experts  and 
other VET and labour market actors, Cedefop promotes knowledge sharing and policy learning. 
It plays a key role in monitoring the implementation of the Council Recommendation on VET 
and  the  Osnabrück  Declaration.  To  support  its  work,  Cedefop  manages  several  networks, 
including  ReferNet,  CareersNet,  SkillsNet,  and  a  community  of  apprenticeship  experts.  In 
2023, Cedefop had 120 staff17 and an annual budget of around EUR 19.6 million, including an 
operational budget of around EUR 4.9 million (programming document 2023-2025).  

ETF – the ETF was established under Council Regulation (EC) No 1360/90, replaced in 2008 
by  Regulation  (EC)  No  1339/200818.  Located  in  Torino,  Italy,  it  focuses  on  EU  policy  and 
cooperation instruments for external relations, which in 2021 were brought together under the 

15 Linked to the strategic areas of intervention (SAIs). 
16 Cedefop’s Founding Regulation was amended in 1993, 1994, 1995, 2003 and 2004 mainly to take account of 
enlargement or treaty changes, and it was ultimately repealed in 2019 by Regulation (EU) 2019/126 of the 
European Parliament and of the Council of 16 January 2019. 

17 Establishment plan 2023; includes officials, temporary agents, contract agents and seconded national experts. 
18  This  2008  Regulation  moved  the  focus  for  ETF  from  vocational  education  and  training  reforms  to  a  much 

broader scope of activities covering human capital development (HCD). 

9 

 
 
 
umbrella of Neighbourhood, Development, and International Cooperation Instrument – Global 
Europe (NDICI - Global Europe). Its main mission is to contribute to EU external relations 
policies, notably on helping partner countries to develop human capital through improved 
VET and lifelong learning systems that are inclusive and innovative.  

The  ETF’s  geographical  scope  includes  the  EU  candidate  countries,  the  potential  candidate 
countries in the western Balkans, the southern Mediterranean, eastern European and southern 
Caucasus partner countries, and other countries, as decided by the ETF Governing Board. In 
2023  it  had  around  129  staff  and  an  annual  budget  of  EUR  22.63  million,  including  an 
operational budget of EUR 4.33 million (ETF annual work programme 2023). 

EU-OSHA – EU-OSHA was established in 1994 by Council Regulation (EC) No 2062/943119. 
Located in Bilbao Spain  its main mission is  to develop, gather and provide reliable and 
relevant  information,  analyses  and  tools  to  advance  knowledge,  raise  awareness  and 
exchange occupational safety and health (OSH) information and good practices.  

The Agency collects and disseminates information and develops tools on health and safety at 
work relevant for stakeholders and policymakers involved in OSH at EU and national level. It 
also  carries  out  awareness  raising  campaigns  and  networking  in  the  OSH  field.  EU-OSHA 
anticipates  change  as  well  as  new  and  emerging  risks.  It  develops  tools  for  good  OSH 
management. It has a network of focal points in Member States, which provides input into EU-
OSHA’s work and disseminates products and information to national stakeholders. In 2023 it 
had around 65 staff and an annual budget of EUR 17.04 million, including an operational budget 
of EUR 7.065 million (programming document 2023-2025).  

Three  of  the  Agencies,  Eurofound,  Cedefop  and  EU-OSHA,  are  ‘tripartite’  Agencies.  This 
means that national authorities, trade unions and employer representatives (one per Member 
State and category) participate in their management boards20 (see details in Section 4.1.2.). 

2.2. 

POINT(S) OF COMPARISON  

Given that the evaluation period is 2017-2022, the baseline year is 2016. The supporting study 
has therefore taken into consideration the key findings of the previous evaluation (2011-2016). 
In  addition,  and  where  possible,  data  relating  to  indicators  comparable  with  those  assessed 
previously has been collected. For specific evaluation criteria (e.g. efficiency), other similar EU 
decentralised agencies have been used as benchmarks; for effectiveness, extensive use is made 
of  the  Agencies  monitoring  systems  and  KPIs  (see  Annex  X  -  list  of  KPIs  per  Agency), 
comparing achievements to targets when the latter exist. 

19 Its Founding Regulation was amended in 1995, 2003 and 2005, mainly to take account of enlargement or treaty 
changes and it was ultimately repealed in 2019 by Regulation (EU) 2019/126 of the European Parliament and 
of the Council of 16 January 2019. 

20 With 27 government representatives, 27 representatives of employers and 27 of workers’ organisations, plus 

three Commission representatives. 

10 

 
 
 
 
 
3.  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? 

3.1. 

 FINANCIAL AND HUMAN RESOURCES 

3.1.1 FINANCIAL RESOURCES  

The Agencies, similarly to other EU decentralised agencies, are almost exclusively funded by 
the  general  EU  budget,  with  annual  contributions  from  the  cohesion,  resilience  and  values 
headings of the multiannual financial framework (annual subsidy to each Agency).21 According 
to their Regulations, all four Agencies can also receive (i) fees charged for their services and 
publications,  (ii)  funding  from  the  Member  States  and  third  countries  participating  in  the 
Agencies’ activities22, and (iii) internal and external revenue23 assigned for specific items of 
expenditure24.  

According to their Founding Regulations, Eurofound, Cedefop and EU-OSHA can also receive 
(iv)  other  funding  from  the  European  Commission  (through  contribution  agreements,  grant 
agreements and financial framework partnerships), provided that specific conditions are met25. 
While the ETF’s Founding Regulation does not mention these agreements explicitly, it permits 
funding from other sources as well (e.g. contribution and grant agreements with the European 
Commission, if they are covered by mandates and are not already funded). 

In practice, all four Agencies received most of their funding from the general EU budget 
(subsidy). The funding received from other sources26 represented less than 5% of their total 
annual revenue27. The ETF is an exception as in 2022, it received additional funding totalling 
about EUR 3 million (approx. 13.8% of the ETF budget) through extra-subsidy projects, mainly 
from  agreements  between  the  European  Commission  and  the  Foundation  (e.g.  with  EU 
Delegations).  This  additional  funding  enabled  the  ETF  to  increase  operational  expenditure, 
improve  visibility  and  amplify  the  impact  of  its  operations  (see  details  of  these  additional   
projects  delivered  by  the  ETF  in  2022  in  Box  1  of  Annex  VII).  In  the  case  of  Eurofound, 
Cedefop and EU OSHA,  some extra-subsidy  funding  comes from  EFTA  countries (Iceland, 
Lichtenstein  and  Norway),  which  in  exchange  participate  in  some  agencies’  activities28  .  

21 European Court of Auditors (2023), Annual Report on EU Agencies for the Financial Year 2022. Available at 
https://www.eca.europa.eu/ECAPublications/SAR-AGENCIES-2022/SAR-AGENCIES-2022_EN.pdf  

22 Explicitly mentioned in the Founding Regulations of the tripartite Agencies. 
23 They are specified in Article 20 of Eurofound’s Founding Regulation. 
24 See Article 15.3 of the ETF Regulation: ‘The revenue of the Foundation shall comprise, without prejudice to 
other  types  of  income,  a  subsidy  from  the  general  budget  of  the  European  Union,  payments  made  as 
remuneration for services performed as well as finance from other sources’. 

25 Eurofound Regulation (EU) No 2019/127 of the European Parliament and of the Council of 16 January 2019, 

OJ L 30 of 16 January 2019 (p. 46). 

26 For instance, Eurofound and EU-OSHA received grant funding from DG NEAR during most of the evaluation 
period  to  support  the  participation  of  enlargement  countries  in  the  Agencies’  activities. E.g.  in  2017,  EU-
OSHA signed a grant agreement to support Western Balkan countries and Türkiye (IPA II 2016 programme), 
for a total amount of EUR 290 000 for a period of 2 years. 

27 According to the data presented in the Agencies’ consolidated annual activity reports for 2016-2022. 
28 E.g. in the period 2020-2024, Eurofound has received EUR 932,022 from Norway for covering the costs of the 
Network  of  Eurofound  Correspondents  related  to  the  agency’s  research  projects  in  which  Norway  has 
participated, and for its participation in the European Working Conditions Survey. 

11 

 
 
 
Eurofound has  also  received funding from  a European  Parliament  Pilot  Project  dedicated to 
minimum wages29.  

The annual contribution to all four Agencies from the EU budget remained stable during 
the  evaluation  period,  and  similar  to  the  baseline.  In  2019,  a  slight  (2%)  increase  was 
introduced to the MFF to partially compensate for inflation. However, this modest rise in annual 
revenues  could  not  keep  pace  with  the  actual  level  of  inflation  and  rising  staff  costs. 
Consequently, the proportion of staff costs increased during the evaluation period, leading 
to a decline in the share of operational expenditure30.  

The  Agencies  faced  significant  pressure  on  their  operational  budgets,  which  limited  their 
operations. For example, Cedefop reduced activities directly targeting Members States, and in 
2022,  EU-OSHA  reduced  activities  related  to  the  priority  areas  ‘Facts  &  figures’,  ‘Raising 
awareness and communication’, and ‘Networking’31.  

Between 2017 and 2022, the Agencies implemented about 100% of their annual budget, 
similar  to  the  previous  evaluation  period  (2011-2016),  indicating  a  high  operational 
efficiency (see Section 4.2 on efficiency). Budget implementation was slightly lower for EU-
OSHA, but remained above the target of 95%. This can be explained by the large share of the 
operational budget which is implemented through procurement actions32.  

Table 1: Annual budget implementation (%) 

Agency 

Eurofound 

2016 
(Baseline) 
99.9 

Cedefop 

ETF 

99.9 

99.9 

EU-OSHA 

96.3 

Source: Annual activity reports 

3.1.2 HUMAN RESOURCES 

2017 

2018 

2019 

2020 

2021 

2022 

100 

99.9 

99.9 

96 

99.6 

100 

99.9 

100 

99.9 

99.9 

99.9 

98 

99.9 

100 

99.8 

97 

100 

100 

99.9 

97 

100 

99.9 

100 

98.7 

The total number of staff  33 of the Agencies remained stable during the evaluation period 
(see Figure 1 and Table 1 in Annex VII). EU-OSHA is the smallest Agency with 65 staff in 
2022.  The  ETF  was  the  largest  with  134  staff  in  2022  down  from  141  in  the  baseline  year 

29 Eurofound database on minimum wage rates applicable to low paid jobs.  It started in 2021, completed in 2023.  

Budget: EUR 1,000,000.  New SLA to be signed in 2024 to update the database by mid-2025. 

30  Agencies’  expenditure  is  divided  into  three  categories:  Title  1:  Staff;  Title  2:  Administrative  (infrastructure 
maintenance of buildings, equipment, furniture, software); Title 3: Operational expenditure (key activities, 
including missions, meetings and interpretation, pilot, studies and projects, and communication). 

31 EU-OSHA (2023), Consolidated annual activity report for 2022 (p. 45). 
32 As a result, the prices received are sometimes lower than those planned. 
33 All figures represent the state of employment as of 31 December. They also include staff members outside the 

establishment plan (e.g. contract agents, seconded national experts and structural service providers). 

12 

 
 
 
(2016). Cedefop staff numbers decreased from 121 in 2016 to 116 in 2022. Staff numbers at 
Eurofound remained largely the same at 110 in 2016 and 108 in 2022. 

The  Agencies  completed  in  2018  (EU-OSHA  in  2017)  the  10%  staff  cut  –  a  policy 
introduced in 2013 for decentralised agencies, applicable to permanent staff (included in the 
establishment plans)34. 

3.2.  DELIVERING OUTPUTS AND ACTIVITIES  

3.2.1 DELIVERY OF THE WORK PROGRAMME  

The Agencies largely achieved their annual programme output targets despite budgetary and 
staff limitations and, in the second half of the evaluation period, the COVID-19 pandemic35. 

EUROFOUND - Eurofound exceeded 80% of its work programme delivery target throughout 
the evaluation period. Delivery was lower than the baseline between 2017-2019; this was linked 
to staff cuts of 10% in 2017, which resulted in delays of some outputs and  a reallocation of 
priorities. Eurofound revised its work programme, adapted to remote working and completed 
the activities during the last months of the financial year, despite initial delays36. 

EU-OSHA – EU-OSHA’s work programme completion rates were higher than the baseline, as 
well as its 90% target. Some outputs were cancelled in 2020  due to the pandemic, but this did 
not affect the overall completion rate, which was very high in 2020 (96%). 

ETF – since 2016, the ETF’s work programme delivery remained above its 90% target, except 
in 2020 due to the COVID-19 pandemic. The pandemic severely impacted Agency operations 
due to an early lockdown in Italy and the Agency working in third countries. Most of the outputs 
were re-programmed for the last quarter of the year, with some of them being postponed, due 
to pressures on staff capacity. 

CEDEFOP – Cedefop does not have a specific quantitative indicator to monitor the completion 
rate  of  its  work  programme.  However,  based  on  the  supporting  study’s  assessment  of  the 
outputs, the Agency has fully implemented the annual work programmes. This includes 2020, 
when Cedefop even exceeded its 2020 work plan targets, as noted by the European Parliament’s 
discharge decision37. 

The Agencies have carried out their activities in line with their operational objectives (see Table 
2). Each Agency collects and monitors the data on the outputs produced per year but the level 
of detail, type of outputs and definitions vary between the Agencies (see list of each Agency’s 
indicators in Annex X). For a detailed analysis on the delivery of each operational objective, 

34 The staff target was set in the Commission Communication to programme human and financial resources for 

decentralised Agencies in 2014-2020 - COM(2013) 519 of 10.7.2013-. 

35 For each Agency, COVID-19 was a highly disruptive factor, e.g. cancellation of some events and activities in 
2020 involving physical presence. Eurofound’s survey fieldwork was profoundly affected, as the survey could 
not be carried out face-to-face. EU-OSHA ESENER outputs were also affected, as no dissemination events 
could be held. 

36 Consolidated Annual Activity Report, 2020. 
37 European Parliament decision of 4 May 2022 on discharge in respect of the implementation of the budget of 

Cedefop for 2020 (2021/2119(DEC)).  

13 

 
 
 
 
see Annex VIII and Section 4.1.1.2 of the supporting study as well as Annexes 1, 2, 3 and 4 of 
the supporting study on the specific situation in each Agency. 

Figure 1: Work programme delivery: Eurofound, ETF and EU-OSHA 38 

Source: Consolidated annual activity reports for Eurofound, ETF and EU-OSHA 2016-2022 

3.3. 

FACTORS FACILITATING/HINDERING EFFECTIVENESS 

Facilitating factors  

At  a  general  level,  the  most  significant  internal  facilitating  factors  contributing  to  the 
effectiveness  of  the  Agencies’  operations  were:  (1)  flexibility  to  adapt  to  changing 
circumstances, as a result of proactive internal measures39; and (2) tripartite representation in 
the three tripartite Agencies, which increased the reliability and credibility of the Agencies   vis-
à-vis stakeholders.  

The COVID-19 pandemic could be considered as an external facilitating factor.  The shift to 
online work not only generated efficiency savings, but also allowed the Agencies to enhance 
their visibility by offering information and guidance on COVID-19 related matters. As a result, 
audiences also became aware of the broader work that the Agencies carry  out (e.g. previous 
Eurofound work on youth and gender)40. 

For individual Agencies, internal facilitating factors include Eurofound’s Network of Expert 
Correspondents, the level of engagement of ETF’s partner countries, as well as of EU-OSHA’s 
focal points in Member States. 

Hindering factors  

38 Share of outputs delivered in the planning year versus planned outputs (%). Cedefop does not use quantitative 

indicators for monitoring work programme delivery. 

39 For example, in the case of EU-OSHA, its ‘portfolio approach’ gave the Agency’s national-level focal points 
(FOPs) several optional activities in addition to their mandatory activities, which they could engage in, based 
on their interest and resources. 

40 See details of achieved impacts in the supporting study’s Annexes 1-4 on individual Agencies, as well as the 

case studies (Annex 8). 

14 

 
 
 
 
In general, the most significant internal hindering factor was the lack of financial resources. 
Staff  costs  increased  for  all  Agencies  due  to  inflation  adjustments,  as  did  administrative 
expenditure  (except  for  ETF).  As  a  result,  and  given  that  the  budget  remained  stable,  the 
operational  expenditure  decreased.  Confronted  with  the  need  to  address  new  EU  policy 
priorities, Agencies had in some cases to revise their activities in order to deliver on their work 
programmes.  

Other internal hindering factors included lack of human resources, lack of time to prepare data 
and finalise outputs, and accessibility issues on outputs linked to the quality of translations. 

The COVID-19 pandemic was another external hindering factor. It led to live events having to 
be cancelled and disrupted work, including surveys. 

For individual Agencies, hindering factors included insufficient tailoring of outputs for specific 
target audiences (ETF, EU-OSHA), unforeseen delays with external contractors (EU-OSHA), 
translation  issues  (EU-OSHA41,  ETF42)  and  a  need  to  improve  engagement  with  training 
(Eurofound).  For  the  ETF,  insecurity,  political  instability  and  economic  volatility  were  also 
among the external factors that limited the engagement of partner countries. 

4.  EVALUATION FINDINGS  

4.1.  TO WHAT EXTENT WAS THE INTERVENTION SUCCESSFUL AND WHY? 

The  evaluation  has  assessed  the  effectiveness,  efficiency  and  coherence  of  the  work  of  the 
Agencies  during  the  current  2017-2022  period,  confirming  for  the  three  criteria  the  overall 
positive  assessment  given  in  the  previous  evaluation  (2011-2016),  though  with  room  for 
improvement in some areas (see in particular Section 5.2. on lessons learnt). 

4.1.1 EFFECTIVENESS 

The  evaluation  has  assessed  the  effectiveness  of  the  Agencies  in  achieving  the  objectives 
outlined in their Founding Regulations and programming documents and as reflected in their 
intervention logics (see Annex VI). The evaluation assessed to the extent possible whether the 
Agencies  successfully  met  their  operational  and  specific  objectives  in  terms  of  outputs  and 
results  (see  Section  4.1.1.1)  and  produced  impacts  for  EU  policymaking  or  policy 
implementation  at  national  level  (see  Section  4.1.1.2).  In  addition,  it  assessed  the  Agencies 
ability to adapt to changes and, notably, their responses to the challenges posed by the COVID-
19 pandemic and Russia’s war of aggression. 

4.1.1.1 ACHIEVEMENT OF OBJECTIVES 

41 Need to improve the quality of EU-OSHA translations by ensuring that the original EN text is of high quality.  
42 Language barriers prevent ETF stakeholders from taking full advantage of the resources and services provided 

by the Agency. 

15 

 
 
 
 
Overall,  and  based  on  the  available  evidence,  it  can  be  concluded  that  the  Agencies  have 
achieved many of their operational and specific objectives to a high degree43. There are 
however opportunities for improvement in certain areas. 

A) OUTPUTS (OPERATIONAL OBJECTIVES - ACTIVITIES) 

Overall, the Agencies largely delivered the outputs planned in their annual programming 
documents,  as  shown  in  Section  3.2.  They  managed  to  do  this  in  spite  of  staff  and  budget 
constraints, and in the difficult context of the COVID-19 pandemic.  

Some delays occurred for Eurofound (mainly due to staffing issues) and EU-OSHA (due to the 
editing of final outputs taking longer than expected and delays in getting the deliverables from 
contractors), as well as for Cedefop and the ETF (mainly due to the COVID-19 pandemic). 

Table 2: Agencies operational objectives 

Eurofound  

Cedefop  

ETF  

EU-OSHA 

Monitoring  
Research and data collection  
Communication and dissemination  
Research  
Contribution to EU and national policies  
Data compilation  
Monitoring  
Support in the development of EU tools  
Carry out activities with other agencies/organisations  
Contribute to capacity building  
Dissemination   
Provide information, policy analysis and advice  
Support capacity building and networking  
Monitor system-wide progress  
Support EU project and programming cycle in an EU external relations context  
Support knowledge dissemination   
Develop foresight activities 
Provide OSH facts and figures 
Develop OSH management tools 
Awareness raising and communication 
Knowledge networking 
Maintain and develop strategic and operational networks 

Source: Intervention logics of the Agencies, see Annex VI 

EUROFOUND  -  Eurofound  successfully  delivered  the  expected  outputs  for  all  its 
objectives.  The  Agency  performed  well  in  carrying  out  research  and  collecting  data  (desk 
research, KPIs44 and confirmed by consultations -see below stakeholders’ views on quality of 

43 See below assessment and additional evidence provided in: (i) Annex VIII and XII; (ii) Section 3.2 of the 

supporting study and Section 4.1.1. of the main report; (iii) Agency-specific reports in the supporting study 
(Annexes 1-4). 

44 See list of Eurofound KPIs in Annex X. 

16 

 
 
 
outputs).  The  core  of  the  Agency’s  work  relies  on  its  pan-European  surveys,45  which  were 
delivered  as  planned.  For  instance,  the  2020  European  working  conditions  survey  (EWCS) 
covered  37  countries,  two  more  than  the  2015  edition.  Additionally,  a  follow-up  telephone 
survey  in  2021  assessed  the  impact  of  COVID-19  on  job  quality.  The  2019-2020  European 
company  survey  (ECS),  carried  out  in  cooperation  with  Cedefop,  was  the  first  large-scale, 
cross-national survey introducing a ‘push-to-web’ approach46. Due to COVID-19, Eurofound 
adopted  computer-assisted  telephone  interviewing  (CATI)  instead  of  face-to-face  interviews 
for surveys. This shift prompted the Agency to explore alternative data collection methods to 
save costs in the long run, without compromising the quality47.  

CEDEFOP - Cedefop successfully delivered the expected outputs for all its objectives in 
VET,  skills  and  qualifications.  Cedefop  performed  particularly  strongly  on  its  operational 
objective  of  ‘carrying  out  research  on  topics  relevant  to  the  policy  agenda’  –  especially  on 
‘Skills  anticipation  and matching’-  as  well  as  on  its  objective  of  ‘Monitoring  and  analysing 
VET  policy  developments.  Cedefop  publications  were  frequently  referenced  in  EU  policy 
documents  (with  524  and  449  references  respectively)48.  Although  the  overall  number  of 
publications  slightly  decreased  during  the  evaluation  period  (compared  to  2016),  this  was  a 
deliberate shift from traditional to online content, enhancing accessibility through interactive 
tools and databases. 

In  terms  of  compiling  relevant  data,  Cedefop  made  strides  by  developing  online  tools  and 
databases49, such as Skills Forecast, the European database on apprenticeship schemes, and the 
European  VET  policy  dashboard.  This  move  aimed  to  enhance  user  interaction  and 
accessibility. Cedefop also contributed to capacity building through events and policy learning 
activities, albeit to a lesser extent than in 2016, with a reduction in events starting before 2020 
due to the COVID-19 pandemic50. 

Finally,  Cedefop  successfully  supported  the  development  and  implementation  of  European 
tools to support VET, skills and qualifications policies.  It produced the expected outputs in its 
support of EQF, ECVET and EQAVET, and played a crucial role in the transition to the new 
Europass.  

ETF – the ETF successfully achieved its five operational  objectives, focusing on  partner 
countries in the area of EU external relations. To provide information and policy analysis, the 
ETF  produced  various  informative  materials,  including  reports,  policy  briefings,  guides  and 
toolkits,  offering  valuable  insights  into  human  capital  development  in  partner  countries. 

45 European Working Conditions Survey (EWCS), European Company Survey (ECS), European Quality of Life 
Survey  (EQLS),  and  the  Living,  working  and  COVID-19  e-survey.  The  share  of  expenditure  allocated  to 
survey management and development has increased considerably during this evaluation period, rising from 
7% of expenditure in 2017 (EUR 1.39 million) to 18% in 2022 (EUR 3.91 million). 

46 Where establishments were contacted by phone to identify a management and an employee respondent, who 

were then asked to fill out the survey questionnaire online, see: 
https://www.eurofound.europa.eu/surveys/2019/european-company-survey-2019  

47 Eurofound (2022), Consolidated annual activity report for 2021, p. 3. 
48 Cedefop consolidated annual activity reports 2016-2022. 
49  According  to  the  2022  user  satisfaction  survey,  they  are  Cedefop’s  second  most  appreciated  outputs,  after 

publications. 

50 Cedefop annual reports 2012-2014. 

17 

 
 
 
Stakeholders assessed this ‘information and policy analysis’ objective positively (see Figure 8 
in Annex VII). 

ETF’s  support  on  capacity  building  and  networking  grew  significantly  with  an  increased 
number of events held compared to the baseline51 (a notable rise from 24 events in 2017 to 143 
events in 2021). ETF stakeholders viewed its support to partner countries as largely or partially 
successfully,  according  to  the  2018  Policy  Delphi  (partner  country  governments,  social 
partners, EU Member States, institutions, international organisations)52.  

The fifth round of the Torino Process (2018-2020) covered 100% of partner countries, while 
the sixth round (2022-2024) adopted a new approach based on two separate stands. The first 
strand focuses  on the collection of statistical data and indicators, covering all partner countries. 
The second strand consists of an in-depth review (known as ‘Level 2’) which became voluntary 
in 202253.  Stakeholders, especially partner countries, highly valued the Torino Process for its 
role  in  facilitating  policy  dialogue,  guiding  education  and  training  strategies,  as  well  as 
evaluating ongoing reforms (see ETF's Case Study 5 in Annex 8 to the supporting study). As 
participating  in  the  ‘in-depth  country  policy  review’  has  become  voluntary,  the  ETF  now 
focuses  efforts  on  those  countries  that  are  motivated  to  participate,  ensuring  that  ETF’s 
resources are used where they can have the greatest impact. However,  this means that some 
countries may have opted out of the process, even though they would need support. 

On the objective of supporting the EU project and programming cycle, the  ETF has offered 
guidance and expertise to EU stakeholders. This includes providing input for policy dialogues 
with EU institutions and agencies54. Although the number of requests from EU departments has 
declined since 2020, the ETF has observed that the requests it receives are more substantial55.  

EU-OSHA  –  EU-OSHA  successfully  met  all  six  operational  objectives,  consistently 
achieving timely delivery, with rates exceeding 90% of its outputs delivered on time for 
most activities. The number of outputs related to facts and figures nearly tripled compared to 
the  baseline,  showcasing  a  significant  increase  in  outputs  related  to  networking  knowledge. 
Outputs in other areas remained similar to the 2016 baseline (see Table 1 in Annex VIII). 

The  most  relevant  outputs  falling  under  facts  and  figures  were  the  European  survey  of 
enterprises  on  new  and  emerging  risks  (ESENER),  and  support  with  compliance  for  SMEs 
(source: KPIs from the  2021 annual activity report). The ‘tools for OSH management’ were 
used more widely during the current evaluation period, to enable workplaces to manage their 

51  Data  for  2015  and  2016  show  that  20  and  29  events  were  organised  in  those  years,  respectively  -  ETF 

consolidated annual activity reports: 2015 and 2016. 

52 PPMI (2018), Evaluation of the ETF Functions: Policy Delphi Results 1st round: June/July 2018. 36% (N=263) 
of respondents to the Policy Delphi indicated that the ETF contributed to strengthening the capacity of partner 
country  stakeholders  and  policymakers  ‘to  a  large  extent’  and  a  further  44%  indicated  that  the  Agency 
contributed ‘to some extent’. 

53 An IAS report had found the previous structure of the  TRP, which involved a review in all partner countries 
even  though  each  country  had  different  levels  of  readiness  and  VET  system  maturity,  posed  risks  to  the 
effective implementation of the initiative - IAS (2017), Audit on progress monitoring in VET through the TRP 
in ETF. 

54 e.g. work with the European Institute for Gender Equality on gender gaps in employment in 2021. 
55 They remained stable between 2016 and 2019 with between 105 and 116 requests, and then dropped to 86 in 

2020, 50 in 2021 and 40 in 2022, mainly explained by the impact of COVID-19 pandemic. 

18 

 
 
 
occupational safety and health risks. Over 72 000 new risk assessments were conducted in 2022, 
surpassing both the target of 30 000 and the baseline)56. 

To  raise  awareness  on  OSH  issues,  the  Agency  actively  developed  the  healthy  workplaces 
campaigns (HWC). Through these campaigns, EU-OSHA engaged with national stakeholders 
ranging from SMEs to larger corporations and national bodies. Awareness was raised about the 
risks of dangerous substances in the workplace and preventing work-related musculoskeletal 
disorders57 (see EU-OSHA Case Study 4 in Annex 8 to the supporting study).   

Finally, on its objective of maintaining and developing strategic and operational networks, EU-
OSHA successfully delivered 91% (21 out of 23) of its outputs on time. However, in 2020, the 
COVID-19 pandemic led to the cancellation of four actions and one event58.  

Common operational objective: conducting communication and dissemination activities 

In addition to the Agency-specific objectives, the Agencies shared the operational objective of 
conducting  communication  and  dissemination  activities.  The  degree  of  achievement  varied 
between the Agencies:  

EU-OSHA’s healthy workplaces campaigns (HWC) were successful in raising awareness and 
communicating on OSH issues59.  

CEDEFOP increased its social media reach and the number of visits to its website compared 
to baseline (as shown in Annex VIII).  

EUROFOUND’s communication and dissemination activities were generally appreciated (and 
improvements were made throughout the evaluation period). However, the level of achievement 
of  this  objective  could  be  improved.  For  instance,  Eurofound  monitored  the  uptake  of 
knowledge through the media as of 202160, but no monitoring data were available for social 
media activities.  

The  ETF  increased  its  social  media  use  and  presence  throughout  the  evaluation  period. 
However, despite these improvements, ETF Case Study 2 showed that the effectiveness of the 
ETF’s  efforts  was  hindered  by  an  imbalance  between  delivering  outputs  and  carrying  out 
dissemination activities61. The case study highlighted that prioritising the production of research 
publications  over  communication  and  communication  initiatives  posed  a  challenge  to 
effectiveness62.  

B) SPECIFIC OBJECTIVES (RESULTS) 

56 In 2016, only 13 281 new risk assessments were performed.  
57 For the 2016-2017 and 2018-2019 HWCs campaigns, EU-OSHA delivered 100% of its planned outputs, and 

96% of its planned outputs for its 2020-2022 campaign. 

58 EU-OSHA (2016). Annual activity report for 2016.  
59 See EU-OSHA Case Study 4 in Annex 8 to the supporting study. 
60 This indicator (KPI 5.5) captures articles referring to Eurofound’s research, data, and analysis in recognised 

local, national and international media, distributed via print, radio, television and online sources. 

61 A 2018 Policy Delphi of key stakeholders for an external evaluation of the ETF’s functions also found mixed 
results of stakeholder assessment of the ETF’s communication, dissemination and networking activities. 

62 Explained by the fact that the incentive structure at ETF has been based on the number of publications. 

19 

 
 
 
 
Given the Agencies’ focus on knowledge generation, results are achieved when stakeholders at 
both EU and national level utilise the knowledge outputs, which are of high quality. As outlined 
in the intervention logics of the Agencies (see Annex VI), one common expected result is that 
policymakers  receive  valuable  and  pertinent  data  from  the  Agencies’  outputs,  supporting 
relevant policies. Another result is that stakeholders, in general, receive relevant, timely, and 
high-quality information (see full list of the Agencies expected results in Table 1 of Annex XII). 

Consequently, the achievement of results (meeting the Agencies’ specific objectives) has been 
assessed based on the use of the Agencies’ outputs and services, as well as their quality63. This 
perspective aligns with the results indicators used by the Agencies, primarily focusing on the 
extent  of  stakeholders’  adoption  of  the  Agencies’  outputs.  The  Agencies  also  assess  these 
results through regular user satisfaction surveys.  

B.1) Use of the Agencies’ outputs 

Based on the Agencies’ KPIs64 (e.g. references in EU policy documents, academic literature, 
social media following and download numbers), along with consultations with stakeholders and 
staff65, the evaluation concludes that each of the Agencies’ services and outputs have been 
used by their key  stakeholders. There is  an overall trend of  increased usage  compared 
with  the  previous  evaluation  period.  Usage  by  EU  institutions  was  greater  and  more 
widespread than at national or international level (as shown in Annex VIII). 

EUROFOUND experienced a significant increase in the usage of its outputs for EU-level 
policymaking,  evidenced  by  a  higher  number  of  references  in  key  European-level  policy 
documents, particularly from the European Commission and the European Parliament66. This 
trend  has  been  consistent  since  2011-2016.  The  number  of  downloads  of  Eurofound 
publications from its website was higher than the 2016 baseline figure of 155 943 throughout 
the  evaluation  period,  with  a  peak  in  2020  of  188 556  pdf  downloads,  supported  by  an 
increasing number of social media users. However, a stakeholder survey suggests that there 
is still room to increase the use of and participation in Eurofound's services and activities. 
Only  49%  of  stakeholders  (N=65)  frequently  used  Eurofound's  information,  resources  or 
services, and only 37% (N=65) participated often in the Agency's activities or events. 

The  CEDEFOP  stakeholder  survey  reported  that  43%  (N=174)  of  respondents  ‘often’ 
consult or use the Agency’s information. However, one fifth of respondents (22% - 38 out of 
174) declared that they seldom or never use the Agency's information resources. Since 2020, 
Cedefop downloads have exceeded the 2016 baseline (383 600). Downloads peaked at 465 000 
in 2020 linked to the shift to only disseminating publications digitally. 

Eurofound  and  Cedefop  have  also  substantially  increased  usage  by  researchers  and 
academia  compared  to  the  previous  period  (see  Table  3).  Despite  this  positive  trend,  the 

63 See Annex XII to this SWD and Section 4.1.1., pages 47-60 of the supporting study, for a detailed analysis of 

the results achieved by each Agency. 

64  See Annex VIII for detailed figures on use of outputs.  
65 A clear majority of staff - 93% (N=48) at EU-OSHA, 86% (N= 51) at Eurofound, 85% (N= 45) at Cedefop and 
78%(N= 51) at ETF) responded that they consider either ‘to a large or moderate extent’ that stakeholders take 
full advantage of the information, resources and services provided by the respective Agency.  

66 See figures in Annex VIII. 

20 

 
 
 
supporting study concluded there is  room to increase the outreach of Cedefop's evidence 
through academia and VET providers to ensure wider dissemination of its outputs67.  

Table 3: Number of citations of each Agency's work in academic literature  

Cedefop 
Eurofound 

2016 
591 
627 

2017 
545 
663 

2018 
611 
720 

2019 
838 
880 

2020 
956 
1 034 

2021 
988 
1 220 

2022 
1 030 
1 134 

Source: Annual activity reports, 2016-2022  

ETF - EU-level stakeholders have continued to use ETF’s services and advice, although 
requests from EU departments have decreased since 2020. Regarding beneficiaries from partner 
countries, feedback from staff, stakeholders’ surveys and public consultation suggests that the 
ETF has room for improvement in disseminating its outputs68.  The ETF stakeholder survey 
revealed that 44% (N=91) of stakeholders use the Agency’s outputs ‘often’ while 43% reported 
using  them  ‘sometimes’.  Downloads  of  ETF  publications  saw  a  dramatic  decrease  from 
1 181 462 in 2016 to 351 219 in 201869 (see Table 4, Annex VIII). 

EU-OSHA - Stakeholders actively engaged with EU-OSHA activities and related outputs, 
as evidenced by various user surveys and indicators  (e.g. huge increase in downloads on 
corporate and campaign websites). The Agency saw a huge leap in downloads in 2019 with a 
peak of 2 259 137 in 202070 (compared to a low baseline of 82 558 downloads in 2016). This 
exceeded the target of a 5% increase in downloads per year. The evaluation survey showed that 
a majority of stakeholders often or sometimes participated in the Agency’s activities or events 
(89% of respondents N=75) and consulted or used its information, resources or services (95% 
of respondents N=75).  

The use of the Agencies’ services at Member State level is less well documented than their 
use  by  European  level  stakeholders71.  Nevertheless,  the  evaluation  provides  evidence  of 
effective  use  by  national  stakeholders  depending  on  the  Agencies’  different  remits  and 
activities.  Eurofound  services  used  by  national  stakeholders  include  its  work  on  minimum 
wages.  This  work  is  continuously  referred  to  by  national  wage-setting  institutes,  including 
governments, trade unions and low pay commissions72, as highlighted in the Eurofound case 
study  on  minimum  wages73.  Cedefop  support  to  national  stakeholders  decreased  during  the 
reporting  period.  This  is  because  direct  support  to  Member  States  (e.g.  by  way  of  thematic 
country  reviews  and  policy  learning  forums)  decreased  following  the  Agency’s  decision  to 
prioritise  EU  level  policy  support  in  the  face  of  intense  budgetary  constraints74.  EU-OSHA 
outputs  and tools  have been used intensively by  national  stakeholders, with  outputs such as 

67 Though for VET providers, lack of translations may be an obstacle. 
68 E.g. only 37% of ETF staff respondents (N=) agreed ‘to a large extent’ that stakeholders take full advantage   of 

the Agency’s services. 

69 Data on the number of downloads of ETF’s publications is only available for 2016-2018. 
70 This can be explained by the Agency’s COVID-19 guidance documents, which have been the most downloaded 

and accessed publications in the Agency’s history.   
71 Partly because the latter group are the primary stakeholders. 
72 See Irish Low Pay Commission, ‘Recommendations for the National Minimum Wage’, July 2022. 
73 See Annex 8 to the supporting study. 
74 While there were 46 outputs at the start of the evaluation period (2017), they decreased to only 5 in 2021. 

21 

 
 
 
 
OiRA75, OSH-wiki and the healthy workplaces campaigns (HWCs) seeing increasing use and 
stakeholder participation compared to the 2016 baseline.   

The importance of social media channels for the visibility, uptake and use of the Agencies’ 
outputs increased during the evaluation period. Cedefop and the ETF showed a year-on-
year growth of social media followers across all the channels used by both Agencies76 (with the 
number of followers at least doubling across most channels during the evaluation period). The 
use  of  social  media  channels  is  particularly  important  for  the  ETF  as  many  stakeholders  in 
partner countries rely on information from these channels. Comparable data are not published 
by EU-OSHA77 and Eurofound. 

B.2) Quality of outputs and other results achievements78 

One  specific  objective  /  expected  result  common  to  all  the  Agencies  is  to  provide 
stakeholders with relevant, timely and high-quality information79. The evaluation, based on 
various  Agencies’  indicators,  consultation  tools  and  case  studies,  concludes  that  during  the 
evaluation period, the Agencies achieved this objective to a great extent. Most of the Agencies’ 
outputs and services have received very positive feedback and high user satisfaction, even 
surpassing scores from the previous evaluation period. Table 4 highlights the most used and 
highest quality outputs, as explained in the below Agency assessment. 

EUROFOUND - Eurofound received increased recognition for the scientific quality of its 
research, peaking at 1 220 references in peer-reviewed journals in 2021 (see KPI in Table 3). 
90% (N=65) of stakeholder survey respondents rated the Eurofound research outputs as ‘very 
or rather high quality’ while 89% of the public consultation respondents (N=31) gave similar 
ratings. The Agency's COVID-19-related research was widely acknowledged and contributed 
to EU policy discussions (e.g. on telework and the right to disconnect). Stakeholder feedback 
highlighted  Eurofound's  significant  contribution  to  understanding  working  and  living 
conditions80  and  emphasised  the  consistently  high  quality  of  its  surveys.  A  2022  external 
evaluation81 highlighted the success of the new monitoring convergence activity, contributing 
to timely and policy-relevant knowledge on EU convergence. Eurofound developed in 2018 the 
convergeEU tool, which permitted the analysis of convergence for the full set of indicators of 
the Social Scoreboard82 accompanying the European Pillar of Social Rights. 

75 10 of the 15 OiRA national partners made an explicit reference to the OiRA tool in their national OSH strategy 

or legislation. 

76 ETF started to use Instagram in 2018 and Cedefop started to use LinkedIn in 2021. 
77 EU-OSHA does not publish data on social media followers. 
78 See the detailed analysis of the results of all Agencies in Annex XII. 
79 As knowledge providing agencies, each specific objective corresponds to a particular thematic field/area of the 
agency work and share the common goal of ‘providing high quality information/support to the EU level and 
other stakeholders’. Therefore, this expected ‘common’ result covers multiple specific objectives. 

80 Supporting study Eurofound stakeholder survey (n=65). Replies to open response questions include as main 
achievements  ‘the  knowledge-based  approach  to  work  and  labour’,  ‘timely  and  objective  knowledge 
information about the world of work in Europe’. 

81 ICF (2022), Evaluation of two new crosscutting activities delivered as part of the 2017-2020 programme. The 
digital  age:  opportunities  and  challenges  for  work  and  Employment  and  Monitoring  convergence  in  the 
European Union. 

82  See:  https://ec.europa.eu/eurostat/web/european-pillar-of-social-rights/indicators/social-scoreboard-indicators 

and https://www.eurofound.europa.eu/data/convergence-hub/convergence-methodology 

22 

 
 
 
However, stakeholders have been requesting more visually accessible digital content since 
202083.  While  the  2022  user  satisfaction  survey  showed  high  satisfaction  levels  (91%  of 
Eurofound  stakeholders  were  satisfied  or  very  satisfied  with  research  reports  and  86%  with 
survey information84), a majority (79%) expressed the desire for more digital content, especially 
data visualisation. Furthermore, consultations highlighted the need for more in-depth outputs, 
accompanied  by  additional  methodological  guidance,  considering  the  extensive  amount  of 
primary data Eurofound reports on (Annex 1 to the supporting study, p.140). 

The outputs that have contributed most to Eurofound’s results are: (i) the three pan-European 
surveys; (ii) the living, working and COVID-19 e-survey;  and (iii) the outputs on minimum 
wages (see Table 4).  

The three surveys mentioned in point (i) are key for achieving results, because they underpin 
most of Eurofound research work, and therefore contribute to several specific objectives and 
results.  They  have,  for  example,  contributed  to  the  improvement  of  working  conditions  and 
sustainability of work, labour market functioning and inclusiveness,  as well as to promoting 
social  cohesion  and  convergence,  e.g.  social  convergence  indicators  are  drawn  from  the 
European quality of life survey (EQLS) and the European working conditions survey (EWCS), 
(see also the added value of Eurofound surveys in Section 4.2). 

The ‘Living, working and COVID-19’ e-survey mentioned in point (ii) was conducted in five 
rounds between April 2020 and May 2022. It contributed to the result ‘Key stakeholders have 
increased  understanding  of  ways  in  which  to  improve  quality  of  life,  public  services,  and 
society’. The timeliness and relevance of this research has been crucial for understanding the 
impact of the COVID-19 pandemic; the case study on Eurofound’s contribution to COVID-19 
policy showed that Eurofound more than doubled its media impact thanks to the knowledge 
gathered from this e-survey.   

Finally,  Eurofound’s  work  on  minimum  wage  mentioned  in  point  (iii)  helped  provide 
knowledge to improve working conditions (e.g. the annual review reports blogs ‘How to ensure 
adequate  minimum  wages  in  an  age  of  inflation’85)  and  supported  the  2022  Directive  on 
adequate  minimum  wages  in  the  EU  (see  details  in  Section  4.1.1.2,  and  in  Eurofound  Case 
Study 3)86. 

CEDEFOP - The users’ satisfaction and appreciation of Cedefop’s outputs is high and has 
significantly increased compared to the previous evaluation period. According to the user 
satisfaction surveys conducted in 2019 (N=453) and 2022 (N=486), 99% of respondents in both 
years  found  information  provided  by  Cedefop  to  be  reliable.  In  2019,  95%  (N=463)  of  the 
respondents and 96% in 2022 (N=507) indicated that Cedefop addressed their needs. This is a 
notable improvement from the stakeholder survey conducted for 2011-2016, where only 70-
75% of respondents (N=213 in total) felt that their needs had been met by the Agency's outputs. 

83 ICF (2021), User feedback programme 2017-2020 - capstone report, p.18. 
84 ICF (2023), User feedback survey 2022 - final draft report, p. 11. 
85 Eurofound (2022), ‘How to ensure adequate minimum wages in an age of inflation’, Blog, June 2022,  
86  Directive  (EU)  2022/2041  of  the  European  Parliament  and  of  the  Council  of  19  October  2022  on  adequate 

minimum wages in the European Union. 

23 

 
 
 
Furthermore,  Cedefop  received  the  highest  rating  among  the  Agencies  for  introducing  new 
concepts or ideas in public policy87. Both staff (89% N=45)88 and stakeholders (79% N=174) 
rated it as either ‘very good’ or ‘rather good’.  

Cedefop’s work on monitoring and analysing VET policy developments, skills anticipation and 
matching, and promoting access to and attractiveness of VET, were the three most referenced 
and downloaded areas throughout the evaluation period (see Table 4). Its work on monitoring 
and  analysing  VET  contributed  to  shaping  the  common  priorities  set  out  in  the  Council 
Recommendation on VET and the Osnabrück Declaration89. VET policy monitoring is highly 
rated  by  stakeholders,  with  72%  of  survey  respondents  rating  them  as  very  or  rather  high 
quality.  The  Agency’s  work  on  skills  anticipation  and  matching  was  also  among  the  most 
widely used and appreciated, informing EU employment strategies and wider EU policies (e.g. 
the EU migration network, and the testing of different scenarios in the work of DG GROW, DG 
ENER, and DG ENV).  Finally, the outputs on promoting access to and attractiveness of VET 
were among the most cited (especially by international organisations) and downloaded, though 
less than the above-mentioned areas. These outputs contributed to the strategic area of ‘Valuing 
VET and skills’.  

Cedefop has also achieved positive results on the policy learning objective, facilitating valuable 
exchanges for EU and national  actors. Although  the overall number of meetings and events 
decreased over the evaluation period, the number of participants has increased each year since 
2017 (from 1 159 in 2017 to 1 961 in 2021)90. In 2021, participants gave a 98% rating for the 
‘quality and expected impact’ of events organised by Cedefop, the highest satisfaction level 
during the evaluation period.  

EU-OSHA  -  Similar  to  the  previous  evaluation  period,  most  of  the  EU-OSHA’s  outputs 
received  high  user  satisfaction  ratings91.  The  EU-OSHA’s  online  risk  assessment  tools 
(OiRA) were particularly valued and widely used, indicating a successful achievement of the 
specific objective to provide small workplaces with relevant tools for health and safety 
management. More than 300 OiRA tools were available in 17 EU languages by the end of the 
reporting period, roughly 200 more than in 2016, covering over 30 sectors from manufacturing 
to education. These tools were widely used, especially by SMEs, helping them to initiate risk 
assessments.  The  healthy  work  campaigns  (HWCs)  were  also  widely  used  and  valued  (see 
Annex XII, Figure 4 on stakeholders’ assessment of HWCs), and have been key to achieving 
the  result  of  making  beneficiaries  and  intermediaries  aware  of  workplace  risks  and  how  to 
prevent them (see also Section 4.1.1.2. on generating impacts). 

Other  Agency  outputs,  including  the  foresight  reports  on  the  circular  economy  and 
digitalisation  have  effectively  contributed  to  the  expected  result  of  raising  awareness 

87 Indeed, 48% of stakeholders responded with ‘very good’  – the highest percentage across all the performance 

areas for the Agency, and the highest of the four Agencies in this category.  

88 Cedefop staff survey, n = 45. 
89  The  Osnabrück  declaration  on  vocational  education  and  training  is  an  agreed  set  of  policy  actions  in  VET 
covering  2021-2025,  signed  by  Ministers  in  charge  of  the  VET  of  the  Member  States,  the  EU  candidate 
countries and the EEA countries, the European social partners and the European Commission.  

90 Cedefop consolidated annual activity reports 2016-2021.  
91  In  terms  of  quality,  results  of  stakeholder  surveys  and  OPCs  conducted  for  this  evaluation  showed  that 
respondents found the quality of EU-OSHA’s outputs under its six activities to be of very high or high quality. 

24 

 
 
 
among  policymakers  on  new  and  emerging  risks,  therefore  helping  them  to  inform  their 
decisions.  In  particular,  EU-OSHA’s  foresight  reports  on  digitalisation  brought  added  value 
even  in  countries  that  were  familiar  with  the  issue,  by  exploring  aspects  that  were  under-
researched at national level. Management Board members and focal points also found foresight 
exercises to have been very relevant for EU and national employers’ organisations. This is a 
considerable improvement compared to the baseline when the foresight studies were used less 
than the other outputs. 

However, focal point members emphasised the need to  customise EU-OSHA outputs to 
the specific needs of target groups, particularly for workplaces, which may involve creating 
concise infographics with key information. 

ETF - Similar to the previous evaluation period, the Torino Process was highly appreciated 
by stakeholders, especially partner countries. The latter consider the Torino Process reports 
as  one  of  the  most  valuable  ETF  outputs  (see  Table  5),  as  they  are  key  to  monitor  the 
performance of lifelong learning policies and systems and to identify partner countries’ needs, 
contributing to policy dialogue. During 2019-2021 the ETF substantially exceeded its target of 
50% of partner countries  using the Torino Process and achieved a figure  of  78% of partner 
countries in 2019 (see also Section A above).  

The  expected  result  of  providing  partner  countries  with  relevant  and  timely  support  to 
modernise national qualification frameworks was also mostly achieved. The ETF consistently 
monitored  and  supported  the  development  of  national  qualification  frameworks  (NQFs), 
playing a crucial role in supporting the development of the African Continental Qualifications 
Framework. The Agency built on its positive work in this area during the previous evaluation 
period.  

The  ETF’s  work  on  facilitating  collaboration  through  the  creation  of  networks  was  often 
highlighted by stakeholders as a key area in which the Agency has achieved significant results, 
including three of the most used and valued ETF outputs during the evaluation period (see Table 
4 below), namely: (i) the ETF’s Network of Excellence, cited by both survey respondents and 
interviewees as being a high-quality output and a success in terms of the number of participants 
involved and its contribution to vocational excellence provision; (ii) the Skills Lab Network, 
set up during the current period, which brings together labour market experts and researchers 
from  different  institutions and countries to  share  knowledge on skills  anticipation  and skills 
matching (see supporting study case study on Skills Lab); and (iii) the New Learning Network, 
which contributed to innovative teaching and learning. 

Table 4: Most used and highest quality outputs / themes per Agency 

Cedefop 

ETF 

Top ranked outputs / themes 
Skills anticipation and matching within the EU 
Monitoring and analysing VET policy developments 
Promoting access to and attractiveness of VET 
Skills demand anticipation in partner countries 
Monitoring and diagnostics reports (Torino process reports, rapid education 
diagnostics reports) 

25 

 
 
 
EU-OSHA 

Eurofound 

Innovative teaching and learning (New Learning Network) 
Vocational excellence provision models (ETF Network for Excellence) 

OiRA 
Healthy workplaces campaigns 
Living, working and COVID-19 
The three pan-European surveys: European working conditions surveys, European 
quality of life Surveys, European company surveys 
Minimum wages 

Source: supporting study, based on triangulated assessment, using several sources/indicators, including outputs 
that have helped to achieve results, most downloaded outputs and most highly rated outputs in the stakeholder 
survey / public consultation.  

The ETF also achieved positive results in other areas. For example, the ETF’s enterprise 
skills development support activities were specifically highlighted in the European Parliament’s 
discharge  report  for  202092.  The  report  welcomed  the  Agency’s  ‘Skills  for  enterprise 
development’ initiative, which helps  enterprises respond to  challenges related to COVID-19 
and the digital and green transitions. In addition, the ETF’s EU external policy support met KPI 
targets (the ETF provided support each year to 58-62% of partner countries, against a target of 
50%), and the rapid education diagnostics reports (RED) enhanced dialogue and supported EU 
policy for Lebanon and Kosovo (see further evidence in Section 4.1.1.2 and Annex XII). 

However,  the  ETF  fell  short  of  meeting  its  targets  in  two  areas:  developing  and 
implementing mechanisms to facilitate the transition to work in 50% of partner countries; and 
labour  market  and  skills  intelligence  to  inform  VET  and  skills  development  policies  and  to 
support entrepreneurial learning (see Annex XII and Section 4.1.1.3. of the supporting study). 
The evaluation of ETF’s activities also suggested room for improvement on the relevance of 
the Agency’s support for capacity building among teachers and trainers, and for work-based 
learning (supporting study, p. 144). It also achieved mixed results on supporting governance 
of human capital systems in partner countries, with good results in Moldova, but insufficient 
indicators and evidence for concluding an overall assessment (see Annex XII).   

One reason for the underachievement of some objectives was the limited willingness of partner 
countries to adopt the advice and support offered by the ETF. This challenge was identified in 
the previous evaluation, highlighting the need to better tailor the ETF support to the needs and 
policy cycle of partner countries.  

4.1.1.2. GENERATING IMPACTS 

During the evaluation period, the Agencies directly contributed to EU policymaking in the 
areas  relevant  to  their  mandates,  confirming  the  positive  assessment  on  impacts 
achievement from the previous evaluation.  

The Agencies also supported the implementation of EU policies at national level (e.g. EU-
OSHA on OSH policies - see Box 4 - CEDEFOP on upskilling and recognition of qualifications, 

92 European Parliament (2022), Discharge 2020: European Training Foundation. 

26 

 
 
 
 
EUROFOUND on minimum wage and ETF on supporting the implementation of EU external 
policies in partner countries). 

The Agencies’ work has also led to specific impacts, as outlined in their respective intervention 
logics93,  along  with  unexpected  impacts,  primarily  driven  by  the  COVID-19  pandemic. 
Moreover,  the  Agencies  contributed  to  several  Sustainable  Development  Goals  (SDG) 
during the evaluation period by supporting EU policymaking in employment, social inclusion 
and  education.  In  particular,  Eurofound  contributed  to  SDG8  (decent  work  and  economic 
growth)  and  SDG10  (reduced  inequalities);  CEDEFOP  and  the  ETF  to  SDG4  (quality 
education) and SDG10 (reduced inequalities); and EU-OSHA to SDG3 (good health and well-
being) and SDG8 (decent work and economic growth). 

EUROFOUND - Similar to the previous evaluation period, Eurofound continued to make a 
direct  impact,  enhancing  policymaking  capacity  in  various  EU  policy  areas.  Notably, 
Eurofound played a significant role in areas such as minimum wage (see Box 1), and emerging 
forms  of  work.  Through  its  research  on  digitalisation  and  new,  non-standard  employment 
forms,  including  the platform  economy,  Eurofound strengthened the ability of the European 
Commission  and  other  EU  bodies  to  draw  up  effective  employment  policies.  A  concrete 
example is Eurofound’s contribution to platform work, which has underpinned the proposed 
Directive on improving working conditions in platform work94.  

Eurofound’s  activities also  contributed to  the lasting impact  of  improving dialogue between 
management  and  labour95.  Notably,  the  2022  Tripartite  Exchange  Seminar96  positively 
influenced social partners’ capacity and improved social dialogue. Eurofound’s work in this 
area has indirectly impacted areas such as the gender pay gap and the production of EU-wide 
comparative  information  on  COVID-19.  The  impact  was  substantial,  fostering  frequent 
communication  with  trade  union  stakeholders.  However,  the  case  study  found  that  these 
impacts  could  be  further  enhanced  if  Eurofound’s  resources  were  available  in  more 
languages. 

Finally, Eurofound has significantly contributed to improving the capacity to address current 
and  emerging  challenges  for  enhancing  living  and  working  conditions.  The  Agency’s 
noteworthy  contributions  include  an  increased  understanding  of  policy  topics  related  to 
COVID-19 (see  Eurofound Case Study  1 in  Annex 8). This  understanding encompasses the 
effects on remote working, platform work, the disproportionate impact of the COVID-19 crisis 
on  lower  paid  and  minimum  wage  workers,  and  the  identification  of  groups  significantly 
affected (see below on overall unexpected impacts and on adaptation to COVID-19 challenges). 
As  it  has  done  every  year  since  2015,  Eurofound  published  its  ‘Minimum  wages  in  2020: 
Annual review’, which was mentioned in 15 Commission key policy documents related to the 
initiative on fair minimum wages for workers in the EU. 

93 See Table 22 in the supporting study. 
94 See: https://ec.europa.eu/social/BlobServlet?docId=24992&langId=en.  
95 Case Study 5 in Annex 8 details Eurofound’s work in communicating knowledge and organising debate with its 

tripartite stakeholders. 

96 Organised in 2022, it helped to build social partner and national government capacity and to promote social 
dialogue through exchanges on current policies and practices. It was followed up by five Teams groups, which 
have enabled relevant parties to keep exchanging and building upon existing dialogue. 

27 

 
 
 
Box 1: Eurofound’s contribution to minimum wage policy (case study)97 

Eurofound  has  significantly  influenced  EU  employment  policies,  including  the  2022  Directive  on 
adequate minimum wages in the EU. This impact is attributed to its production of country reports on 
minimum  wage  settings  and  adequacy,  coupled  with  its  active  participation  in  meetings  and  social 
partner hearings organised by DG EMPL and other European stakeholders. Eurofound's expertise has 
directly contributed to shaping these policies.  

Moreover, Eurofound’s influence extends to the national level, where its outputs and data on minimum 
wages  are  referenced.  For  instance,  discussions  on  the  potential  introduction  of  a  national  statutory 
minimum wage in Italy have drawn upon Eurofound's information. In addition, national stakeholders, 
such as the Dutch Ministry of Social Affairs and Employment, have sought Eurofound's expertise on 
minimum wage. 

CEDEFOP  -  Cedefop  successfully  achieved  its  expected  direct  impacts  by  providing 
knowledge,  evidence  and  data,  and  facilitating  peer  exchanges.  This  support  was 
instrumental in helping both EU and national policymakers to develop or enhance policies and 
systems on VET, skills and qualifications. At EU level, Cedefop actively participated in shaping 
EU  VET  policies  by  giving  presentations  at  various  EU-level  forums  (see  Box  2).  Notably, 
Cedefop significantly contributed to enhancing transparency and recognition of qualifications 
(see Cedefop Case Study 3 in Annex 8 to the supporting study). Its work in the area contributed 
to  advancing  the  use  of  the  European  Qualification  Framework  (EQF),  by  continuously 
comparing and analysing its implementation across the EU at national and sectoral level.  

Cedefop  achieved  significant  impact  by  aligning  with  the  priorities  of  the  2019-2024 
Commission,  notably  the  2020  renewed  Skills  Agenda98  and  the  first-ever  Council 
Recommendation on vocational education and training for sustainable competitiveness, social 
fairness and resilience. This contribution was acknowledged by the European Parliament in its 
2022 discharge report99. Cedefop also provided valuable input to the Osnabrück Declaration.  

The  Agency’s  analytical  framework  on  apprenticeships  informed  the  2018  Council 
Recommendation on a European framework for quality and effective apprenticeships. Cedefop 
co-organised policy learning forums on upskilling and reskilling of low-skilled adults, together 
with  the  European  Economic  and  Social  Committee  (EESC).  The  European  Parliament 
recognised Cedefop's expertise and potential, suggesting that further reference be made to it 
when preparing guidelines for foresight-based policy analysis100. 

At national level, Cedefop made an impact by conducting country-focused thematic reviews 
and contributing to national events. For instance, in 2021, thematic reviews and peer learning 
activities were organised on skills governance in Slovenia and on upskilling pathways in France 
and Italy. Cedefop also provided support to Slovenia’s Labour Ministry in preparing a national 
skills forecast and developing a career platform101. 

97 See Eurofound Case Study 3 in Annex 8 to the supporting study. 
98  European  Commission  (2020)  European  Skills  Agenda  for  sustainable  competitiveness,  social  fairness  and 

resilience.  

99 European Parliament (2022), Discharge report for 2020. 
100 Cedefop (2022), Consolidated annual activity report for 2021, p.13. 
101 Cedefop annual report for 2021. 

28 

 
 
 
Box 2: Cedefop’s impact on VET policy at EU level (case study)102 

Cedefop has made a significant contribution to the renewed VET policy monitoring cycle. This is due 
to  the  Agency  taking  an  active  role  in  shaping  EU  policies  by  giving  presentations  at  the  Advisory 
Committee  on  Vocational  Training  (ACVT)  and  to  the  Directors-General  for  Vocational  Training 
(DGVT)  (in  cooperation  with  the  ETF).  Cedefop  also  contributed  to  the  development  and 
implementation of the new monitoring framework on VET policy.  

ETF  -  Evidence  shows  that  the  ETF’s  work  on  supporting  human  capital  development 
reforms in partner countries has had a positive impact in at least three areas listed in its 
intervention logic: (i) adapting VET governance to changing skills demands; (ii) enhancing 
labour  market  policies  and  practices;  and  (iii)  contributing  to  EU  external  relations  policy 
through advice. In particular: 

•  ETF actions effectively contributed, with concrete inputs, to the development of national 
qualifications frameworks (NQFs) / regional qualifications frameworks (RQFs) in at least 
nine countries and two regions of Africa. 

•  The ETF designed new/upgraded tools for analysing skills needs, benefiting stakeholders 

in partner countries (see Box 3). 

•  Since 2013, the ETF collaborated with Türkiye on a key priority for the country: a reform 
to set up centres for recognition of prior learning. The ETF partnered with EU delegations, 
the national qualifications agency and relevant sectoral committees. As a result, 25 centres 
for VET recognition have been set up in the country. 

Box  3:  ETF  support  on  improving  active  labour  market  policies  and  practices  (case 
study)103  

The  ETF  has  designed  new/upgraded  methodological  tools  to  analyse  skills  needs,  benefiting 
stakeholders  in  partner  countries.  The  current  approach  involves  gathering  data  using  regional 
knowledge  by  building  regional  research  networks  involving  EU  experts  (e.g.  Skills  Lab)  and  by 
conducting in-depth analysis (e.g. country reports in selected sectors). The study revealed that the in-
depth approach, focusing on specific sectoral studies in fewer countries, based on their unique needs, 
produces detailed and granular data. These data are valuable for both partner countries and the European 
Union. The impact depends on the  extent to which partner countries are involved with their national 
stakeholders, and provide inputs and insights on changing skills needs. 

EU-OSHA – Overall, EU-OSHA has achieved its expected direct impacts, contributing to EU 
OSH legislation, Member State policies, as well as initiatives by social partner and other OSH 
stakeholders. This contribution has helped these entities anticipate and manage change, prevent 
workplace accidents and increases readiness for future health crises. 

•  On contributing to OSH legislation and strategies, EU-OSHA significantly contributed to 
the EU’s strategic frameworks on health and safety at work, collaborating extensively with 
the  Commission.  The  Agency’s  work  is  cited  in  the  2014-2020  strategic  framework, 

102 See Cedefop Case Study 2 in Annex 8 for details.  
103 See ETF Case Study 2 in Annex 8 

29 

 
 
 
 
particularly its work in supporting SMEs, including through the OiRA tool104. EU-OSHA’s 
healthy workplace campaigns influenced legislation and strategies, such as the European 
Commission’s proposals on carcinogens (see Box 4 below). 

•  EU-OSHA  supported  stakeholders  in  preventing  workplace  accidents  and  illnesses,  and 
increased  readiness  for  potential  future  health  crises.  For  instance,  the  OiRA  tool  for 
COVID-19, introduced in June 2020, helped with COVID-19 risk management. The tool 
was actively promoted in Belgium105 and France106 among other Member States. In France 
it provided 3 797 risk assessments in just over 2 months. It is, however, difficult to assess 
the actual use of tools such as OiRA in companies, and its longer-term impacts (e.g. whether 
companies acted on the information from OiRA and applied changes to prevent workplace 
accidents and illnesses), as the use of the tools is anonymous107. 
In  terms  of  enabling  anticipation  and  management  of  change,  EU-OSHA  successfully 
provided foresight information on OSH. Most respondents across the three consultations 
(staff, stakeholders and the broader public) responded that the Agency was successful in 
delivering foresight information either to a large extent or to a moderate extent during the 
evaluation period108. 

• 

Box 4: EU-OSHA’s impact on contributing to preventing workplace accidents (case 
study)109 

Two of EU-OSHA’s healthy workplaces campaigns (HWCs) demonstrate the Agency’s contribution to 
policy on preventing workplace accidents particularly well: ‘2018-2019: Healthy workplaces manage 
dangerous substances’ and ‘2020-2022: Healthy  workplaces lighten the  load’. These HWCs allowed 
EU-OSHA to engage with stakeholders at national level, from SMEs to larger corporations and national 
bodies. They raised awareness about dangerous substances in the workplace, their associated risks, and 
methods  to  prevent  work-related  musculoskeletal  disorders  (MSDs).  The  ‘dangerous  substances’ 
campaign coincided with the revision of EU directives on dangerous substances and carcinogens and 
mutagens and helped their transposition and implementation at national level. The campaigns’ impact 
has had a lasting effect, as some Member States, like Finland and Cyprus, have chosen to continue to 
address the topics even after the campaigns ended 110.   

4.1.1.3. UNEXPECTED IMPACTS 

The COVID-19 pandemic had a significant impact on the Agencies’ outputs and activities, with 
the  specific  impact  depending  on  the  mandate  and  way  of  working  of  each  Agency.  The 
COVID-19 pandemic had a significant impact on the Agencies’ outputs and activities – impacts 

104 European Commission (2014), Communication on an EU Strategic Framework on Health and Safety at Work 

2014-2020, Brussels 06/06/2014, COM(2014) 332 final, p. 14. 

105 See: https://www.beswic.be/nl/in-de-praktijk/nieuwe-oira-tool-helpt-bij-covid-19-risicobeheer  
106 In France, the carrying out of risk assessments was associated with access to a grant scheme for micro and small 

enterprises (MSEs).  

107 Ipsos (2020), ‘Mid-Term evaluation of the OiRA activity - Final Report’, p.39. 
108 96% of staff survey respondents (n=48), 95% of stakeholder survey respondents (n=75), and 93% of public 
consultation respondents (n=41)). It is nevertheless worth noting that a sizeable minority of stakeholders found 
that the Agency was successful in delivering foresight information on OSH to a ‘moderate extent’. 

109 See EU-OSHA Case Study 4 in Annex 8 to the supporting study. 
110 Ipsos (2020), ‘Evaluation of the Healthy Workplaces Campaign 2018-2019’, p.41. 

30 

 
 
 
 
differentiated  according  to  their  mandates–,  and  their  way  of  working.  For  all  agencies, 
COVID-19 was  a highly disruptive factor,  e.g.  cancellation of some  events  and activities in 
2020 involving physical presence. Eurofound’s survey fieldwork was profoundly affected, as it 
could not be carried out face-to-face, and the EU OSHA ESENER outputs, as no dissemination 
events could be held. The number of events organised by Cedefop reduced by around half in 
2020 and 2021. The pandemic also affected the ETF’s Partner Countries ability to engage in 
some activities that were planned in 2020 with some of them having to be postponed to the 
following year.  

However, these changes also led to several positive outcomes and unexpected effects. The 
transition to  online  work not  only  resulted in  cost-savings but also  allowed the Agencies to 
increase  their  visibility  by  providing  information  and  guidance  on  COVID-related  issues, 
expanding their audience.  

For example, Eurofound’s work on remote and platform work fed into policy discussions on 
COVID-19, particularly its findings on youth and gender111. This unexpected event led to higher 
engagement,  measured  through  policy  references,  media  coverage,  academic  citations  and 
downloads, showing the increased interest in their expertise on gender and youth112. 

Other  broader  achievements  and  unexpected  impacts  included  EU-OSHA’s  publications 
reaching a wider audience, positive impacts from the ETF’s work related to Russia’s war of 
aggression against Ukraine, and heightened visibility for Cedefop’s work due to its activities 
related to Ukraine (see below). 

Agencies’ adaptation to changes, COVID-19 pandemic and Russia’s war of aggression 

During the 2017-2022 evaluation period, the EU was faced with new challenges, alongside its 
ongoing  long-term  issues113.  Overall,  the  Agencies  adjusted  their  activities  and  outputs 
responding to: (a) changes in EU policies, including the political priorities of the 2019-2024 
Commission,  focusing  more  on  green  and  digital  transitions  policies  (see  Tables  4  and  5  in 
Annex VII); and (b) the impacts of the COVID-19 pandemic and Russia’s war of aggression 
against Ukraine.  

A) Regarding the adjustment to EU policy priorities, such as digitalisation and the European 
Green  Deal,  the  Agencies  adapted  their  activities,  within  their  mandates,  and  worked  on 
addressing  these  megatrends  during  the  evaluation  period.  This  involved  realigning  work 
programmes and creating specific outputs on these topics such as publications, studies, surveys 
and  collaborative  projects  (see  Table  6  in  Annex  VII)114.  The  Agencies  also  revised  their 
environmental  policies  to  align  with  EU  priorities  around  climate  change.  The  evaluation 
identified  a  need  to  improve  dissemination  of  outputs  related  to  these  topics,  to  ensure  that 
stakeholders are well-informed of the Agencies’ efforts in these areas. 

111 As Eurofound highlighted through its work that women and young people were disproportionately affected by 

the pandemic.  

112 Eurofound (2020), Consolidated annual activity report for 2020, p. 7. 
113 From 2011 to 2016, the previous evaluation identified major socio-economic challenges (post-crisis recovery, 
immigration crisis), and long-term developments (ageing workforce, changing working patterns, new forms 
of employment) to which the Agencies needed to adapt to generate evidence for policymaking.  

114 See also the detailed analysis in Section 4.1.1 of the supporting study.  

31 

 
 
 
On  the  European  Green  Deal  in  particular,  Eurofound  published  a  range  of  reports  on 
sustainability  and  related  social  challenges115  in  addition  to  a  pilot  project  proposed  by  the 
European Parliament on the Future of Manufacturing in Europe. Eurofound and EU-OSHA 
are active partners in the EU Climate and Health Observatory. This collaboration enhances their 
ability  to  address  the  intersecting  challenges  of  climate  change  and  health,  providing 
comprehensive data, analysis, and tools to stakeholders across Europe. Their work within the 
Observatory focuses on assessing and mitigating the impacts of climate change on working and 
living conditions, thereby contributing to more resilient and adaptive occupational and public 
health policies, which monitors and evaluates the impacts of climate change on public health.  
Cedefop produced outputs analysing the impact of the green transition on jobs and skills across 
sectors and occupations, united under the Cedefop Green Observatory initiative. Publications116 
included  the  sectoral skills foresight, which focused on occupational profiles for the green 
transition of smart and green cities, waste management, agri-food and circular economy, and 
the green employment and skills transformation report (2021), which assessed the effects of the 
European  Green  Deal  on  different  sectors  and  identified  opportunities  for  upskilling  and 
reskilling.  

The ETF carried out a significant amount of work on green skills, including: (i) pilot reviews 
in  eight  countries117  in  sectors  such  as  agri-tech,  agri-food,  automotive  and  energy;  (ii)  the 
GRETA initiative118 to support the green transition through peer learning; and (iii) the annual 
ETF  Green  Skills  Award  that  recognises  successful  efforts  in  creating  circular  and  carbon-
neutral economies (see ETF Case Study 4).  Finally, EU-OSHA produced a growing number 
of thematic publications on green jobs, including foresight studies aimed at anticipating OSH 
challenges relating to green jobs and the circular economy, and the ‘heat at work’ guidance for 
workplaces.  

B) The COVID-19 pandemic significantly influenced the latter part of the evaluation period. 
All Agencies swiftly adapted to the challenges posed by COVID-19, both in terms of internal 
processes and thematic focus. They produced knowledge and dissemination tools to address 
COVID-19  challenges  in  line  with  their  mandates.  For  instance,  EU-OSHA  and  Eurofound 
responded promply to provide advice on occupational health and safety, and support for living 
and working conditions, respectively. Cedefop and the ETF also quickly adapted their activities 
to assist stakeholders in facing the challenges brought about by the pandemic. 

EUROFOUND adjusted its work on ‘living and working conditions’, by introducing two new 
tools  for  monitoring  and  data  collection:  the  COVID-19  EU  PolicyWatch  database  and  the 

115 For example: Eurofound and the European Environment Agency (2021),  Exploring the social challenges of 
low-carbon energy policies in Europe, Publications Office of the European Union, Luxembourg; Eurofound 
(2022), EU-ANSA mapping report: Socioeconomic aspects of sustainable development, Publications Office 
of the European Union, Luxembourg. 

116  A total of 11 publications were produced during the evaluation period, Cedefop (2022), Consolidated annual 

activity report for 2021, p.10. 

117 Algeria, Azerbaijan, Georgia, Morocco, North Macedonia, Serbia, Ukraine and Uzbekistan. 
118  GRETA  (greening  responses  to  excellence  through  thematic  actions)  is  an  ETF  initiative  supporting  the 
greening of vocational education and training (VET) as a response to the green and digital transitions. 
32 

 
 
 
‘living, working and COVID-19’ e-survey119. The e-survey was particularly helpful in assessing 
the impact of the pandemic on living and working conditions.  

The  EU  PolicyWatch  tool  provided  valuable  information  to  the  European  Commission  and 
European Parliament by showcasing initiatives responding to crises. This resulted in Eurofound 
being one of the first organisations to provide insights into the pandemic’s effects120. Since then, 
Eurofound has produced 145 publications related to COVID-19121, which were highly effective 
and widely downloaded122.  

Table 4: Agencies’ responses to COVID-19 

Agency 

Main responses 

Eurofound 

•  Produced two key data collection tools in 2020 to monitor the impact of the pandemic. It 

continued to update these tools in 2021 and 2022: 

o 
o 

the COVID-19 EU PolicyWatch database; and  

the Living, working and COVID-19 e-survey.  

•  Produced a joint European company survey with Cedefop to explore the impact of the 

pandemic in companies in the EU-27. 

•  There were 25 216 downloads of publications on the subject of COVID-19 in 2021.  

Cedefop 

• 

Information flyer ‘Rising to the Coronavirus challenge’, which informed stakeholders 

about COVID-19's impact on the labour market, skills and VET. 

•  Cooperation with Eurofound to design a survey to gather information on how 

organisations coped with the consequences of the pandemic, as a follow-up to the 

European company survey. 

•  Led and coordinated an international survey on career guidance policy and practice 

during the pandemic, which resulted from the collaboration of an inter-agency working 

group with representatives of UNESCO, ILO, OECD, the European Commission and the 

ETF. 

EU-OSHA 

•  Development of a COVID-19 OiRA tool. 
•  Development of a webpage, ‘COVID-19: Resources for the workplace’, which 

contained 20 resources developed in 2021123, including: 

two guides on long COVID for managers and workers, 

o 
o  OSH-Wiki articles on COVID-19, 
o 

awareness-raising tools such as videos and factsheets. 

ETF 

•  Set up a system for monitoring the impact of the pandemic on partner countries’ 

education and training systems124. 

•  Adaptation of the #LearningConnects campaign to offer further support during the 

pandemic through: 

119 It entered its fifth round in 2022 and is now also assessing the effects of the war in Ukraine, again demonstrating 

the Agency’s ability to refocus its attention on areas where research is most needed. 

120 Eurofound (2020), Consolidated annual activity report for 2020, p. 1, p. 4. 
121 Compared to 122 publications on digitalisation, on which Eurofound has been working since 2012. 
122 25 216 downloads of publications in 2021. The next highest number of downloads was 13 073 on the topic of 

working conditions and sustainable work.  

123 European Commission (2021), EU strategic framework on health and safety at work 2021-2027. 
124 ETF (2020), Single Programming Document 2021 – 2023: Work Programme 2021, GB/20/DEC/011, p. 33. 

33 

 
 
 
o  organisation of webinars and activities,  
o  publication of news articles and reports on good practices identified across 

partner countries125. 

•  Adapted its activities in partner countries following the pandemic to take a flexible 

approach to address their more immediate needs126.  

The  evaluation  study  and  Case  Study  1  confirmed  that  Eurofound  responded  rapidly  and 
effectively to the challenge posed by COVID-19. Stakeholder interviews and staff surveys127 
indicated that Eurofound’s response to the crisis was effective, timely and impactful. Eurofound 
has  largely  aligned  its  research  areas  and  key  outputs  with  the  European  Commission’s  six 
strategic priorities for 2019-2024128, making significant contributions. 

EU-OSHA - As detailed in EU-OSHA Case Study 1129, the pandemic provided an opportunity 
for the Agency to significantly contribute to an increased focus on OSH issues, both in terms 
of practical needs and in policy debates. The Agency responded promptly by offering various 
tools to manage the pandemic in  European workplaces130. These tools included guidance for 
workplaces,  a  dedicated  webpage  (‘COVID-19:  Resources  for  the  workplace’),  and  20 
resources  developed  in  2021131,  such  as  EU  guidance  documents,  OSH-Wiki  articles,  and 
awareness-raising materials132.  In  addition,  the  Agency developed  the OiRA  risk assessment 
tool for COVID-19 and made it available to national partners in June 2020133.  

The ETF carried out several activities to address the urgent needs arising from the COVID-19 
pandemic. Initially, it set up a monitoring system to track the pandemic’s impact on education 
and training systems in partner countries, along with measures to mitigate negative effects134. 
The findings were shared through reports for partner countries and Commission departments135 
as  well  as  public  webinars.  The  #LearningConnects  campaign  provided  further  support  by 
organising webinars and other activities, and publishing articles and reports on good practice in 
partner countries. The ETF stepped up its collaboration with other Agencies and international 
organisations, focusing on skills, youth and career guidance136. 

CEDEFOP collaborated with Eurofound on a survey to gather information from organisations 
on how they were  coping with  the pandemic and organised a seminar with Members of the 
European Parliament on post-pandemic employment, skills and social policies. Cedefop also 

125 For example, see: ETF (2020), LearningConnects  – What can Turin teach us?: Highlights from the  City of 

Turin on dealing with learning under lockdown due to the COVID-19 crisis, [online]. 

126 ETF (2021), Consolidated annual activity report for 2020, GB/21/DEC/003, p. 24. 
127  72%  of  stakeholder  survey  respondents  (N=65)  and  92%  (N=51)  of  staff  survey  respondents  viewed 

Eurofound’s response to the COVID-19 crisis as effective ‘to a large extent’. 

128 https://commission.europa.eu/strategy-and-policy/priorities-2019-2024_en 
129 See Annex 8 to the supporting study. 
130 The Agency’s AAR in 2020 reported 30 COVID-19-related actions, delivered mainly in 2020 and 2021. 
131 European Commission (2021), EU strategic framework on health and safety at work 2021-2027 Occupational 

safety and health in a changing world of work.  

132 EU-OSHA (2021),   Annual activity report for 2020, pp.18-19: https://osha.europa.eu/en/publications/annual-

activity-report-2020-0  

133 Ipsos (2020), Mid-Term evaluation of the OiRA activity - final report, p.37. 
134 ETF (2020), Single programming document 2021-2023: Work programme 2021, GB/20/DEC/011, p. 33. 
135 ETF (2021), Consolidated annual activity report for 2020, GB/21/DEC/003, p. 6-7. 
136 Such as UNICEF, UNIDO, EBRD, CEDEFOP and ILO.  

34 

 
 
 
led  an  international  survey  on  career  guidance  policy  and  practice  during  the  pandemic,  in 
collaboration with an inter-Agency working group including representatives of UNESCO, ILO, 
OECD,  the  European  Commission  and  the  ETF. It  also  produced  products  and  resources 
informing stakeholders about COVID-19's impact on the labour market, skills and VET, and 
specifically, the tourism sector137. A user satisfaction survey run by Cedefop in 2022 revealed 
that approximatively 90% of users were highly satisfied with the Agency's COVID-19 outputs, 
finding them clear, sound, attractive, and very useful for understanding the pandemic's impact 
on VET, skills and qualifications138.  

Joint outputs - In addition to their individual actions, the Agencies cooperated on joint outputs 
and activities to address the cross-cutting challenge of COVID-19139. These included: (i) a joint 
European  company  survey  run  by  Eurofound  with  Cedefop,  exploring    the  impact  of  the 
pandemic on companies in the EU-27; (ii) an e-survey by Eurofound on ‘Living, working and 
COVID-19’,  distributed  by  the  ETF  to  its  partner  countries;  and  (iii)  collaboration  between 
Eurofound  and  EU-OSHA,  such  as  on  organising  discussions  at  the  European  Parliament’s 
EMPL  Committee  on  the  COVID-19  impacts,  and  the  participation  of  Eurofound  in  EU-
OSHA’s psychosocial risk assessment. 

Russia’s war of aggression against Ukraine also meant that the Agencies needed to respond 
and adapt their activities to varying degrees, depending on their mandates, over the course of 
2022. The situation in Ukraine was most relevant for the ETF; it was of intermediate relevance 
for  Eurofound  and  Cedefop  and  of  least  relevance  for  EU-OSHA.  This  is  reflected  in  the 
varying intensity of the Agencies’ reactions, and in the responses to the staff and stakeholder 
surveys.  

ETF - For the ETF, Ukraine has long been a partner country, allowing the Agency to provide 
tailored support. The ETF’s range of activities in this area is therefore much greater than that 
of  the  other  three  Agencies.  Stakholders  largely  perceived  the  ETF  as  responsive140.  The 
Agency’s initiative in monitoring and responding to the Ukrainian crisis was also highlighted 
in the 2021 European Parliament discharge report on the ETF’s activities141.  

The  ETF’s  initiatives  included  work  on  the  recognition  of  qualifications  to  help  Ukrainian 
refugees  enter the workforce, through the report  ‘Comparing Qualifications Frameworks for 
Inclusion:  a  Ukrainian  case  study’142.  In  addition,  in  collaboration  with  Eurofound,  the  ETF 
published a report in 2022 that provides an assessment of the impact of Russia’s invasion on 
the economy and education143. In addition to other outputs and resources144, the ETF carried out 
two key initiatives to support the country in setting up a new education paradigm: the UA Re-

137 Cedefop (2021), Consolidated annual activity report for 2020, p.16. 
138 PPMI (2022), User satisfaction survey 2022, p.8. 
139 See cross-cutting Case Study 2. 
140 On the split between Management Board (MB) and non-MB members, 80% (N=5) of MB members judged that 
the ETF was responsive to a large or moderate extent, compared with 39% of non-MB (N= 80) members. 
141 European Parliament (2023), draft report on discharge in respect of the implementation of the budget of the 

European Training Foundation for the financial year 2021, p. 8. 

142 ETF (2023), Comparing qualifications frameworks for inclusion: A Ukrainian case study.  
143 ETF and Eurofound (2022), Working life in Ukraine. 
144 For example, see: ETF (2022), Education in a time of war. 

35 

 
 
 
Emerge(ncy) programme, focused on vocational e-learning, reskilling and upskilling courses145, 
and the Ukrainian Resource  Hub: to  provide  wide education and  work information  for  both 
Ukraine  and  EU  countries  hosting  Ukrainian  refugees,  including  schools,  universities  and 
employers146. 

CEDEFOP  –  For  Cedefop,  the  Ukrainian  crisis  has  been  of  more  moderate  relevance.  The 
Agency  focused  on  supporting  displaced  Ukrainian  students  in  VET  institutions  of  the  host 
countries, providing guidance to teachers and trainers involved in their reception and learning 
pathways147  (e.g.  Using  ReferNet  2022  national  outputs,  Cedefop  analysed  Member  States' 
national policies to support the integration of Ukrainian refugees into VET)148.  

EUROFOUND, for which the Ukrainian crisis was less relevant than the COVID-19 pandemic, 
had  fewer  research  outputs  on  Ukraine.  Survey  responses  indicated  a  lower  perception  of 
Eurofound's  effectiveness  in  responding  to  the  Ukraine  crisis  compared  to  COVID-19  (see 
Annex 13 to the supporting study).  

4.1.2. EFFICIENCY 

The  available  evidence  indicates  that  the  four  Agencies  were  cost-effective  during  the 
evaluation period. The four Agencies have fully implemented their budgets and delivered to a 
large extent the planned outputs set out in their annual programming documents, indicating high 
operational  efficiency  (see  Sections  3.1.1  and  3.2).  They  managed  to  lower  costs  through 
various measures while maintaining the quality of their outputs. The cost-effectiveness analysis 
of the Agencies’ key activities concluded that, for most of them, there was no scope to reduce 
costs without compromising quality (see Annex XI)149. 

There are differences in the annual expenditure among the four Agencies as well as between 
the Agencies and those decentralised agencies used as benchmarks. This includes differences 
in the share of expenditure allocated to administrative and operational activities. However, these 
differences  are  based  on  reasonable  factors  and  do  not  indicate  any  specific  or  obvious 
inefficiencies.  

The Agencies have taken action to improve their cost-effectiveness over the evaluation period. 
Key examples include: 

- 

- 
- 

cost-saving  measures,  e.g.  joint  procurement  with  other  agencies  (the  agencies  share 
information on upcoming procurements through the EU Agencies Network (EUAN));  
sharing services (e.g. the ETF and EU-OSHA now share accounting services); 
sharing costs related to research activities (e.g. Eurofound’s preparation of the fourth 
European company survey was completed in collaboration with Cedefop); 

145 ETF (2022), UA Re-Emerge(ncy): e-learning and skills development to rebuild Ukraine. 
146 ETF(2022), Education and work information for Ukrainians and EU countries.  
147 Cedefop (2022), Skills comparison between Ukraine and the EU-27.  
148 Cedefop (2022), Making VET inclusive for Ukrainian refugee students.  
149 See also supporting study in Annex 6.  

36 

 
 
 
 
- 

investing  in  actions  that,  while  increasing  short-term  costs,  should  ensure  long-term 
savings (e.g. increasing the energy efficiency of offices, disseminating outputs in digital 
formats rather than physical formats). 

These examples are described in detail below. See also Table 3 in Annex VII. 

The evaluation also examined how the Agencies responded to audit reports. It found that the 
Agencies were highly responsive in putting audit recommendations (both by the Internal Audit 
Service  (IAS)  and  the  European  Court  of  Auditors  (ECA))  into  practice  and  transparently 
publishing follow-up to audit recommendations in consolidated annual activity reports of audit 
recommendation implementation. 

Stakeholders  perceived  the  four  Agencies’  operations  to  be  cost-effective  although  the 
evaluation  stakeholder  survey  showed  differences  between  the  main  stakeholder  groups. 
Members of the Agencies' management or governing boards (MBs) were far more positive than 
other  stakeholders.  Across  all  four  Agencies,  83%  or  more  of  MB  members  rated  the  cost-
effectiveness of the Agencies’ operations as either high or medium, compared to 61% of other 
Eurofound stakeholders, 57% of other EU-OSHA stakeholders, 49% of other ETF stakeholders 
and 46% of other Cedefop stakeholders. The percentage of survey respondents rating the cost-
effectiveness  as  low was very  low across all four Agencies, at  6% or less (see Figure  12 in 
Annex VII).  

However, the interviews, survey responses, consultations and analysis of expenditure data 
identified  some  key  areas  where  the  Agencies’  cost-effectiveness  could  be  further 
improved, in particular staff resources, inter-agency cooperation, procurement and contracting 
procedures (see below).  

The evaluation analysed the efficiency of the Agencies in terms of: (i) balance of operational 
and  administrative  expenditure  (and  staff);  (ii)  staff  resources  and  workload;  (iii)  cost-
effectiveness of key activities; (iv) internal monitoring and reporting mechanisms and potential 
for simplification and  administrative burden reduction; and (v)  the efficiency of governance 
structures. The following sections provide an overview of the main evidence by area. 

4.1.2.1 BALANCE OF OPERATIONAL AND ADMINISTRATIVE EXPENDITURE 

A key factor contributing to cost-effectiveness is the balance between the Agencies’ ‘front-line’ 
operational activities and their staff and administrative costs. Although administrative roles are 
essential to an agency’s performance, these should be streamlined to maximise the proportion 
of  expenditure  on  operational  activities.  To  assess  the  balance  of  administrative,  staff  and 
operational  expenditure150,  the  four  Agencies  were  benchmarked  against  other  decentralised 
agencies working in related policy areas: the European Institute for Gender Equality (EIGE), 
the Fundamental Rights Agency (FRA) and the European Labour Authority (ELA)151.  

150 Following the Agencies’ expenditure classification (Title 1: Staff; Title 2 Administrative; Title 3 Operational). 
151 This is despite the limits of such a comparison, given the different remits, objectives and activities of each 
agency. The ELA was chosen as it is the fifth agency working under the remit of DG EMPL. The ELA has 
only been running since 2020, so comparisons are only possible for 2020-2022 and need to be treated with 
additional caution. 

37 

 
 
 
Compared to the previous evaluation period, the four Agencies reported increases in staff 
costs (and their share in total budget) between 2016 and 2022 (see Figure 2). Rather than being 
driven by increases in staff numbers (see Section 3.1 and Annex VII), these increases were due 
to  high  inflation  rates  and  corresponding  wage  adjustments  through  the  correction  of  the 
country coefficients152 (e.g. Eurofound, headquartered in  Dublin,  saw its  country coefficient 
rise from 118.3 in 2016 to 136.9 in 2022). Among the four Agencies, the ETF had the smallest 
increase in the share of staff costs to total expenditure (4 pps.) and EU-OSHA had the largest 
(9 pps.), although it began with the lowest share in 2016. 

The increase in staff costs, coupled with relatively stable budgets (which are both factors out of 
the Agencies’ control), led to decreases in the share of expenditure devoted to operational 
activities when comparing 2022 to 2016, the baseline year. This suggests that higher staff costs, 
mostly driven by the country correction coefficients, limited the Agencies’ ability to allocate 
financial  resources  to  the  operational  activities  that  are  necessary  to  fulfil  their  mandates. 
Eurofound saw the largest decrease in the share of operational expenditure to the total budget 
(9 pps.), while the ETF saw the lowest (3 pps.). 

The shares of expenditure devoted to administrative costs saw smaller fluctuations. When 
benchmarked with similar agencies, the four Agencies showed generally similar or lower shares 
of administrative expenditure, which was already the case in the previous evaluation period. 
Three  of  the  four  Agencies  experienced  slight  increases  from  2016  to  2022:  Eurofound 
(following  a  large  decrease  in  the  previous  period),  EU-OSHA153  (remaining  substantially 
below  the  2013  value  when  the  share  of  administrative  costs  peaked  at  12%)  and  Cedefop 
(mainly  due  to  increased  investment  in  its  building’s  digital  infrastructure  and  energy 
efficiency). The ETF’s administrative expenditure share decreased, continuing the downward 
trend since 2013154. 

Although  the  share  of  administrative  expenditure  has  only  slightly  increased  in  the  current 
period, it is expected to grow in the coming years as the Agencies invest in energy efficiency 
to achieve the net-zero target. This will put even more pressure on the Agencies’ operational 
budgets  (if  total  budgets  remain  stable)  but  does  not  relate  to  any  specific  or  obvious 
inefficiencies on their part.  

152 Salaries in the four Agencies are adjusted annually in line with inflation and purchasing power in EU 

countries by applying a coefficient based on the cost of living in the host country, compared with Brussels.  
153 EU-OSHA explained the rise in its administrative costs in 2022 resulted from expenditure on office supplies 
and maintenance to support the hybrid way of working – see EU-OSHA (2022). Consolidated Annual 
Activity Report (2022).  

154 When the share of administrative expenditure peaked at 9%. 

38 

 
 
 
Figure 2: Type of expenditure (% of total expenditure 2016 and 2022) 

Source: Consolidated Annual Activity Reports, 2016 & 2022. Data not available for ELA in 2016. 

4.1.2.2 STAFF RESOURCES AND WORKLOAD 

Skilled staff are  essential  for the Agencies to carry out  their work plans to a high  standard. 
However,  the  four  Agencies  have  faced  staffing  challenges  to  varying  degrees155.  These 
challenges  are:  (i)  cuts  to  and  subsequent  caps  on  staff  numbers  (see  Section  3.1);  (ii)  an 
unexpected  increase  in  workload  due  to  the  response  to  COVID-19  and  Russia’s  war  of 
aggression  against Ukraine without a corresponding increase in staff; and (iii) the  Agencies 
expanding  certain  areas  of  their  work.  These  difficulties  have  undermined  the  Agencies' 
ability to keep achieving their targets and maintaining a manageable workload for their 
staff.  

According to the evaluation’s staff surveys, most staff in the four Agencies considered their 
workload to be well balanced or mostly balanced. This ranged from 90% in EU-OSHA to 56% 
in Eurofound. However, high percentages of staff reported that their workload was either mostly 
unbalanced  or  unbalanced,  especially  in  Eurofound  (44%),  followed  by  the  ETF  (29%), 
Cedefop (27%), and EU-OSHA (10%).  

The  supporting  study  found  that  all  four  Agencies  have  coped  with  the  demands  by 
increasing  overtime  and  the  use  of  contract  staff,  which  is  unlikely  to  be  sustainable156. 
Interviews complementing staff surveys corroborated this conclusion157. In particular, Cedefop 
and  EU-OSHA  have  relied  on  external  contractors  to  support  both  administrative  and 

155 See detailed assessment in the supporting study’s main report and in the agency-specific reports. 
156 These challenges are also likely to continue in the future and may worsen as long-serving staff retire, creating 

vacant positions and a potential loss of knowledge. 

157 For example, interviews with Eurofound staff revealed that, as an organisation, it had reached the upper limit 

of its capacity. 

39 

 
 
 
operational  tasks158.  Although  this  has  reduced  pressure  on  statutory  staff  (especially  for 
carrying  out administrative tasks), it  comes  with significant  budget  implications.  EU-OSHA 
staff  and  executive  board  members  pointed  to  concerns  about  the  quality  of  contractors, 
difficulties in finding contractors specialised in OSH, and a time-consuming process to review 
deliverables.  Based  on  these  elements,  bringing  tasks  in-house  by  hiring  more  staff  would 
enable the agency to have more control over work processes and the quality of outputs. For the 
ETF,  the  Court  of  Auditors  recently  highlighted  a  need  to  avoid  overreliance  on  external 
consultants  and  maintain  expertise  in-house159.  Eurofound  had  also  relied  on  external 
contractors although, over the evaluation period, it increased the amount of work done in-house, 
one of the drivers being to have more control over research quality. On the gender balance of 
ETF staff, the European Parliament discharge reports between 2018 and 2021 noted that further 
efforts were needed to achieve a better staff gender balance (67.7% women and 32.3% men in 
2020). 

Regarding  their  headquarters,  Eurofound,  the  ETF  and  EU-OSHA  staff  reported  high 
satisfaction about the fulfilment of their respective host Member States’ obligations. However, 
Cedefop staff had mixed views (45% satisfaction), primarily due to delays in signing a new 
headquarters agreement. Moreover, EU-OSHA, the ETF and Cedefop staff had mixed views 
on schooling (around  a  50% satisfaction rate), while Eurofound,  the ETF and Cedefop staff 
identified transport connections as an area for improvement (see Figure 6 in Annex VII). 

4.1.2.3  EFFICIENCY  AND  ADEQUACY  OF  INTERNAL  MONITORING  AND  REPORTING 
MECHANISMS AND THE POTENTIAL FOR REDUCING ADMINISTRATIVE BURDENS 

Overall,  the  Agencies  have  complied  well  with  the  measures  necessary  for  ensuring 
accountability and assessing performance.  The existing mechanisms cover different areas of 
the  Agencies’  operations  (including  financial,  human  resources  and  core  business 
performance), allowing for an assessment of how resources are spent and how they convert into 
the planned output.  

In  the  current  period,  the  four  Agencies  collaboratively  implemented  an  internal  control 
framework between 2018 and 2019. The framework allows the Agencies to assess how they 
mitigate problems and maximise their effectiveness in five control areas156. It also enables the 
four  Agencies  to  better  align  their  reporting  and  evaluation  output.  The  Agencies  made 
additional efforts to streamline their indicator systems160 and reduce the administrative burden 
created by the existing mechanisms. 

However, the current evaluation has identified two main areas for improvement. 

(i) While the monitoring and reporting systems are extensive, a number of limitations were 
identified.  In  particular,  some  monitoring  data  were  not  fully  complete  or  missing 
(particularly for the ETF). Moreover, for some performance indicators there is an absence of 
target  indicator  values  (across  all  four  Agencies).  Another  major  gap  is  the  absence  of 
indicators  for  monitoring  agencies  results  at  national  level.  Some  agency-specific  issues 

158 Deloitte (2018), Ex-post evaluations of Cedefop activities 2016-2017. 
159 ECA (2020), Special Report: Future of EU Agencies – Potential for more flexibility and cooperation:  
160 Eurofound revised its Performance Monitoring System (EPMS) in 2017, making it KPI-based and easier to 

monitor. 

40 

 
 
 
were identified:  for  Eurofound, although the indicators had no problems, reporting could be 
further improved by using more user-friendly software and by streamlining project-reporting 
procedures; Cedefop, unlike other Agencies, does not have a separate indicator for monitoring 
the work programme completion rate; for the  ETF, some indicators  are  not  available across 
partner countries, and some performance indicators are missing data for certain years (e.g. the 
number of downloads from the ETF’s website).  

(ii) There is potential to reduce the administrative burden associated with accountability 
mechanisms.  Internal  mechanisms  are  generally  sufficient  for  ensuring  accountability  and 
assessing performance (and the Agencies’ staff and Parliament discharge reports corroborated 
this  overall  positive  assessment).  However,  these  mechanisms  do  require  extensive 
administrative resources.  Most  staff survey respondents  still experience some administrative 
burden161, which is perceived as higher by Cedefop and EU-OSHA staff (see Figure 3 in Annex 
VII). These results are in line with the previous evaluation, indicating only slight shifts162. In 
particular, Cedefop staff see accountability mechanisms as very burdensome, viewing them as 
a  source  of  additional  workload,  despite  simplification  measures  piloted  by  the  agency’s 
administration163.  As  for  EU-OSHA,  which  is  the  smallest  of  the  four  Agencies  and  has  the 
highest  administrative  costs  per  staff  member164,  its  staff  and  focal  points  members  (FOPs) 
spoke  of  the  increasing  administrative  burden  associated  with  reporting  and  monitoring 
mechanisms165 as multiple online tools are used. The associated workload was disproportionate 
to the size of the agency.  

Moreover, interviews with staff of all Agencies revealed that administrative staff perceived 
their workload as less balanced than the workload of operational staff. This suggests that 
the  efforts  needed  to  meet  the  reporting,  monitoring  and  evaluation  requirements  may  be 
disproportionate to the relative sizes of the Agencies and are hindering effectiveness.  

4.1.2.4 AGENCIES’ ACTIONS TO IMPROVE COST-EFFECTIVENESS AND POTENTIAL AREAS FOR 
COST REDUCTION 

During the evaluation period, each agency looked to adapt to the budget and human resources 
challenges  and  decrease  the  administrative  burden  with  the  aim  to  achieve  efficiency  gains. 
These have been mostly achieved through: a) cooperation and sharing costs with partners; b) 
digitisation/automation; and c) streamlining of work processes.  

First,  the  Agencies  looked  for  ways  to  share  services,  including  recruitment  (e.g.  sharing 
reserve lists) and joint public procurement contracts (see agency-specific details in Table 3 in 
Annex VII). These efforts were facilitated by coordination through the EU Agencies Network. 
Moreover,  the  digitalisation  of  processes,  which  was  strengthened  during  the  COVID-19 

161 Question: Do you think that administrative tasks related to programming, monitoring, reporting, and evaluation 

hindered the implementation of your primary tasks in the period 2017-2022? 

162 There has been a minor change, with a decrease in perceived burdens at Eurofound (from medium to small) 

and an increase at EU-OSHA (from small to medium).  

163 Such as preparing a shorter annual report and avoiding duplication with CAAR. 
164  This  reflects  that  smaller  agencies  must  deal  with  relatively  higher  administrative  workloads  since  the 
administrative requirements for programming,  monitoring, reporting and evaluation are the same for all of 
them. 

165 Staff survey, question: Please, indicate, in your opinion, how distribution of tasks could be improved, indicating 

if there are tasks that have become redundant or tasks that are duplicated, and what these tasks are.  

41 

 
 
 
pandemic,  enabled  some  savings.  As  regards  work  processes,  the  four  Agencies  have 
implemented an activity based management (ABM) approach, where budgeting is  organised 
under key activity headings, ensuring that sufficient budget is allocated to different activities. 

The Agencies have made efficiency gains and implemented simplification measures. However, 
there  is  room  for  further  action  to  improve  cost  effectiveness  and  reduce  the  administrative 
burden in the following areas.  

1)  Strengthening  inter-agency  cooperation:  although  there  is  already  evidence  of 
cooperation  between  the  Agencies,  there  is  room  to  deepen  it  further  and  create 
synergies  (e.g.  more  information  sharing  and  joint  events  to  improve  external 
stakeholders’ access to information across the four Agencies; identifying further areas 
for collaboration on research/survey development and implementation)166.   

2)  Expanding  joint  procurement  and  contracting  procedures  to  cope  with  the 
increasing number of regulatory requirements and to improve procurement procedures, 
particularly in EU-OSHA167.  

3)  Streamlining performance indicators and reporting activities (see section above).  
4)  Reducing  administrative  burdens  of  the  tripartite  boards  of  Eurofound,  Cedefop 

and EU-OSHA (see section below). 

5)  Balancing operational and administrative staffing: explore possibilities to increase 
the ratio of operational staff to administrative staff and reduce the reliance on external 
contractors. This could be based on: (i) potential gains in efficiency (as highlighted in 
this section); and (ii) prioritising activities and output without compromising quality.  

4.1.2.5 EFFICIENCY OF GOVERNANCE STRUCTURES 

To provide further insights into the efficiency assessment, the evaluation considered a specific 
aspect  of  the  Agencies'  administration  process,  namely  the  efficiency  of  their  governance 
arrangements.  

Eurofound,  Cedefop  and  EU-OSHA’s  managing  boards  have  a  tripartite  structure.  Each 
Member State is represented in each board by a government, an employer and a trade union 
member (81 representatives in total). The European Commission has three representatives, and 
there is an independent expert (without voting rights) appointed by the European Parliament168.  

The smaller size of the ETF’s governing board includes 27 representatives from Member States 
(without social partners), three Commission representatives (who share one vote in the Board), 
three experts appointed by the European Parliament (compared to one per tripartite Agency), 
and three partner country representatives appointed by the Commission. 

166 For example, by contributing to the development and implementation of Eurofound’s surveys, Cedefop,  the 
ETF  and  EU-OSHA  could  lower  Eurofound’s  production costs  and  make  their  surveys  more  effective  by 
ensuring the surveys capture key information relevant to them. 

167 For EU-OSHA, this could be achieved by: i) diversifying the contractor base, such as by requesting focal 
points to disseminate procurement opportunities in their respective countries and/or by promoting them 
through social media; and ii) adopting a more selective approach to contractors by setting out more stringent 
criteria, thereby reducing the time needed to review outputs. 

168  In  addition,  Cedefop  and  Eurofound  have  two  coordinators:  one  from  ETUC  for  employees  and  one  from 

Business Europe for employers. 

42 

 
 
 
The  key  benefits  of  the  tripartite  governance  composition  are  the  representation  and 
strategic  direction  provided  by  social  partners,  in  addition  to  the  government 
representatives. Respondents across these three groups unanimously recognise the value of 
being  heard  and  represented.  In  addition,  the  management  boards,  with  such  an  extended 
membership,  constitute  a  significant  pool  of  experts  contributing  to  the  Agencies’  annual 
programmes with their knowledge and experience and serving as multipliers for disseminating 
evidence at national level.  

However, governance efficiency faces challenge from the size and the number of voting 
members in this tripartite model in terms of: (i) the costs incurred by the Agencies to run the 
boards,  including  the  organisation  of  meetings,  information  support,  logistics;  and  (ii)  the 
efficiency of the decision-making process and the way in which stakeholders are involved. 

This tripartite structure is also not in line with the Common Approach169, which suggests that 
the management board should be limited in size170 so that it can operate as a true supervisory 
body  rather  than  a  consultative  assembly171.  This  matter  has  been  flagged  repeatedly  by  the 
European Parliament in its discharge reports. It was also raised in the study requested by the 
BUDG Committee on the management boards of the decentralised agencies (published October 
2021)  and  in  the  ECA  Special  Report  on  the  Future  of  EU  Agencies  –  Potential  for  more 
flexibility and cooperation (2020)172. 

However, the previous evaluation showed that the size and composition of the governing boards 
were not a significant concern for the Agencies themselves or for most of their stakeholders. 
Most respondents to targeted surveys positively evaluated the size and composition, and this 
assessment remains unchanged in the current evaluation (see Figures 4 and 5 in Annex VII). In 
contrast, as in the previous evaluation, a significant share of respondents to the Eurofound staff 
survey173,  corroborated  by  staff  interviewed  in  Eurofound  and  Cedefop,  consider  that, 
compared to the small number of internal staff, particularly research staff, the management 
board is disproportionally large, bringing with it financial, administrative and logistical 
challenges.  

In assessing efficiency and the potential for streamlining, the evaluation explored alternative 
governance models with different ways of involving social partners. It compared the advantages 
and disadvantages of the tripartite management boards of Cedefop, Eurofound, and EU-OSHA 

169 The Common Approach on EU decentralised agencies, agreed in July 2012 by the European Parliament, the 
Council and the Commission, sets out the framework for the functioning of EU decentralised agencies. It aims 
to improve coherence, effectiveness, efficiency, accountability and transparency. 

170 According to the Common Approach, an agency’s managing board should consist of one representative from 
each  Member  State,  two  representatives  from  the  Commission,  one  member  designated  by  the  European 
Parliament  (where  appropriate)  and  a  ‘fairly  limited’  number  of  stakeholder’s  representatives  (where 
appropriate). 

171 The ETF governing board’s limited size is in line with the Common Approach. It includes 27 representatives 
from Member States (without social partners), three representatives of the Commission (who share one vote 
in the Board), three experts appointed by the European Parliament (as opposed to one per tripartite agency), 
and three representatives of partner countries appointed by the Commission. 

172 This was also highlighted in the previous 2019 evaluation of the Agencies. 
173 Only 51% of Eurofound staff (N=52) found the size of their governing board appropriate. It was 48% (N=84) 

in the previous round. 

43 

 
 
 
with the ETF’s Governing Board and other models (ELA, EIGE, ILO)174. This comparison took 
into account the particular features of these agencies, including which countries are covered by 
the Agencies’ analysis. The comparative analysis of the different models led to the following 
conclusions. 

•  Meeting costs for larger boards are higher, especially if they are held in person, although 
this  can  be  moderated  to  some  extent  by  adopting  online  or  hybrid  meetings  formats175. 
Since 2020, there has been a shift to more online meetings, which is reflected in the reduced 
overall costs  (see Table 5)176.  

Table  5  –  Amount  spent  on  tripartite  board  meetings  as  a  percentage  of  operational 
expenditure (Cedefop, Eurofound) (%)*  

2016 

2017 

Cedefop  
Eurofound  

2.78 
2.44 
* Data for EU-OSHA is not available. 

2.66 
2.26 

2018 

2.71 
2.65 

2019 

2.70 
3.67 

2020 

0.66 
2.45 

2021 

1.63 
2.30 

2022 

1.88 
1.17 

•  The productivity of management board meetings is dependent on the level of its members’ 
involvement. The engagement and level of knowledge will vary among members in such 
large  governing  bodies.  In  particular,  social  partner  representatives  (especially  national 
ones)  often  lack  background/experience  in  (EU-specific)  administrative,  financial, 
institutional  and  other  governance/  matters.  Consequently,  in  some  Agencies,  time  is 
sometimes lost on explaining issues or discussing problems raised as a result of a lack of 
knowledge or a misunderstanding of the institutional context.  

•  While  applicable  in  varying  degrees  to  the  Agencies  concerned  and  in  line  with  the 
Founding  Regulations,  a  portion  of  the  topics  covered  by  management  board  meetings 
concern  administrative  matters,  sometimes  overshadowing  core  business  topics.  In 
Agencies where such discussions represent a significant part of the discussions, this raises 
the question whether this is the best use of the time of people who are mainly policy experts. 
Although stakeholder expertise in the Agencies’ work is an important asset that cannot be 
compromised for the sake of financial efficiency, there are other models that might allow 
involving stakeholders in an equally effective but more efficient manner. 

•  Even  though  the  mission  of  each  agency  is  laid  down  in  their  respective  founding  acts, 
groups will tend to represent their specific interests; this can result in a careful balancing 
act  for  the  Agencies’  management  and  a  potential  loss  of  focus  in  their  objectives  and 
activities177. 

The ETF’s governing model is considered to be financially and procedurally efficient. It is also 
in  line  with  the  Common  Approach  mentioned  above.  A  smaller  board  facilitates  smooth 
logistics for in-person meetings and information support from the agency. Along with effective 

174 See the comparative SWOT analysis in the supporting study, Section 4.1.2.4.  
175 See supporting study (Table 35) for a breakdown of expenditure on tripartite bodies’ meetings by agency. 
176 For example, after the year of lockdown, EU-OSHA decided to merge executive board and management board 

meetings, reducing them to one face-to-face meeting per year. 

177 See the previous 2019 evaluation, where it is stated that ‘There is also a “multiple principals” situation, as the 
social  partners,  the  Member  States  and  the  Commission  tend  to  have  different  views  on  the  Agencies’ 
objectives and priorities’.  

44 

 
 
 
 
and efficient decision-making, this allows for clearly defined priorities and focused activities.178 
On the other hand, as there is no involvement of social partners, the main concern is lack of 
ownership and less precise targeting of policies and tools.  

Smaller  sized  executive  boards  mitigate  to  some  extent  some  of  the  tripartite  model’s 
inefficiencies  described  above.  However,  the  analysis  of  other  governance  models  in  this 
evaluation’s supporting study points to possible alternatives that could keep the current model’s 
strengths (notably the involvement of stakeholders, in particular social partners) and address its 
weaknesses. 

In this context, the evaluation considered the broader EU governance structure in the domains 
concerned, with the Advisory Committee on Vocational Training (ACVT)179 and the CEDEFOP 
management board both covering the same policies (VET), even if their roles are different. The 
same applies to the Advisory Committee on Safety and Health at Work180 and the EU-OSHA 
management board. A clearer distinction between the advisory and governance roles in the 
management boards could allow stakeholders to focus on their domain of expertise and 
offer significant efficiency gains.  

A  case  in  point  is  the  fact  that  the  composition  of  existing  advisory  committees  and  the 
corresponding Management Board overlaps significantly (ACVT/Cedefop, and OSH Advisory 
Committee/EU-OSHA181). In this context, the ELA model could be explored (while the ELA 
management board is not tripartite, social partners are involved through a separate stakeholder 
group, which gives thematic expertise to the management board)182. 

4.1.3. COHERENCE 

4.1.3.1 COHERENCE BETWEEN THE AGENCIES MANDATES AND ACTIVITIES 

During  the  evaluation  period,  the  mandates  and  activities  of  the  four  Agencies  were 
coherent. The four Agencies’ mandates identify broad policy themes in which  they operate 
(see  Section  2.1).  While  the  four  Agencies  worked  on  related  policy  areas  (labour  markets, 
vocational  education  and  training,  skills,  qualifications,  health  and  safety  at  work,  and  the 

178 Based on interviews with ETF and Commission staff. 
179 Since its creation in 1963, the Advisory Committee on Vocational Training (ACVT) is a tripartite consultative 
body that has the task of assisting the Commission in implementing  EU vocational education and training 
policy.  

180 The Advisory Committee on Safety and Health at Work is a tripartite consultative body whose remit is to assist 
the Commission in preparing and implementing decisions taken in the field of safety and health at work and 
to facilitate cooperation between national administrations, trade unions and employer organisations. 

181 52% of the government representatives in the ACVT and the Cedefop management board are the same, as are 
44%  of  employer  and  41%  of  trade  union  representatives.  There  is  total  overlap  among  EU-OSHA 
management board members and members of the Advisory Committee on Safety and Health at Work. 
182 Pending the outcome of the ELA evaluation, which is currently underway. The ELA management board is not 
tripartite;  it  is  made  up  of:  (a)  one  member  from  each  Member  State;  (b)  two  members  representing  the 
Commission;  (c)  one  independent  expert  appointed  by  the  European  Parliament;  and  (d)  four  members, 
representing social partner organisations at EU level, without voting rights. In addition, there is an advisory 
body  called  the  ‘Stakeholder group’,  composed of  social  partners  (and  Commission  representatives).  This 
group gives expert advice on certain areas to the management board. Adopting the ELA model would involve 
changing the Agencies’ Founding Regulations. 

45 

 
 
 
 
 
working  environment),  each  Agency  approached  the  broad  policy  themes  from  different 
perspectives. Where two or more Agencies’ mandates covered the same policy, the Agencies 
cooperated. Cooperation between the Agencies was closer and deeper than in the previous 
evaluation period, and the evaluation found no instances of activities being duplicated. 

Considering the Agencies’ mandates and strategic objectives, there are shared themes addressed 
by  two  or  more  Agencies.  However,  evidence  shows  that  the  Agencies  approached  these 
common policy themes from different perspectives or a different geographical focus. The most 
significant cases concerned three pairs of agencies.  

(i) Cedefop and the ETF both worked on issues such as VET183. However, the actual focus of 
the two Agencies’ work in this area was clearly distinguished: the  ETF focused on capacity 
building and policy implementation in partner countries outside the EU, while Cedefop focused 
on research and policymaking in the EU.  

(ii)  Cedefop  and  Eurofound  worked  on  the  intersection  between  VET,  skills  mismatches, 
working  and  living  conditions  and  labour  market  policies184.  The  two  Agencies  approached 
these topics from different angles: Cedefop focused on skills mismatches and VET providers 
from the supply side, while Eurofound focused on the demand side. 

(iii) Eurofound and EU-OSHA worked on occupational health and safety as this is connected 
to  Eurofound’s  wider  ‘working  conditions’  field.  It  is  noteworthy  that  EU-OSHA  was 
established  after  Eurofound.  The  Founding  Regulation  of  EU-OSHA  established  a  specific 
focus on providing highly technical and specialised support for protecting the safety and health 
of  workers.  This  mandate  requires  a  high  level  of  expertise,  especially  in  light  of  the  rapid 
societal,  technological  and  scientific  developments  in  occupational  safety  and  health.  The 
Agencies’ focus and ways of working are also different. Eurofound focuses on broad research 
(e.g.  European  working  conditions  survey),  whereas  EU-OSHA  focuses  on  providing 
workplaces  with  practical  knowledge  and  tools  for  implementing  OSH  rules  and  raising 
awareness, using their network of national focal points. 

According to the open public consultation, most stakeholders familiar with the Agencies’ 
mandates and activities agree that they are coherent or very coherent (53% of respondents, 
N=101). Only 3% of respondents believed that the mandates and activities were incoherent or 
very incoherent. 46% did not know if the mandates and activities were coherent and were not 
able to answer this question. 

The current evaluation has found no instances of duplicated work across the Agencies185. 
While  the  Agencies’  mandates  covered  broad,  complex  and  interconnected  themes,  the 

183 PPMI, ECORYS (2018), Evaluation of the EU Agencies under the remit of DG Employment: EUROFOUND, 

CEDEFOP, ETF and EU-OSHA. 

184 Ipsos Mori (2016), Ex-ante evaluation of Eurofound’s Draft Programming Document 2017-2020, p. 21. 
185 The evaluation used several methods to determine if there was any duplication: (i) a detailed analysis of each 
agency’s output and consolidated activity reports, (ii)  Mapping of collaboration agreements, cooperation at 
the Management/Governing Board level, and joint activities to determine the mechanisms in place to avoid 
potential duplication; and (iii) the staff survey included a question asking if staff  were aware of duplicated 
work with other decentralised agencies. 

46 

 
 
 
 
common thematic areas were well understood by most policymakers and agency staff. Thematic 
overlaps in the mandates did not necessarily mean duplicating work. In fact, common thematic 
areas were both accepted and used as a basis for inter-agency cooperation, which increased 
consistency. For examples of joint output and activities delivered by the agency pairs related to 
common thematic areas, see the dedicated section below and Table 4 in Annex VII. Moreover, 
staff in all agencies were much more aware of each other’s work than in the previous evaluation 
and were therefore able to complement each other’s work rather than duplicate it.  

The evaluation also assessed whether the creation of the European Labour Authority (ELA) 
could prompt a reassessment of the previous evaluation’s conclusions on merging the Agencies. 
In this regard, the risk of the Agencies duplicating the work of ELA was limited. EU-OSHA, 
Cedefop, and Eurofound were predominantly research-focused and did not have an operational 
and cross-border focus as the ELA did. The potential for merging the ELA with the ETF was 
also limited as the ETF only dealt with non-EU countries, mainly in the EU’s neighbourhood.  

Cooperation between the four Agencies and the ELA was at an early stage due to the recent 
establishment of the ELA. Nevertheless, the three tripartite agencies had begun to cooperate 
with ELA on certain issues, for example on seasonal workers campaigns (EU-OSHA) and skills 
intelligence and job shortages (Eurofound and Cedefop). All five agencies cooperated when the 
ELA  participated  in  the  September  2022  event  ‘Employment,  skills  and  social  policies  that 
work for young Europeans in times of uncertainly’. 

Nevertheless, there is scope for further cooperation with the ELA. For instance, the four 
Agencies could support the ELA by sharing their existing research in their respective areas of 
competence; this could also lead to pooling resources or producing joint reports on issues of 
common  concern,  e.g.  on  restructuring  or  respecting  occupational  health  and  safety  rules  in 
mobility  situations,  including  posting,  in  the  context  of  risk  assessment.  The  evaluation 
identified two further areas for cooperation with ELA:  (i) digital labour and labour mobility 
with ETF; and (ii) OSH implementation and workplace working conditions with EU-OSHA. 

4.1.3.2 COOPERATION BETWEEN THE FOUR AGENCIES 

Joint  work  and  cooperation  increased  during  the  evaluation  period.  The  2011-2016 
evaluation showed some cooperation between the agencies, particularly between Cedefop and 
ETF,  Eurofound  and  EU-OSHA,  and  Eurofound  and  Cedefop.  Formal  agreements  already 
existed between them, but cooperation was mostly limited to the mere exchange of information 
and knowledge, without leading to joint outputs and lacking strategic planning. Therefore, it 
identified a need for strengthening cooperation, engaging in joint-value creation, developing 
mechanisms for joint delivery, and sharing support services.  

From 2017 to 2022, cooperation and synergies among the Agencies intensified, with a rise 
in the number of initiatives and projects where Agencies have cooperated. This includes the 
development  of  joint  research,  tools,  events,  and  publications,  demonstrating  genuine 
cooperation  with  clear  shared  input  and  output  from  two  or  more  agencies.  This  increased 
cooperation was at both the strategic (board) level and project/initiative level. 

47 

 
 
 
 
Cooperation to share  administrative functions, such as HR and finance, also  increased 
during the evaluation period. The Agencies were more likely to procure services together than 
in  the  previous  evaluation  period.  They  launched  joint  framework  contracts  and  cooperated 
with other agencies and the Commission. As discussed below, opportunities could be explored 
for sharing more staff and office space in Brussels. 

The  benefits  of  cooperation  between  the  Agencies  are  in  three  main  areas.  First,  improved 
coordination  and  complementarity  arising  from  interaction  at  different  levels  on  different 
activities:  there  are  over  20  formal  agreements  in  place,  and  over  30  joint  projects  (events, 
research,  procurement)  have  been  delivered  as  a  result  of  cooperation.  The  benefits  of  the 
Agencies complementing each other during this evaluation period are best illustrated through 
the  joint  work  of  the  Agencies  on  EU  cross-cutting  priorities  (the  green  transition,  digital 
economy and migration). Second, improved relevance and quality of joint Agencies outputs. 
Joint  outputs  benefit  from  co-production,  through  broader  evidence  or  expertise  and  the 
possibility  to  reflect  multiple  perspectives  (e.g.  Eurofound-Cedefop  joint  working  on  skills 
forecast).  Moreover,  joint  dissemination  has  a  wider  reach  due  to  the  use  of  the  Agencies’ 
multiple  communication  channels.  This  is  a  marked  advance  compared  to  the  previous 
evaluation period where separate reports were produced on similar issues for the same audience. 
Third, improved efficiency of individual Agency operations thanks to co-financing. A case 
in point was the joint Eurofound / Cedefop European Company Survey (2019).  

The extent of cooperation between the four Agencies is analysed below under four headings: 
(i) joint cooperation agreements, activities, events and data collection; (ii) cooperation between 
management boards; (iii) cooperation on administrative and organisational functions; and (iv) 
joint communication activities. 

(i) Joint cooperation agreements, activities, events and data collection 

Overall, the number of joint action plans and cooperation agreements between the Agencies 
increased (in particular, between Cedefop and Eurofound, EU-OSHA and Eurofound, Cedefop 
and  the  ETF,  and  the  ETF  and  Eurofound)186.  Staff  surveys  show  that  formal  cooperation 
agreements increased staff awareness and clarified each agency’s role in projects. A number of 
joint activities and events were organised, and shared data collections were created.  

•  Cooperation between Cedefop and Eurofound became more extensive. The two Agencies 
launched  an  EU-wide  joint  European  company  survey.  Moreover,  joint  actions  and 
publications  focused  on  skills  forecasting  and  manufacturing.  These  included  the  ‘Skills 
forecast: trends and challenges to 2030’ report187 and the joint report on ‘Workplace practices 
unlocking  employee  potential’188.  A  recently  revised  memorandum  of  understanding 
includes contact points for all areas of joint interest. 

186  The  mapping  identified  the  following  new  framework  agreements:  (i)  ETF  and  Cedefop  had  collaborated 
extensively  prior  to  the  current  evaluation  period  but  had  two  framework  agreements  during  the  current 
evaluation period (for 2018-2020 and 2021-2023), compared to one during the previous period; (ii) for the 
first time, Cedefop and EU-OSHA prepared a cooperation agreement, which was signed in 2023; and (iii) 
Cedefop and Eurofound updated their memorandum of understanding in 2020, which was signed for the first 
time in 2006. 

187 Cedefop, Eurofound (2018), ‘Skills forecast: trends and challenges to 2030’, Luxembourg.  
188 https://www.cedefop.europa.eu/en/publications/2228  

48 

 
 
 
•  Cooperation  between  Eurofound  and  EU-OSHA  is  not  new,  and  formal  agreements 
followed by action plans date back to 2007. The two agencies have focused their cooperation 
on  a  range  of  themes,  including  psychological  risks,  mental  health,  self-employment, 
working-time  patterns,  burn-out,  older  workers,  and  foresight189.  During  the  COVID-19 
pandemic  (May  2020),  the  European  Parliament’s  Employment  and  Social  Affairs 
Committee (EP EMPL) held an exchange of views with the two agencies on the social and 
employment impact of the pandemic190. For details, see Annex IX, Section 4(i). 

•  Cooperation between Cedefop and the ETF, specifically on VET policy monitoring, is 
envisaged in their respective mandates. The agencies have a 3-year framework agreement191,  
which is implemented through a joint annual work programme annexed to each  agency’s 
programming documents. The two agencies report on progress to the European Parliament 
every 2 years. The following joint actions were implemented during the evaluation period: 
o 
joint support for monitoring the Riga Conclusions deliverables in the VET field;  
o  regular  joint  contributions  to  work  on  national  and  regional  qualifications 

o 

frameworks in cooperation with UNESCO (2017-2018, 2020); 
joint  implementation  of  the  European  Skills  and  Jobs  Survey  (ESJS)  and  joint 
monitoring of the implementation of the Osnabrück Declaration, supporting digital 
skills and inclusion in VET (from 2020);  

o  Cedefop and the ETF shared expertise on validating refugee work skills in Türkiye. 

•  Cooperation between the ETF and Eurofound increased, especially on social partnership 
and  dialogue,  public-private  partnerships  in  VET,  skills  shortages  and  skills  mismatches. 
The two agencies launched a joint online survey comparing the impact of the COVID-19 
pandemic  on  EU  Member  States  and  selected  ETF  partner  countries.  The  agencies  also 
cooperated  on  themes  related  to  platform  work,  NEETs  and  capacity  building  of  social 
partners through sharing data, contributing to events and knowledge-sharing activities. In 
particular, they participated in or jointly organised seminars on the digital economy, platform 
work  (2019,  2020,  2021)  and  the  impact  of  COVID-19  on  businesses  (2021).  Moreover, 
Eurofound  participated  in  the  advisory  board  of  the  ETF’s  initiative  on  skills  demand  in 
partner countries (2020). 

•  Although the ETF and EU-OSHA do not have a joint action plan or framework agreement 
(due to limited possibilities for thematic cooperation), they do have a service level agreement 
in place to share an accounting officer (see point (iv) below). 

•  Cooperation between the Agencies increased on pan-European surveys. This cooperation 
led to wider benefits, such as cost sharing (particularly for the European company survey), 
aligning definitions and tools, facilitating sharing data and best practices, and overcoming 
methodological difficulties. For more details, see Annex IX, Section 4(ii). 

189 Eurofound (2017), Consolidated annual activity report of the Authorising Officer for the year 2017, p. 25.  
190 See: https://www.eurofound.europa.eu/nl/events/european-parliament-discussion-with-eurofound-and-eu-

osha-impacts-of-the-COVID-19-pandemic.  

191 ETF, Cedefop (2018), Framework agreement for cooperation between the European Centre For The 

Development Of Vocational Training (Cedefop) and The European Training Foundation (ETF).  

49 

 
 
 
•  The  Agencies  were  also  involved  in  several  multilateral  (more  than  two  agencies),  in 
particular  on  digitalisation,  the  COVID-19  pandemic,  Russia’s  war  of  aggression  against 
Ukraine  and  readiness  for  future  challenges/foresight.  For  more  details  see  Annex  IX, 
Section 4(iii)192. 

•  Overall, various stakeholder groups across both the public consultation and the stakeholder 
survey  gave  positive  ratings  to  the  joint  publications,  joint  data  collection  and  joint 
information  or  expertise  shared.  However,  when  analysing  these  surveys  by  specific 
stakeholder groups, there were no conclusive patterns on the perceived quality of these joint 
activities.  

(ii) Cooperation between management boards 

Each  agency  has  a  management  board  to  provide  leadership  and  strategic  direction.  All  the 
boards strengthened inter-agency cooperation by exchanging information at their level 
on activities, priorities and operational issues. Most boards now include representatives from 
the other agencies, and there is a general agreement on cross-participation among boards. This 
leads  to  better  understanding  the  ongoing  work  of  other  agencies,  sharing  programming 
documents earlier, developing joint action plans and generally making the Agencies’ activities 
more consistent. Previously, cooperation between the Agencies was primarily bottom-up rather 
than  based  on  joint  strategic  planning193.  Although  board-level  cooperation  is  positive,  the 
Agencies’ staff194 did point out that participation in other board meetings is time-consuming.  

(iii) Cooperation on administrative and organisational functions 

Beyond  operational  matters,  the  four  Agencies  cooperated  on  various  administrative  and 
organisational  functions;  however,  there  could  be  room  to  improve  the  scope  of  this 
cooperation.  In  terms  of  sharing  and  co-funding  staffing,  there  are  a  few  instances  of 
collaboration in HR, IT, data protection and legal services. One particular example is the formal 
ETF–EU-OSHA agreement to share an accounting officer through a service level agreement195. 
Where feasible, opportunities could be further explored for sharing roles that require technical 
or  specialist  skills,  for  example,  in  cybersecurity,  employment  statistics  and  labour  market 
economics.  

On office space, close cooperation takes place between the staff of the three existing Brussels 
Liaison Offices (Brussels Liaison Offices were set up by all four Agencies, although the ETF 
closed its office in 2019). These offices support partnership with stakeholders, monitor changes 
in EU policymaking and contribute to the Agencies’ strategic management and communication. 
The  recommendation  from  the  previous  evaluation  to  merge  agency  offices  was  not 
implemented. The possibility to share office space in Brussels could be considered. Locating 
all liaison office staff in one Brussels could stimulate ongoing and ad hoc collaboration and 

192  In  addition,  Cedefop  and  EU-OSHA  prepared  a  cooperation  agreement,  even  though  its  signature  falls 
outside the scope of this evaluation (September 2023). The cooperation between the two Agencies will focus 
on the collection, analysis, and dissemination of information on the protection and development of skills of 
workers. 

193 PPMI, ECORYS (2018), Evaluation of the EU Agencies under the remit of DG Employment: 

EUROFOUND, CEDEFOP, ETF and EU-OSHA, p.157. 

194 Interviews with agency staff carried out between February and June 2023. 
195 EU-OSHA (2021). Annual Activity Report 2021. 

50 

 
 
 
allow agency staff visiting from their headquarters to work together and save costs across all 
four Agencies.  

On  joint  procurement,  in  contrast  to  the  previous  evaluation  period,  the  Agencies  were 
increasingly likely to procure work together. Between 2017 and 2022, Cedefop participated in 
18 joint procurements led by other EU agencies and launched one joint procedure for its ICT 
helpdesk service with ENISA (the EU Agency for Cybersecurity) in 2019. It also participated 
in  40 Commission-led procurements  for HR, ICT, audits  and communications  activities and 
launched  one  joint  procedure  with  Eurostat.  Eurofound  launched  two  multiple  framework 
contracts for evaluation and feedback services. The 2017-2020 contract was shared among eight 
EU  agencies.  The  2021-2025  contract  is  shared  among  15  EU  agencies.  Eurofound  also 
launched a joint procurement procedure in cooperation with Cedefop to align its performance 
measurement system with other agencies. The ETF has participated in five joint tenders since 
2017  (although  not  all  were  with  other  agencies),  and  16%  of  its  contracts  were  concluded 
following 
tenders  with  other  agencies  or  with  Commission 
departments. Lastly,  EU-OSHA  also  participated  in  joint  procurement  activities  with  other 
agencies  and  the  Commission  to  save  costs,  improve  quality  and  increase  efficiency.  The 
agency also chaired the EUAN Shared Services Task Force.  

joint  procurement 

(iv) Joint communication activities 

Joint communication activities were largely focused on outputs and events. The Agencies’ 
staff emphasised that joint dissemination ‘reach’ was greater when two or more agencies used 
their dedicated networks, databases and other means to engage with stakeholders.  

For example, the report titled ‘Towards age-friendly work in Europe: a life-course perspective 
on work and ageing from EU Agencies’196 and another report, ‘Health and safety risks at the 
workplace:  a  joint  analysis  of  three  major  surveys’197,  were  shared  across  multiple  channels 
(membership databases, agency websites, newsletters, mentioned at events).  

Staff highly rated the cooperation and joint activities with other agencies under the remit of DG 
EMPL (Figure 11 in Annex VII)198. However, cooperation with the ELA was assessed slightly 
less positively compared to other agencies. This may be attributed to the ELA being a newly 
established agency, which has allowed for less time for develop joint activities. 

4.1.3.3 COHERENCE WITH THE EUROPEAN COMMISSION’S STRATEGIC PRIORITIES FOR 2019-
2024  

The mandates, activities and outputs of the four Agencies were coherent with the Commission’s 
strategic priorities for 2019-2024. In particular, the Agencies produced many tailored outputs 
related to the priorities199 ‘A Europe fit for the digital age’ and ‘An economy that works for 
people’. The Agencies’  work was aligned  with new strategic priorities tailored to  initiatives 
such as the European Green Deal (see Table 5 in Annex VII for more details about the coherence 

196 EU-OSHA, Cedefop, Eurofound, EIGE (2017), ‘Towards age-friendly work in Europe: a life-course 

perspective on work and ageing from EU Agencies’. 

197 https://osha.europa.eu/sites/default/files/Health_and_safety_risks_at_work_joint_analysis_summary.pdf 
198 Staff views may be subject to bias. 
199 Von der Leyen, U. (2019), ‘A Union that strives for more – My agenda for Europe – Political Guidelines for 

the next European Commission 2019–2024’, 16 July 2019.  

51 

 
 
 
of the  four  Agencies’ mandates  and activities with  the Commission’s strategic priorities  for 
2019–2024).  

Figure  3:  Coherence  of 
programmes/initiatives 

the  Agencies’  mandates  and  activities  with  other  EU 

To what extent do you think that the Agencies’ mandates and activities are 
coherent with other relevant EU programmes / initiatives?

60%

50%

40%

30%

20%

10%

0%

51%

32%

32%

28%

45%

45%

47%

39%

2%

0%

0%

0%

0%

0%

6%

2%

Very coherent

Coherent

Incoherent

Very incoherent

Eurofound
N = 31

Cedefop
N = 44

ETF
N = 32

EU-OSHA
N = 41

Source: Public consultation, question: To what extent do you think that the Agencies’ mandates and activities are 
coherent with other relevant EU programmes/initiatives? 

Most respondents to the public consultation believed that the Agencies’ mandates and activities 
were very coherent or coherent with those of the relevant EU strategic priorities (see Figure 3).  

4.1.3.4 COHERENCE WITH DG EMPL AND OTHER COMMISSION DEPARTMENTS 

The Agencies’ mandates and activities were coherent with those of  DG EMPL: all four 
Agencies worked in thematic areas that were relevant to and coherent with DG EMPL policies, 
in particular the implementation of the European Pillar of Social Rights (see Section 4.1.1 on 
effectiveness for detailed evidence).  

The ETF’s mandate is a special case as it focuses exclusively on the partner countries outside 
the EU, while DG EMPL focuses largely on EU Member States. Consequently, the ETF also 
worked with other DGs with an international profile (e.g. DG NEAR, DG INTPA).  

DG EMPL was represented in the management boards of each agency, providing input to the 
programming cycle and decision-making at board level. In particular, DG EMPL participated 
in the executive and management boards of Eurofound, Cedefop and EU-OSHA, with director-
level representation in each. The ETF Governing Board is chaired by the Director-General of 
DG EMPL.  

Overall,  DG  EMPL  and  the  Agencies  worked  collaboratively  during  the  evaluation 
period, and the Agencies’ outputs were used to inform EU policies, particularly those of 
DG EMPL. 

During the evaluation period, progress was made on having more regular formal and informal 
communication  between  DG  EMPL  and  the  Agencies.  Each  agency  has  set  up  cooperation 

52 

 
 
channels with DG EMPL, ranging from day-to-day cooperation at project level to interaction 
between senior managers in the Agencies and DG EMPL.  

On  thematic  contributions  to  DG  EMPL  policies,  events,  working  groups  and  advisory 
committees, each agency worked with DG EMPL based on its specific role. Most often, the 
Agencies provide research, expertise and input into  policy documents or proposals  and  also 
participate in a variety of formal committees and other meetings. For further details, see Annex 
IX (iv) and Section 4.1.1 on effectiveness. 

Coherence with other Commission departments 

The relationship between the ETF and DG EMPL is distinctive due to  each body’s different 
geographical coverage. This requires coordination among several DGs. The structured dialogue 
process200  between  DG  EMPL,  DG  NEAR  and  DG  INTPA  is  well-established201.  However, 
despite coordination, the different DGs have competing expectations of where the ETF should 
focus its  work  (reflecting the different  DGs’  objectives and geographic  responsibilities). An 
agreed joint vision between the different DGs and the ETF could benefit the agency’s work. 

For  EU-OSHA,  the  evaluation  found  that  cooperation  with  other  Commission  departments 
could  be  strengthened  for  small  and  medium-sized  companies,  research  dissemination  and 
mental  health.  This  could  involve  closer  cooperation  with  several  Commission  departments 
(DG JUST, DG FISMA, DG GROW and DG SANTE) and EASME, the executive agency for 
SMEs (supporting study Annex 4, pp. 104-5). 

4.1.3.5 COHERENCE WITH OTHER EU DECENTRALISED AGENCIES  

The mandates and activities of the four Agencies were consistent with those of other relevant 
decentralised agencies. Most cooperation between the four Agencies and other decentralised 
agencies took place through the EU Agencies Network (EUAN) and the EUAN sub-network, 
EU-ANSA (EU Agencies Network on Scientific Advice)202. For further details, see Annex IX, 
4(v).  

Via the EUAN, the four Agencies and other EU agencies pooled existing expertise to address 
new  cross-cutting  themes,  such  as  the  socio-economic  impact  of  the  COVID-19  pandemic, 
digitalisation and the green transition. Sub-networks enabled agencies working with primary 
research to hold regular in-depth methodological exchanges, for example, on sustainability and 
strategic  foresight.  Another  benefit  of  cooperation  within  EUAN  was  staff  cooperation  on 
identifying  solutions  to  common  challenges  and  cost-saving  measures  (e.g.  on  risk 
management, the sustainable management of buildings and teleworking arrangements). 

Within  the  Performance  Development  Network  (PDN),  a  sub-network  of  the  EUAN203, 
Eurofound played a leading role in developing an evaluation handbook and catalogue of key 

200 Structured Dialogue is the consultation process between DG EMPL, DG NEAR, DG INTPA and ETF to prepare 
for governing board meetings and coordinate the decision-making process between the three DGs. It takes 
place twice a year in line with the governing board meetings. 

201 Apart from the structured dialogue, ETF also cooperated with other DGs, such as DG HOME.  
202 The four Agencies are active members of the EUAN and their sub-networks. 
203 European Environment Agency, 2021-2022 Work Programme of the EU Agencies Network.  

53 

 
 
 
performance indicators across all four Agencies. The Agencies also took part in PDN exchanges 
on experiences with user satisfaction.  

EU-OSHA  intensified  cooperation  with  the  European  Centre  for  Disease  Prevention  and 
Control (ECDC), particularly in response to the COVID-19 pandemic. Notably, this was the 
first time EU-OSHA cooperated with other agencies on public health issues. 

In  2021,  the  ETF  began  cooperation  with  EIGE  on  the  gender  equality  index.  Given  the 
overlapping geographical mandates of ETF and EIGE, which include EU candidate countries 
and potential candidates, there is an opportunity to improve the consistency of the two agencies’ 
activities. This would give ETF the opportunity to mainstream the gender equality part of the 
European Social Pillar in EU partner countries. 

4.1.3.6 COHERENCE WITH OTHER INTERNATIONAL ORGANISATIONS  

All four Agencies cooperated with international organisations active in the same policy areas 
through joint publications, events and surveys. This cooperation intensified, in particular, with 
the ILO and the OECD through cooperation on new thematic areas. 

All four Agencies had already cooperated with the ILO before 2017 leading to joint outputs 
produced between 2017 and 2022. Cooperation has increased, in particular as a result of the 
COVID-19 pandemic, as has the use of joint online surveys. For further details, see Annex IX, 
4(vi).  

Eurofound,  Cedefop  and  the  ETF  cooperated  with  the  OECD.  Eurofound  and  the  OECD 
cooperated on job quality, collective bargaining, minimum and living wages and platform work. 
Cedefop and the OECD have cooperated on the role of VET in employing refugee and migrant 
workers  since  2017,  and  in  2019,  a  2-yearly  joint  event  on  skills  was  launched.  The  ETF 
cooperated with the OECD on skills-demand analysis and foresight, qualification systems and 
working with companies on skills development.  

EU-OSHA  cooperated with the  World Health  Organization, including  via data sharing on 
healthcare.  The  two  organisations  attended  each  other’s  events  and  organised  joint 
workshops204. 

The ETF collaborated with the ILO, the OECD, UNICEF and UNESCO, and this cooperation 
has  intensified  since  the  start  of  the  COVID-19  pandemic  (see  supporting  study,  Annex  4, 
Section  6.2.3.1).  However,  as  the  international  landscape  for  human  capital  development 
becomes  more  complex,  involving  a  growing  number  of  non-traditional  players,  there  is  an 
opportunity for the ETF to strengthen cooperation with international donors and financial 
institutions (within the scope of its mandate). 

The four Agencies also continued to cooperate multilaterally with international organisations 
and  regularly  conducted  joint  activities  with  them.  These  included  the  ILO,  the  OECD, 
UNICEF and UNESCO.  

4.1.3.7 COHERENCE WITH THE NATIONAL LEVEL 

204 EU-OSHA (2023), Consolidated Annual Activity Report 2022.  

54 

 
 
 
Consistency with national-level work is supported through the representation of Member States 
in  all  the  Agencies’  management/governing  boards.  National  social  partners  are  also 
represented  in  the  management  boards  of  the  three  tripartite  Agencies.  Member  States  and 
social partners have two roles: (i) they provide input to the Agencies’ programming documents; 
and (ii) act as multipliers for disseminating the Agencies’ output. The extent to which the four 
Agencies provide direct  support to  Member States and  partner countries varies according to 
their mandates and their operations. A number of examples can be given for each agency. 

EU-OSHA  conducts  OSH  communication  campaigns  in  Member  States  that  would  not 
otherwise be possible in those that have less funding available for OSH systems. The OiRA 
tool is well regarded in many Member States as the tool is referred to in national OSH strategies 
and legislation. Broader references to risk assessment tools suggest that EU-OSHA awareness-
raising has influenced national policies. 

Cedefop prepared country-focused thematic reviews and attended events organised at national 
level.  In  2021,  thematic  reviews  and  peer-learning  activities  were  organised  on  skills 
governance  in  Slovenia  and  on  upskilling  pathways  in  France  and  Italy.  Cedefop  supported 
Slovenia’s Labour Ministry in developing national skills forecast and a career platform. The 
agency contributed to a total of 18 events organised at national level.  

Eurofound’s outputs are not directly aimed at specific Member States. Nevertheless, Belgium, 
France,  Slovenia  and  Spain  provided  additional  funding  to  increase  the  sample  size  of  the 
European working conditions survey (EWCS) for their countries for a more in-depth analysis. 
Eurofound also participated in the research days of the Irish Low Pay Commission, and an Irish 
government speech from April 2022 acknowledged Eurofound research. 

The  ETF  works  to  support  human  capital  development  reforms  in  partner  countries.  In 
Moldova, support for VET reforms contributed to sectoral committees being set up to address 
labour market needs. In Türkiye, the support provided for setting up centres for the recognition 
of prior learning contributed to establishing 25 centres for VET recognition. National ministries 
in  Ukraine  and  Azerbaijan  have  used  SELFIE  (Self-reflection  on  Effective  Learning  by 
Fostering the use of Innovative Educational technologies) as a source for policy discussions.  

4.1.3.8 POTENTIAL FOR MERGERS: PREVIOUS EVALUATION FINDINGS AND CURRENT 
EVALUATION ASSESSMENT 

Based on the assessment of the Agencies’ mandates and common thematic areas, the previous 
cross-cutting evaluation in 2019 identified which Agencies might be more suitable for a merger 
and assessed the benefits and risks of each merger scenario. 

The  previous  evaluation  concluded  that  there  was  no  straightforward  scenario  for 
mergers  and  that,  to  improve  efficiency,  there  was  rather  room  for  strengthening 
cooperation.  Although  Cedefop/ETF  and  Cedefop/Eurofound  merger  options  could  be 
considered, they were high-risk scenarios, and there were difficulties in balancing the positive 
and negative effects.  In  both  merger options, potential  synergies  could  be exploited through 
developing methodological expertise and the pooling of knowledge (e.g., in skills anticipation 
and matching). Cedefop/ETF merger cost savings were possible for research roles (estimated 
at  up  to  EUR 2  million)  and  for  back-office  roles  (up  to  EUR 1.5  million),  whereas  for 

55 

 
 
Cedefop/Eurofound,  they  were  estimated  to  amount  to  up  to  EUR 1.5  million  for  research 
roles and up to EUR 1.5 million for back-office roles (see more details on costs and benefits of 
these two merger options in Annex IX, Table 3: Overview of benefits and risks of agency 
merger scenarios). 

A  Eurofound/EU-OSHA  merger  was  also  considered.  Cost  savings  for  research  roles  are 
estimated at up to EUR 1 million and up to EUR 0.5-1 million for back-office roles). Potential 
synergies  could  be  exploited  by  pooling  methodological  expertise  in  researching  working 
conditions  and  OSH  issues.  However,  it  was  concluded  that  a  possible  merger  would  face 
significant ‘stumbling blocks’, especially because the work of EU-OSHA is highly specialised. 
Moreover,  there  is  a  clear  risk  that  Eurofound,  as  the  larger  Agency,  would  become  the 
dominant partner and OSH policy issues are given insufficient priority relative to other issues. 
This would reduce their visibility and impact at the EU level. 

In  all  merger  scenarios,  closure  costs  are  high,  around  EUR   2-3  million  for  each  agency. 
Overall, as individual agencies were shown to be overall efficient and effective, the previous 
evaluation noted that one of the key reasons for a merger – joining a ‘weaker’ organisation to a 
‘stronger’ one – was absent.  

The support study for this evaluation confirmed the previous conclusions on the feasibility 
of mergers as there were no substantial changes to the mandates and activities during the 
current evaluation period, and the potential for duplication with the ELA is limited. The 
2019 revision of the tripartite Agencies’ Founding Regulations did not change or expand the 
scope  of  the  Agencies’  mandates,  and  the  Agencies  are  putting  in  place  cooperation 
mechanisms with the ELA to create synergies where possible. For a detailed analysis of the 
benefits and risks of the different agency merger scenarios, see Table 3 in Annex IX205. 

4.2.  HOW DID THE EU INTERVENTION MAKE A DIFFERENCE AND TO WHOM? 

This chapter examines the added value of the Agencies’ work and how they made a difference 
to  their stakeholders.  It  analyses the extent  to  which the Agencies’ contributions  are unique 
when  compared  to  those  of  other  agencies  and  organisations  and  the  extent  to  which  their 
activities could be substituted by other EU, international or national organisations.  

Like  the  previous  evaluation206,  this  evaluation  found  that  the  four  Agencies  continued  to 
bring  unique  value  through:  (i)  specialised  thematic  knowledge;  (ii)  high-quality  data, 
tools  and  methodologies;  (iii)  extensive  European  coverage  in  their  respective  fields, 
ensuring consistent data availability; and (iv) maintaining a tailored focus on EU policy 
needs207. This is confirmed by the results of the stakeholder survey (see Figure 7 in Annex VII).  

205 See also Section 4.1.2.7 of the supporting study. 
206 ‘Unique’ activities in the sense that other organisations or institutions are not concurrently engaged in activities 

with the same objectives, methodology, target groups and geographical scope. 

207 The Agencies’ work programmes are approved by their governing boards based on their relevance to the EU’s 
policy agenda (mostly driven by the Commission as it has a right of initiative). Few other research institutions 
possess such mandates and links to policymaking.  

56 

 
 
  
 
Overall,  the  Agencies’  stakeholders  considered  that  other  organisations  could  either 
partially or not at all replace the Agencies’ activities, emphasising the Agencies’ unique 
contribution. Around 75-80% of Cedefop and the ETF’s stakeholders believed that ceasing the 
Agencies’ activities would affect their work; for Eurofound and EU-OSHA208, over 90% of their 
stakeholders  shared  this  view.  This  feedback  was  comparable  to  the  previous  evaluation’s 
findings.  For  the  ETF,  some  stakeholders  (35  %  of  respondents  to  the  stakeholders  survey, 
N=91) indicated that international organisations, such as UNESCO, UNICEF, the OECD, and 
the ILO could carry out similar work to ETF, matching its level of expertise and organisational 
capacity209.  

The Agencies generate significant value for various stakeholders, albeit to different degrees.  

The Commission remained the primary beneficiary of all four Agencies, and the evaluation 
found extensive evidence of the added-value  and benefits  that the Commission  gained from 
their support. Examples include Eurofound’s knowledge and input in preparing one of the six 
political  priorities  of  the  Von  der  Leyen  Commission  in  2019  (An  economy  that  works  for 
people). Cedefop supported the Commission through its initiatives and support for empowering 
adults  through  upskilling  and  reskilling210,  among  others.  EU-OSHA  played  a  key  role  in 
shaping  EU OSH policy,  particularly  the EU’s  strategic  frameworks on  health and  safety  at 
work  (see  EU-OSHA  Case  Study  5).  Lastly,  the  ETF  provided  effective  support  to  EU 
development cooperation and external relations. (See Section 4.1.1.3). 

The  evaluation  underscored  the  added  value  provided  by  the  Agencies  to  Member  State 
stakeholders (and EU partner countries for ETF), offering quality data, extensive European 
coverage and thematic expertise. Cedefop provided support to education and training providers, 
academic/research institutions and social partners. EU-OSHA provided insights into national 
policymaking,  benefiting  social  partners  in  particular,  especially  during  the  COVID-19 
pandemic. EU-OSHA and Eurofound supported employers and trade union associations. The 
ETF supported education and training providers211. The evaluation suggested that the ETF could 
strengthen its collaboration with businesses and social partners. 

Other  stakeholders  benefited  from  the  Agencies’  support  less  often:  the  EU  public, 
businesses, EEA/ETFA national public authorities, other EU decentralised agencies and non-
governmental organisations (NGOs).  The public  and NGOs do not  directly benefit  from  the 
Agencies. However, considering the expending thematic scope and current budget constraints, 
the evaluation identified opportunities to involve more NGOs. NGOs could serve as multipliers 
to  increase  the  reach  and  dissemination  of  the  Agencies’  outputs  with  a  minimal  impact  on 
costs. 

208 See Table 4 in Annex VII – percentages do not include ‘do not know’ answers. 
209 These mixed views could be explained by the fact that, while support in the field of human capital development 
is not matched by any other national or European organisation in the EU,  at international level, the ETF’s 
work on education, training and skills is matched by other large organisations, such as UNESCO and other 
bilateral and international donors. 

210 As documented in the supporting study, Cedefop Case Study 1.  
211 One example of this ETF work is the virtual training courses organised by the ETF in 2021 (see  ETF Case 
Study 3), which benefited education providers, stakeholders working in education and the labour market, and 
partner countries officials. 

57 

 
 
 
Main sources of added value across the Agencies are described below.  

EUROFOUND. Eurofound provided strong added value through its pan-European surveys212. 
The surveys offer a unique and established source of comparative information on the quality of 
living and working conditions across the EU, providing EU-wide data that is consistent, reliable 
and  covers  a  wide  range  of  themes.  Eurofound’s  surveys  offer  much  more  detailed 
harmonised information that complements the yearly data that Eurostat produces in the 
Labour  Force  Survey.  The  EU-level  data  helped  to  understand  dynamics  that  were  not 
apparent at individual Member State level. The surveys inform and guide a substantial part of 
Eurofound’s analytical work. The methodology and questionnaires used in Eurofound’s pan-
European surveys are freely available for use by other researchers, subject to certain copyright 
conditions.  Moreover, they serve  as an inspiration for other surveys worldwide, such as the 
American Working Conditions Survey, the South Korea Working Conditions Survey, the Israel 
Social  Survey  and  the  China  Urban  Labour  Survey213.  The  United  Nations  Development 
Programme  (UNDP)  has  used  Eurofound’s  European  quality  of  life  survey  (EQLS) 
questionnaire to carry out surveys on the quality of life in Croatia and North Macedonia. 

Eurofound’s  representativeness  studies  were  important  for  social  dialogue  and  EU 
lawmaking214. As highlighted in the validation workshop for this study, the representativeness 
studies  were  crucial  for  ensuring  proper  social  dialogue  at  EU  level  and  facilitating  social 
dialogue at  national  level  (thus also  having  a national impact).  Eurofound’s methodology is 
reliable and extensive (having conducted these studies since 2006), and this has strengthened 
these  studies’  credibility.  While  these  studies  were  beneficial  for  social  partners,  providing 
visibility  and  influencing  national  agendas,  some  interviewees,  including  members  of  the 
executive and management boards, expressed concerns about the high cost of these studies and 
questioned their overall added value to stakeholders, especially at national level. In the final 
years of the current evaluation period, Eurofound increased its national focus as outlined in its 
2021-2024 work programme. To improve national adoption of its research, Eurofound built on 
its  Network  of  Eurofound  Correspondents  and  contributed  to  national  events  on  working 
conditions and pay. Eurofound’s work has been used more extensively at national level in recent 
years, particularly from 2020, demonstrating its use beyond EU level. 

CEDEFOP. Cedefop brought EU added value by providing evidence with unique thematic and 
European coverage. In the current evaluation period, Cedefop’s increased focus on supporting 
EU  policymaking  and  implementation  was  recognised  by  the  Commission  as  a  significant 
source of added value, in particular to support the 2020 European Skills Agenda and the 2020 
Council Recommendation on VET (see detailed evidence in Section 4.1.1.3). It has also added 
value by providing consistent definitions and concepts across Member States to facilitate the 

212 European Company Survey (ECS), European Working Conditions Survey (EWCS), European Quality of Life 

Survey (EQLS). 

213 Argentina, Chile and Uruguay and countries in Central America used EWCS questionnaires to carry out national 

surveys. 

214 Article 155 of the Treaty on the Functioning of the European Union (TFEU) provides the possibility to put 
social partner agreements into practice in EU law. However, this can only be done if the social partners who 
negotiated the agreement are considered as representative. This assessment is based on the representativeness 
studies.  Article  154  TFEU  obliges  the  Commission  to  consult  social  partners  in  certain  instances;  the 
representativeness studies are used to determine which social partners should be consulted. 

58 

 
 
 
 
implementation  of  agreed  EU  policies  (e.g.  apprenticeships,  transparency  tools,  upskilling 
pathways, individual learning accounts and micro-credentials). 

Cedefop adds value to Eurostat’s statistical data, complementing and contextualising it with 
quantitative  and  qualitative  information  obtained  from  surveys  and  ReferNet  partners215. 
Cedefop’s data have also provided added value for the EU migration network, and, beyond the 
employment domain, have been used to test different scenarios in the work of DG GROW, DG 
ENER and DG ENV. 

At national level, Cedefop’s high added value also stemmed from producing comparable EU-
level data, such as the VET opinion survey and  skills forecast, which national organisations 
often lack the capacity to produce. Additionally, during the current evaluation period, Cedefop 
developed  tools  and  repositories  accessible  across  Europe,  including  the  Financing  adult 
learning database, the European Skill Index (ESI) and the Skills Online Vacancy Analysis Tool 
for  Europe  (Skills  OVATE).  The  ex-post  evaluation  of  Cedefop’s  projects  and  tools216 
confirmed how unique they were, with no national or regional equivalents. The agency provided 
EU comparable data on VET, skills and qualifications to researchers and academics, which they 
could  not  have  accessed  otherwise.  Cedefop’s  2022  user  satisfaction  survey  highlighted  the 
frequent use of Cedefop output to strengthen knowledge, understand policy issues, trends or 
challenges, and support research/policy analysis.217  

ETF.  The  ETF’s  added  value  comes  from  its  distinctive  expertise  in  both  EU  and  partner 
countries’  policies,  a  breadth  of  knowledge  not  found  in  other  EU  institutions  or  partner 
countries. The agency served as a bridge, transferring experience and expertise from Member 
States  and  EU  institutions  to  partner  countries,  thereby  strengthening  the  capacity  of  those 
countries’ administrations. The ETF also added value to EU external policy through the advice 
it  provided  to  Commission  departments  drawing  on  its  expertise  in  neighbourhood  and 
enlargement countries. The ETF’s role in providing both EU and non-EU expertise, producing 
reports  and  sharing  methodologies  has  proven  beneficial  for  VET  and  TVET  providers  in 
partner  countries.  The  Agency’s  networks  of  VET  and  skills  policy  experts  (see  Box  5) 
provided readily available thematic and synthetised information that informed VET policy. This 
information would not have been available without the ETF218. The ETF works closely with 
other data organisations, including Eurostat, for example for the Western Balkans, Türkiye and 
the neighbourhood countries, to avoid any duplication. All data produced by the ETF on country 
intelligence are not available anywhere else. 

Box 5: Supporting education and training providers – the ETF’s networking activities 

VET providers considered the ETF as a primary source of information. The agency collaborates with 
VET centres and providers, sharing research methodologies, organising events with NGOs and business 
organisations and supporting implementation of reforms. The ETF Network of Excellence included an 

215 Coordination and complementarity are ensured through bilateral memoranda of understanding and participation 
in different forums, including Eurostat’s big data task force, the network of European statistical systems (ESS-
net) and the Web Intelligence Hub. 

216 Deloitte (2018), Ex-post evaluations of Cedefop activities. p. 10. 
217 PPMI (2022), Cedefop user satisfaction survey 2022. 
218  The  ETF’s  expertise  in  supporting  the  development  of  partner  countries’  VET  systems  is  recognised  by 
governmental organisations of EU Member States and other European and international organisations. 
59 

 
 
 
International Network of Centres of Vocational Excellence (CoVEs), involving over 250 organisations 
from  the  EU,  its  neighbouring  regions,  Central  Asia  and  Africa.  The  GRETA  initiative,  operating 
through  this  network,  supports  18  centres  of  vocational  excellence  in  eight  countries  in  their  green 
transition219.  

Additionally, the ETF’s unique work on Russia’s war of aggression against Ukraine (including 
its manuals to include refugees in the job market) has become a primary source of information. 
An example of uptake of the ETF work by the Commission and Member States is the Ukrainian 
Resource Hub, which has had more than 27 000 visitors since its launch220.  

EU-OSHA  was  particularly  valued  for  its  support  to  Member  States,  extensive  thematic 
coverage, and effective awareness-raising efforts, confirming the added value of EU-OSHA at 
national  level.  In addition, the agency  brings  added value to the Commission  by supporting 
OSH  policymaking  and  implementation.  The  agency’s  stakeholders  emphasised  the  added 
value of EU-OSHA’s OSH assessment tools, awareness campaigns and networking activities 
involving  focal  points,  including  the  ESENER  survey,  the  OiRA  tool  and  the  healthy 
workplaces campaigns (e.g. the 2019 campaign, which supported EU policymaking in OSH, 
see  Box  4  in  Section  4.1.1.3).    The  European  Parliament  study,  ‘Cost  of  Non-EU  Agencies 
Focusing on the Health and Safety Cluster of the EU Decentralised Agencies’221, highlighted 
that EU-OSHA provided significant support to Member States by bringing the EU perspective 
to the work of national health and safety agencies. Additionally, the evaluation demonstrated 
EU-OSHA’s  essential  role  in  conducting  awareness-raising  campaigns  on  OSH  at  the 
workplace,  addressing  gaps  at  the  national  level222  (case  study  on  EU-OSHA’s  healthy 
workplaces campaigns). In particular, EU-OSHA brought significant added value to countries 
with less advanced OSH systems223. There is potential to increase EU-OSHA’s impact in those 
countries if the required resources are available. 

Box 6: EU-OSHA European coverage of publications – ESENER survey 

EU-OSHA’s European survey of enterprises on new and emerging risks (ESENER) is recognised as the 
sole source of European information on OSH that is comparable and reliable across countries. It fills a 
gap, providing insights into compliance with OSH legislation and its effectiveness across all sectors224. 
National focal points’ representatives appreciate ESENER as a valuable tool for rethinking their OSH 
policies and encouraging businesses and workers to address specific risks. This shows that ESENER 
plays a crucial role in shaping OSH strategies in Member States, helping to prevent workplace accidents 
and illnesses.  

219 Covering the greening of curricula, teachers’ development and stakeholder engagement. 
220 For more info: Education and work information for Ukrainians and EU countries | ETF (europa.eu); Key policy 

developments in education, training and employment – Ukraine 2023 | ETF (europa.eu). 

221 EP (2021), ‘Cost of Non-EU Agencies Focusing on the Health and Safety Cluster of the EU Decentralised 

Agencies’.  

222 The evaluation of the campaign on dangerous substances and interviews with focal points show that, without 
the campaign, they would not have been able to carry out so many activities, and those activities carried out 
would not have been of such good quality. 

223 Such as Central and Eastern and European countries. 
224  The  2019  version  collected  data  from  over  45 000  establishments  in  33  countries  across  all  sectors  and 

consistently received high satisfaction ratings in the annual EU-OSHA survey. 

60 

 
 
 
4.3. 

IS THE INTERVENTION STILL RELEVANT? 

The Agencies’ mandates, objectives and activities are highly relevant. The Agencies contribute 
very  significantly  to  the  EU  policy  priorities.  They  also  address  their  stakeholders’  needs, 
although there is still room for improvement.  

4.3.1 CONTRIBUTION TO EU POLICY PRIORITIES 

The four Agencies developed their work programmes in response to EU policies and priorities. 
There was a clear link between EU priorities and the Agencies’ key programming and 
reporting documents – the Agencies’ work was often referred to in EU policy documents. The 
expertise of the Agencies’ staff was leveraged to inform policy and programme development. 
Table 8 in Annex VII shows Agencies’ outputs corresponding to the Commission’s 2019-2024 
strategic priorities. See also Annex IX for examples of the Agencies’ work that contributes to 
EU policy needs. 

EUROFOUND  closely  aligned  its  activities  with  DG  EMPL’s  three  main  areas  of  action: 
supporting the European Semester, fostering stronger social dialogue and promoting decent and 
safe working conditions for all225.  

EU-OSHA successfully contributed to the EU’s OSH legislation and strategies. Tools for OSH 
management, especially OiRA, have contributed to EU policy needs by promoting a culture of 
prevention, particularly in small and medium-sized companies. 

CEDEFOP closely aligned its work with EU upskilling and reskilling policies, in particular 
the European Skills Agenda, the Council recommendations on VET, the Osnabrück Declaration 
and the Digital Education action plan226.  

The  ETF  closely  aligned  its  work  with  EU  policy  priorities.  The  agency  also  significantly 
contributed to the EU’s external relations policies and programmes. While the ETF’s work is 
highly  relevant,  its  close  relations  with  different  Commission  departments  led  to  competing 
expectations about the geographical scope and focus of its activities (e.g. focusing on INTPA 
countries versus neighbourhood/candidate countries). Consequently, there is a need to clarify 
the agency’s long-term vision and which of its activities should be prioritised. 

4.3.2 CONTRIBUTION TO ADDRESSING STAKEHOLDERS’ NEEDS 

The Agencies’ priorities, mandates and activities addressed their stakeholders’ needs. The 
Agencies were able to monitor and adjust to  them thanks to their respective monitoring and 
feedback systems (dedicated user surveys227 or participatory activities228). 

Over  90%  of  EU-OSHA  stakeholders  and  over  75%  of  ETF,  Cedefop  and  Eurofound 
stakeholders considered that the Agencies met at least partially their needs (evaluation feedback 
survey – see Figure 3). Analysis by stakeholder group (to be treated with caution as the sample 
is relatively small) shows that over 80% of trade unions and consultancy organisations found 

225 DG EMPL (2020), ‘Annual Activity Report 2019’.  
226 European Commission (2018), Communication on the Digital Education Action Plan. 
227 Used by Cedefop, Eurofound, and EU-OSHA. 
228 Used by ETF through the Torino Process. 

61 

 
 
 
Eurofound’s activities and output very relevant to their work. This figure was around 45% for 
the EU institutions and national governments.  

On Cedefop, 75% of trade union respondents considered the Agency’s activities and outputs to 
be very relevant, compared to 42% of employer institutions and VET providers. A 2022 user 
satisfaction survey suggests that Cedefop output related to forecasts and future trends are the 
most valuable229. Nonetheless, a limited number of trade union representatives argued that the 
agency should not focus as much on micro-credentials and individual learning outcomes. Other 
stakeholders indicated that the agency could respond more quickly to emerging trends and have 
a greater focus at regional level. 

Figure  4:  Stakeholder  survey  responses  on  the  extent  to  which  the  Agencies'  services 
responded to their needs 

Source: Stakeholder survey.  

Regarding  the  ETF,  75%  of  research  or  consultancy  organisations  considered  the  agency’s 
outputs  and  activities  most  relevant,  compared  to  about  50%  of  VET  providers  and  EU 
institutions. A Policy Delphi review from 2018 found that partner country stakeholders found 
the  ETF’s  outputs  related  to  VET  governance,  the  modernisation  of  VET  systems,  the 
facilitation of labour market reforms, and support for more effective skills and employability 
policies to be highly useful230. Moreover, the agency received 100% positive feedback on its 
support to EU services from 2017 to 2020. 

Taken  together,  the  survey,  public  consultation  and  other  evaluation  feedback  suggests  that 
greater efforts are needed to increase the visibility and uptake of ETF resources at national level 
231. 

Figure  4  above  shows  that  EU-OSHA  had  the  lowest  proportion  of  stakeholder  survey 
respondents who indicated that the agency’s activities and outputs were only slightly or not at 
all relevant to their work. Stakeholders highlighted the agency’s outputs such as ESENER-2 

229 PPMI (2022), User satisfaction survey 2022. 
230 PPMI (2018), Evaluation of the ETF Functions: Policy Delphi Results 1st round: June/July 2018. 
231 For examples, see: PPMI (2018), Evaluation of the ETF Functions: Policy Delphi Results 1st round: June/July 

2018. and PPMI (2022), Evaluation of the ETF Monitoring and Assessment. 

62 

 
 
 
data (which was used by research organisations for studies232), the healthy workplaces campaign 
(which enriched national policy debate on OSH233) and activities on strategic and operational 
networking (which was rated very highly in a recent user feedback survey234). While the OiRA 
tool for OSH management has been widely used by EU-OSHA’s stakeholders, there is room to 
improve the tool by making the layout more appealing235. 

Stakeholders  and  staff  from  Cedefop,  EU-OSHA  and  Eurofound  pointed  out  that  different 
stakeholder groups may perceive relevance differently; therefore, meeting all stakeholder needs 
can be a challenge. Nevertheless,  Cedefop,  EU-OSHA and Eurofound all balanced different 
stakeholders’  needs  effectively.  In  particular,  for  EU-OSHA,  the  tripartite  structure  of  the 
management board was seen as fostering a strong level of ownership among stakeholders that 
contributed to the effective delivery of the agency’s work programme236.  

Figure 5: Stakeholder survey responses on the extent to which they were able to provide 
input on the Agencies’ priorities, activities and/or outputs 

To what extent, if at all, did you feel that you were able to inform Agency’s 
priorities, activities and/or outputs during the period 2017-2022?

ETF (n=91)

24%

32%

13%

8%

14%

9%

Cedefop (n=174)

22%

30%

14%

9%

13%

12%

EU-OSHA (n=75)

41%

33%

11%

8%

5%

Eurofound (n=65)

35%

34%

6%

14%

8%

3%

1%

0%

20%

40%

To a large extent
To a small extent
Do not know / cannot answer

60%

80%

100%

To a moderate extent
Not at all
Not applicable

Source: Stakeholder survey. 

While the Agencies’ activities and outputs were largely aligned with stakeholder needs, more 
opportunities could be given to stakeholders to provide feedback and shape the Agencies’ 
approaches.  Figure  5  shows  that,  for  both  EU-OSHA  and  Eurofound,  over  two  thirds  of 
stakeholders indicated they were able to shape the Agencies’ priorities to a large or moderate 
extent. However, a lower proportion of ETF and Cedefop stakeholders indicated the same level 
of influence. The ETF is the only one of the four Agencies that does not regularly conduct user 
satisfaction surveys. 

Finally, the COVID-19 pandemic and the war of aggression against Ukraine have not changed 
the relevance of the Agencies’ mandates or work. Each agency adapted its work programme 
where relevant and feasible to respond to the two crises by producing additional targeted output. 

232 Iñigo I. and Corral A. (IKEI) (2018), ‘Ex-post evaluation of the Second European Survey of Enterprises on new 

and emerging Risks (ESENER-2)’. 

233 Ipsos (2020), ‘Evaluation of the Healthy Workplaces Campaign 2018-2019’. 
234 EU-OSHA (2022), Bi-annual stakeholder survey 2022. 
235 EU-OSHA (2023), Summary - Risk assessment using OiRA at French workplaces: a qualitative study.  
236 Based on contributions at the study’s final validation workshop. 

63 

 
 
 
Looking ahead, the Agencies are already working on highly relevant future trends, such as 
climate  change  and  digitalisation,  which  stakeholders  agree  as  important  areas  of  focus. 
However, stakeholders also highlighted demographic change as a fundamental area of work, 
given  Europe’s  ageing  population  (see  Figure  13,  Annex  VII).  The  Agencies’  mandates  are 
already  broad  enough  to  accommodate  these  megatrends  and  the  work  programmes  already 
include them to stay aligned with EU policy needs (see Section 4.1.1.4). Therefore, there is no 
need to modify the Agencies’ mandates or intervention logics. 

5.  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? 

5.1.  CONCLUSIONS  

5.1.1. EFFECTIVENESS 

Based on the available evidence, the four Agencies operated effectively during the 2017-2022 
evaluation  period.  Overall,  the  Agencies  achieved  many  of  their  general,  specific  and 
operational  objectives  to  a  high  degree,  as  was  the  case  during  the  previous  2011-2016 
evaluation period. There is however room for improvement in certain areas (see Section 5.2. on 
lessons-learned) 

Operational objectives (activities) 

The four Agencies completed the activities and outputs set out in their work programmes 
to a high level, despite budget  and staff limitations and the COVID-19  pandemic. Delivery 
against key performance targets was good. 

While the Agencies have improved their communication and dissemination activities (including 
increased social media use and presence) the evaluation highlighted scope for improvement in 
that field. 

Specific objectives (results)  

The four Agencies achieved to a great extent  the expected result of providing EU (and  ETF 
Partner  Countries’)  policymakers  and  stakeholders  with  high-quality  information  to  support 
relevant policies. The evidence suggests that the quality of the Agencies’ outputs, as well as 
the use of these outputs by stakeholders, was high. 

 Use  of  outputs  -  The  Agencies’  KPI  as  well  as  staff  and  stakeholder  consultations 
indicate that each of the Agencies’ services and outputs have been used by their key 
stakeholders.  Use  has  increased  compared  to  the  previous  evaluation  period  and  is 
more  widespread  across  the  EU  institutions  compared  to  international  or  national 
stakeholders. 

Quality of outputs – The user satisfaction of the Agencies’ outputs was very high (and 
has even increased for Cedefop compared with the previous period).  

For each Agency, the outputs that were most used and of the highest quality were:  

Eurofound - research related to COVID-19 and on working and living conditions, the 
three pan-European surveys (as in the previous period), and work on minimum wages;  

64 

 
 
 
Cedefop  -  skills  anticipation  and  matching,  monitoring  and  analysing  VET  policy 
developments and promoting access to and the VET’s appeal;  

ETF – predicting skills demand, monitoring and diagnostics reports, innovative teaching 
and learning and vocational excellence provision models;  

EU-OSHA – the OiRA tool and the healthy workplaces campaigns. 

However,  other  results  were  achieved  to  a  lesser  extent,  particularly  for  the  ETF.  Unlike 
agencies operating under EU boundaries, support to partner countries is naturally limited by 
external factors beyond the ETF’s control, such as insecurity, political instability, and economic 
volatility.  The  ETF’s  achievement  of  results  depends,  to  some  extent,  on  the  willingness  of 
partner countries to engage with the support offered by the ETF. 

Impacts: achievement of the general objectives 

Each  Agency  achieved  its  general  objectives,  by  positively  contributing  to  EU  policy 
making and implementation in the policy areas established in its mandate. Overall, the 
activities  and  outputs  of  the  Agencies  were  aligned  to  the  Commission’s  priorities  and 
adapted well to changes in EU policies (as in the previous evaluation period). 

Examples of the most significant impacts include:  

Eurofound:  the  work  on  digitalisation  and  new  and  non-standard  forms  of  work 
(including  the  platform  economy)  helped  the  Commission  and  other  EU  bodies  to 
underpin the EU legislative initiative on platform work.  

Cedefop:  the  evidence  and  data  gathered  by  Cedefop  contributed  to  the  renewed 
European Skills Agenda, the first Council Recommendation on VET and the Osnabrück 
Declaration. Cedefop has therefore contributed to improving policies on VET, skills and 
qualifications.  

EU-OSHA made a significant contribution to the EU’s strategic frameworks on health 
and safety at work. It strongly supported policymaking with its work on COVID-19 and 
two healthy workplace campaigns.  

ETF  supported  policymaking  by  improving  the  adaptation  of  VET  governance  to 
evolving skills demand and labour market transitions. Its work also helped to improve 
labour market policies and practices and informed EU external relations policy. 

In  addition,  the  Agencies  generated  wider  impacts  and  demonstrated  adaptability  by 
responding, within the scope of their mandates, to unexpected events such as the COVID-
19 pandemic and Russia's war of aggression against Ukraine.  

Eurofound leveraged its longstanding work on structural changes in policy debates on 
how the COVID-19 pandemic was accelerating change in the workplace.  

Cedefop supported the integration of Ukrainian refugees into VET.  

The ETF played a key role in facilitating Commission support to Ukraine in response 
to  Russia’s war of aggression. The  ETF’s longstanding presence in  and  expertise on 
Eastern Partnership countries prepared the Agency for this unexpected role.  

65 

 
 
EU-OSHA’s healthy workplace campaigns contributed to improved social dialogue and 
cooperation among the national governments and social partners. 

5.1.2. EFFICIENCY 

The available evidence shows that overall, the four Agencies were cost-effective. This was 
the case also in the previous evaluation. They lowered costs through a variety of measures while 
maintaining the quality of their outputs.  The cost-effectiveness  analysis of the key activities 
concluded that, for most of them, there was no scope to reduce costs without compromising 
quality. Stakeholders continued to highly rate the Agencies'  expertise and outputs and had a 
generally positive view of the Agencies’ cost-effectiveness. 

During the evaluation period, the Agencies took a variety of cost-saving measures to boost 
cost-effectiveness. For instance, joint procurement and sharing accounting services, identifying 
deliverables where costs could be shared (e.g. the joint Eurofound-Cedefop European company 
survey), and investing in actions that lead to long-term savings (e.g. increasing office energy 
efficiency, dissemination of digital rather than print outputs). The rise in digital usage during 
the COVID-19 pandemic also contributed to cost savings.  

However, the assessment of administrative costs, staff allocation, and monitoring and reporting 
systems  shows  that  the  four  Agencies  have  the  potential  to  reduce  the  administrative 
burden and simplify processes even further. In this context, it should be noted that, during 
the reporting period, a higher proportion of administrative staff than two other Agencies – the 
EU  Agency  for  Fundamental  Rights  (FRA)  and  the  European  Institute  for  Gender  Equality 
(EIGE).  

Monitoring and reporting systems 

During the evaluation period, the four Agencies have developed internal control frameworks to 
monitor,  report  and  evaluate  their  activities.  These  frameworks  cover  financial,  human 
resources, and core business performance. They allow for a broad assessment of how resources 
are spent and translated into the planned outputs.  

Despite these positive developments,  the evaluation highlighted that the Agencies should 
continue to work on their monitoring and reporting systems. There is room to: (i) improve 
their  monitoring  systems  and  KPIs  to  better  define  targets  and  assess  the  scale  of 
success/achievement of results; and (ii) simplify reporting. Such simplifications would help 
to reduce further the workload of administrative staff while maintaining the Agencies’ capacity 
to ensure accountability and assess performance. 

Agencies’ governance 

The tripartite governance of Eurofound, Cedefop, and EU-OSHA offers significant advantages, 
such as representation,  strategic direction, and  knowledge-sharing facilitated by government 
representatives  and  social  partners.  However,  a  primary  challenge  comes  from  the  size  of 
management boards and the number of voting members who must navigate compromises not 
only  on  the  Agencies’  core  businesses  but  also  on  administrative  decisions,  budgets,  and 
infrastructure. 

66 

 
 
The evaluation explored alternative governance models with different ways to involve the 
social partners. Although expenditures on the larger boards are higher, they can be reduced to 
some extent through hybrid meetings. However, the size of tripartite management boards and 
the  diversity  of  participating  stakeholders  raises  questions  about  how  to  ensure  that  the 
discussions remain focused, and how to efficiently use stakeholder expertise in board meetings 
given that operations, budget, and administration can overshadow core business topics. While 
stakeholder  expertise  is  an  asset  that  contributes  to  the  Agencies’  work  and  cannot  be 
compromised  for  the  sake  of  financial  efficiency,  other  models  may  allow  involving 
stakeholders in an equally effective, but more efficient manner. 

5.1.3 COHERENCE 

Coherence among the Agencies 

The four Agencies cooperated more actively and intensively than before 2017. Cooperation 
among the Agencies moved beyond passive information sharing and took place: (i) in a variety 
of forms (joint research, tool development, events and publications); (ii) on a variety of topics 
(labour markets, vocational education and training, skills, qualifications, health and safety at 
work, and the working environment); and (iii) at several levels (board level, project / initiative 
level, administrative functions, such as HR/finance). This cooperation led to a higher degree 
of coherence among the Agencies. 

The assessment of the coherence across the four Agencies’ mandates shows that there are some 
overlaps in the thematic areas of the mandates (e.g. Cedefop and ETF, both dealing with VET; 
Eurofound,  Cedefop  and  ETF  on  skills  mismatches).  However,  these  overlaps  do  not 
materialise in activity or output duplications because the Agencies approach these topics from 
different angles based on their expertise. As a result, the Agencies have complemented rather 
that  duplicated  their  work.  The  Agencies  have  intensified  inter-agency  cooperation, 
producing  more  joint  outputs  based  on  the  common  thematic  areas.  In  fact,  the  current 
evaluation has not found any example of duplication of activities.  

Regarding the potential for mergers between the four Agencies, and based on the cross-cutting 
analysis  of  the  four  Agencies’  mandates  and  activities,  as  well  as  the  fact  that  the  current 
Agencies  mandates  have  remained  unchanged,  the  evaluation  has  not  found  evidence  to 
change the conclusions from the previous 2019 evaluation. The 2019 evaluation concluded 
that: (i) merger options show difficulties in balancing positive and negative effects; and 
(ii) efficiency can be improved through better cooperation. 

Concerning a potential merger of the European Labour Authority (ELA) with any of the other 
four  EMPL  Agencies,  the  ELA’s  highly  specialised  mission  and  its  very  distinct  types  of 
activities do not suggest significant overlaps with the mandates of other agencies.  

Coherence between the Agencies and other stakeholders  

The four Agencies’ mandates and activities continued to be coherent with DG EMPL and 
EU  policies.  The  mandates  and  work  of  the  Agencies  were  in  line  with  the  Commission’s 
strategic priorities for 2019 to 24. The thematic areas covered by the Agencies were coherent 
with DG EMPL’s policies and the policies of other relevant Commission departments.  

67 

 
 
 
Cooperation  between  the  Agencies  and  DG  EMPL  was  good  even  though  there  remains 
scope for even stronger cooperation. DG EMPL participated in each Agency’s Management / 
Governing Board and commented on the Agencies work programmes. Day-to-day coordination 
also took place at operational level.  

The four Agencies continued to cooperate  with  other relevant EU decentralised agencies, in 
particular,  through  the  EU  Agencies  Network  (EUAN).  The  four  Agencies  also  intensified 
cooperation  with  international  organisations,  such  as  the  ILO,  the  OECD,  UNESCO. 
Cooperation with the recently established ELA has begun.  

5.1.4. EU ADDED VALUE 

The four Agencies generated EU added value across a range of areas, as was the case in 
the previous evaluation. Stakeholders particularly appreciated the EU added value in relation 
to the “quality of information produced, specific thematic knowledge not available elsewhere 
and European coverage of the publications, surveys and data, and a focus on EU policy needs”. 

The degree of EU added value varied according to each Agency’s specific mandate and area of 
work.  Eurofound  ranked  highly  in  terms  of  European  coverage  /  EU  level  comparability. 
Cedefop performed well in terms of quality of information produced. The ETF ranked highest 
in terms of representing the Commission’s external dimension in partner countries. EU-OSHA 
ranked particularly highly in terms of awareness raising.  

Other examples of EU added value provided by the Agencies are:  

- for Eurofound, a key area of EU added value are its surveys, providing unique insights into 
key living and working topics on a comparable basis. Eurofound scored highly in terms of its 
EU-wide comparable data which is reliable and has continuity over the years, and its thematic 
coverage. 

- for Cedefop, added value arose from publications, surveys, and data with a unique thematic 
and European coverage, including the work on skills anticipation and forecasting. Also, for the 
tools and repositories shared across the EU, such as the financing adult learning database, the 
European Skill Index (ESI) and the Skills Online Vacancy Analysis Tool for Europe (Skills 
OVATE).  

- for the ETF, its EU added value arose from its specific expertise in policies of both the EU 
and partner countries, which does not exist at such a scale within other EU institutions or partner 
countries. Stakeholders also praised the added value of the ETF’s work in respect of Russia’s 
war of aggression against Ukraine. 

-  EU-OSHA  was  valued  particularly  highly  for  its  support  to  Member  States  and  thematic 
coverage that adds a European dimension to the work of national health and safety agencies. 
This  added value was particularly  high in  countries with  less developed OSH systems.  Key 
tools  that  provided  EU  added  value  include  the  ESENER  survey,  the  healthy  workplaces 
campaigns and the OiRA.  

Overall, most stakeholders indicated that other organisations could only substitute to a small 
extent or not at all the Agencies’ services, and that ending the Agencies would affect their work. 

68 

 
 
 
This suggests the Agencies generate strong EU added value as they make a distinct contribution 
that cannot be replaced.  

5.1.5 RELEVANCE 

The  activities,  objectives  and  mandates  of  the  Agencies  were  highly  relevant  and 
responded to both EU policy and stakeholders’ needs. EU policies and priority areas were 
reflected in key programming documents. The work of the Agencies was often referred to in 
EU and other policy documents. Feedback from stakeholders also suggests that the Agencies 
responded to their needs. 

No issues were identified which would put into question the future relevance of the mandates 
or work of the Agencies. The Agencies already work on highly relevant future trends, such as 
climate change and digitalisation. The Agencies’ stakeholders agree with the policy focus of 
the Agencies, but also pointed out that the ‘mega-trend’ of an ageing population could receive 
greater attention from the Agencies.  

The  Agencies  demonstrated  adaptability  by  responding  to  the  COVID-19  pandemic  and 
Russia’s war of aggression against Ukraine with additional targeted deliverables where relevant 
and feasible. 

The Agencies responded promptly and transparently to audit recommendations. 

5.2.  LESSONS-LEARNED 

5.2.1. GENERAL LESSONS-LEARNED FOR ALL AGENCIES  

Effectiveness 

1.  Overall,  the  use  of  and  the  quality  of  the  four  Agencies’  outputs  is  high.  However, 
communication  and  dissemination  could  be  further  improved  to  better  reach  target 
audiences. The agencies could: (i) produce more ‘tailored’ outputs; and (ii) consider dedicating 
an appropriate level of budget and focus to the dissemination and communication activities (see 
Section 4.1.1.1.). 

2. The four Agencies could explore how to better engage with the national level, developing 
a  more  targeted  approach  to  awareness-raising  and  dissemination  to  better  reach  target 
audiences  in  Member  States  and  the  ETF  partner  countries  (see  Section  4.1.1.1.  and  Annex 
VIII). 

3. The four Agencies contribute well to relevant EU political priorities and have incorporated 
the key EU themes into their work. The Agencies could continue to ensure that: (i) they address 
key themes such as green and digital transitions; and (ii) outputs related to key themes 
are properly disseminated  so that stakeholders  are  aware of the  Agencies’ efforts in  these 
areas (see Section 4.1.1.5.). 

Efficiency 

69 

 
 
 
4. There may be scope for the four Agencies to further reduce their administrative burden 
and  simplify  processes.  The  Agencies  could:  (i)  examine  how  to  further  reduce  their 
administrative  burden  on  staff  to  maintain  sustainable  workload  (e.g.  further  scope  for 
cooperation and synergies between the Agencies); and (ii) reduce the share of staff engaged in 
administrative activities and increase the share of staff focused on the Agencies’ operations (see 
Section 4.1.2.). 

5.  The  four  Agencies  could  consider  cutting  costs  and  increasing  cost-effectiveness  by 
organising  more  hybrid  or  online  meetings  where  possible,  and  without  compromising  the 
Agencies'  outreach/presence  among  stakeholders.  The  Agencies  could  also  further  prioritise 
activities  and  outputs,  using  transparent  criteria  agreed  with  their  governance  bodies  (see 
Section 4.1.2.4.). 

6. Eurofound, Cedefop and EU-OSHA could consider how to address the balance between 
research  needs  and  budgetary  restrictions.  They  could  explore  additional  funding 
possibilities, as was successfully by the ETF while remaining within the scope of their mandates 
(see Section 3.1.1.). 

7.  The  governance  of  the  tripartite  Agencies  offers  significant  advantages,  such  as 
representation, strategic direction, and knowledge-sharing facilitated by government and social 
partner representatives from all Member States. At the same time, due to their larger size, the 
Agencies’ management boards have relatively high operating costs, meetings tend to be long 
and complex to manage, decision-making can be challenging, and the overall management can 
be burdensome.  

Where deemed necessary, a case-by-case exploration of the governance arrangements could 
be carried out while giving due consideration to the importance of social dialogue and the role 
of social partners in tripartite agencies. In this context, the future results from the ongoing ELA 
evaluation could be considered, as relevant, taking also into account the different nature of the 
respective governance arrangements (see Section 4.1.2.5.). 

8.  While  the  four  Agencies’  control  mechanisms  are  effective  and  overall  efficient,  the 
Agencies  could  review  and  significantly  improve  their  monitoring  systems,  including 
KPIs,  to  properly  assess  the  scale  of  success  of  their  interventions,  and  to  make  the 
evaluations more useful in improving the Agencies’ performance. This could be achieved by: 
(i)  defining  SMART  objectives;  (ii)  identifying  meaningful  KPIs  to  measure  result 
achievement; (iii) systematically setting targets for all KPIs; (iv) filling data gaps (v) further 
harmonising and aligning indicators across the Agencies; and (vi) better monitoring of services 
/ outputs use at national level. Moreover, the Agencies could further streamline monitoring 
processes and tools (streamlining of reporting) (see Section 1.1.2. and 4.1.2.3.). 

Coherence 

70 

 
 
  
9. The Agencies could continue to explore shared or central services, where this adds value. 
This covers functions such as HR, legal and IT, and more specialist and technical roles (such 
as cyber security, employment statistics, and labour market economics)  (see Section 4.1.2.4. 
and 4.1.3.2.). 

10. The Agencies could explore shared office space, using the Brussels Liaison Office, to 
contribute to both coherence and efficiency (see Section 4.1.3.2.). 

11.  The  Agencies  could  continue  to  explore  with  the  Commission  those  areas  in  which 
working together brings most added value to quality of outputs and stakeholders’ use. These 
include  cooperation  between  Eurofound  and  Cedefop  on  the  European  company  survey, 
Cedefop  and  the  ETF  on  VET  policy  monitoring,  and  Eurofound  and  EU-OSHA  at  the 
intersection of working environment, working conditions and occupational health and safety, 
including psychosocial risks and a focus on climate and environmental health issues. (Section 
4.1.2.4. and 4.1.3.2.). 

12. Better coordination could be needed between Cedefop, the ELA and Eurofound to avoid 
duplication and ensure completeness and consistency  in areas concerning skills forecast and 
labour  shortages,  while  joint  products  in  this  domain  should  be  considered  (see  Section 
4.1.3.1.). 

EU added value 

13. The Agencies’ visibility could be strengthened, notably vis-à-vis EU citizens, businesses 
at  national  level,  other  EU  decentralised  agencies,  and  NGOs.  The  Agencies  could,  where 
resources permit, reach out to relevant NGOs as potential multipliers to increase the reach and 
dissemination of the Agencies’ outputs (see Section 4.2.2.2.). 

Relevance 

14. The Agencies already work on ‘megatrends’ such as climate change and digitalisation. The 
Agencies  could  also  provide  added  value  by  also  considering  the  implications  of  an  ageing 
population (see Section 4.3.2.). 

5.2.2.  LESSONS-LEARNED FOR INDIVIDUAL AGENCIES  

5.2.2.1 EUROFOUND 

15. While general satisfaction with Eurofound outputs is high, the Agency could add value for 
stakeholders by: (i) developing its digital content and making its products more visual (country 
comparisons,  survey  research,  policy  evaluations,  qualitative  research,  datasets);  and  (ii) 
providing more in-depth outputs, with more methodological explanation to help stakeholders 
interpret findings (see Section 4.1.1.1.). 

16. Evidence from external and internal consultations suggests Eurofound could try to increase 
stakeholder participation in its services and activities (see Section 4.1.1.1.). 

71 

 
 
17. Training could be further developed by producing formal guidance and a more structured 
approach that supports ongoing learning for staff (see Section 4.1.1.2.). 

18.  In  a  context  of  limited  operational  budgets,  Eurofound  could  continue  to  reflect  on  the 
cost/quality trade-off between face-to-face and CATI / online surveys methods. Any decision 
about carrying out these surveys online (for cost-efficiency reasons) needs to ensure that these 
surveys continue to provide high quality and representative data (see Section 4.1.1.1.). 

19.  Eurofound  could,  together  with  the  Commission  and  the  social  partners,  review  the 
approach to conducting representativeness studies,  as also mentioned in the 2023 Commission 
Communication on strengthening social dialogue (see Section 4.2.1.). 

5.2.2.2. CEDEFOP 

20. Cedefop could explore how to increase user numbers, for example in academia and with 
VET providers (see Section 4.1.1.1.). 

21.  Cedefop  working  arrangements  with  the  management  board  could  benefit  from  being 
reviewed  to  increase  efficiency  and  effectiveness,  including  a  clearer  orientation  of  agenda 
items towards policy topics rather than administrative matters (see Section 4.1.2.5.).  

22. Cedefop could cooperate and align with other agencies on KPIs and introduce a KPI on a 
quantitative work programme delivery that would be reported on in agency annual reports (see 
Section 1.1.2. pages 3-4 and Section 4.1.2.3.). 

23. Evidence from stakeholder survey suggests that Cedefop could  consider providing more 
possibilities for stakeholders, outside the management board, to provide input on its activities 
(e.g. VET providers), and explore how to improve the extent to which its services correspond 
to the needs of its stakeholders (see Section 4.3.2.).  

5.2.2.3. ETF 

24. The ETF  could reflect on how to support  partner countries that do not participate in the 
voluntary strand of the Torino Process (see Section 1.1.1.1.). 

25. The ETF could work to improve the gender balance among its staff (see Section 4.1.2.2.). 

26. The ETF could investigate how to improve internal monitoring, especially monitoring and 
evaluating, overcoming data gaps, and linking specific targets to reporting data (see Sections 
1.1.2. and 4.1.2.3.). 

27.  The  ETF  already  cooperates  with  several  relevant  international  organisations.  It  could 
proactively  develop  cooperation  with  international  donors  and  international  financial 
institutions  as  international  human  capital  development  actors  are  diversifying  (see  Section 
4.1.3.5). 

72 

 
 
28. The  ETF is  well embedded in  EU policy structures through structured dialogue with  its 
partner Commission departments (DGs EMPL, INTPA and NEAR in particular). Despite these 
efforts, challenges remain as the latter have competing priorities and varied visions for the ETF. 
The  ETF  could  develop  a  joint  long-term  vision  together  with  DG  EMPL  and  its  partner 
Commission departments to focus its efforts and resources on the areas of greatest impact and 
need (see Section 4.1.3.3.). 

29. In its work with partner countries, the ETF could further cooperate with EIGE on gender 
equality, and with ELA on digital work and job mobility (see Sections 4.1.3.4. and 4.1.3.1.). 

30. Efforts to  tailor national  activities could continue,  through sensitivity  to  partner  country 
needs, awareness of the policy cycle and maintaining regular contacts (see Section 4.1.1.1.). 

31.  While  the  ETF’s  activities  provide  significant  EU  added  value  to  most  stakeholders, 
cooperation  with  businesses  and  social  partners  could  be  further  strengthened  (see  Section 
4.2.2.2.). 

32. The ETF could work on improving the relevance of its outputs for stakeholder in the areas 
of: (i) capacity building of teachers and trainers and support for work-based learning; and (ii) 
specific partner country needs (see Section 4.1.1.1.). 

33. The ETF could provide more opportunities for stakeholders, in particular partner countries, 
to provide suggestions and feedback on its services including through a user satisfaction survey 
(see Section 4.3.2.).  

5.2.2.4. EU-OSHA 

34. EU-OSHA could consider how to adapt its outputs to target audience needs and increase 
reader numbers. A distinction could be made between outputs destined for workplaces and those 
for  OSH  professionals/researchers,  as  workplaces  often  require  short  documents  with 
infographics or visuals summarising key information (see Section 4.1.1.1.). 

35. EU-OSHA could review how well its activities meet stakeholder needs on the topics of an 
ageing society and the green and digital transitions (see Section 4.1.1.1.). 

36. While EU-OSHA impact on supporting health and safety policies is overall very good, it 
could  further  increase  its  impact  by  exploring  how  to  better  support  countries  that  are  less 
advanced in OSH. Such efforts may depend on available resources (see Section 4.2.2.1.). 

37. EU-OSHA has already improved the translation quality to increase the usability of its work. 
Strict  quality  assurance  mechanisms  for  the  original  English  text  could  be  considered  to 
increase cost-effectiveness (see Section 4.1.1.1.). 

38. EU-OSHA could improve the cost-effectiveness of procurement and contracting procedures 
for instance by diversifying, when possible, the contractor base through better dissemination 
and publicity (see Section 4.1.2.4.). 

73 

 
 
39.  EU-OSHA  is  very  well  embedded  in  the  broader  EU  policy  governance  structure  and 
cooperation with the relevant Commission departments greatly intensified during and after the 
COVID-19  pandemic.   EU-OSHA  could  however,  depending  on  the  available  resources, 
consider  developing  cooperation  with  additional  Commission  departments237  (e.g.  on  SME 
policies, with DG JUST, DG FISMA, DG GROW, and with EASME, the executive Agency 
for  SMEs).  EU-OSHA  could  also  increase  efforts  to  disseminate  research  findings  in  the 
Commission’s  publications  and  increase  the  visibility  of  its  work  in  DG  GROW’s  Europe 
Enterprise Network (see Section 4.1.3.3). 

40. EU-OSHA already cooperates well with the other three Agencies covered by the evaluation. 
Stronger cooperation could take place on areas such as the green and digital transitions, climate 
change, and foresight studies. Cooperation with the ELA has the potential to be strengthened, 
notably in relation to OSH implementation and workplace working conditions. The Framework 
for cooperation between EU-OSHA and ELA signed by these Agencies in May 2023 is a good 
example of a realistic approach to how cooperation could be strengthened (see Section 4.1.3.1.). 

74 

 
 
 
 
 
ANNEX I:   PROCEDURAL INFORMATION 

Lead DG, Decide Planning/CWP references 

The  evaluation  was  led  by  the  Directorate-General  for  Employment,  Inclusion  and  Social 
Affairs, with a Decide planning reference PLAN/2022/487. 

Organisation and timing 
An inter-service steering group (ISG) was set up, coordinated by the Directorate-General for 
Employment, Inclusion and Social Affairs (represented by staff from the Better Regulation unit, 
the four lead units in charge of the Agencies, and the coordination unit) and including members 
from  Eurostat,  the  Secretariat-General,  and  the  Directorate-Generals  for  Education,  Youth, 
Sport  and  Culture  (EAC);  International  Partnerships  (INTPA);  Budget  (BUDG);  Human 
Resources  and  Security  (HR);  Neighbourhood  &  Enlargement  Negotiations  (NEAR); 
Migration and Home Affairs (HOME); Justice and Consumers (JUST); Health and Food Safety 
(SANTE); and the Joint Research Centre (JRC). 

The ISG also acted as steering group for the external study supporting the evaluation, which 
was carried out by Ecorys. Contract VC/2022/0294 was awarded after reopening of competition 
(tender  VT/2022/015)  within  the  multiple  framework  contract  VC/2021/0336).  The  contract 
VC/2021/0381  started  on  17 October 2022.  The  draft  final  report  of  the  external  study  was 
received  on  21 August 2023,  the  inter-service  steering  group  provided  comments  by 
20 September 2023 and the contractor delivered the revised final report on 12 January 2024, 
along with the agreed annexes (including an annex on the stakeholders’ consultation).  

The call for evidence published on 2 March 2023 explained the context, purpose and scope of 
the evaluation and informed stakeholders that an external evaluation study was being carried 
out, supported by a public consultation. The public consultation was open for 12 weeks from 
2 March 2023 to 25 May 2023. The factual summary was published on 6 July 2023. Targeted 
consultations were carried out mainly within Q1 and Q2-2023. See Annex V for further details 
on the stakeholder consultations carried out. 

Exceptions to the better regulation guidelines 
All Better Regulation requirements were fulfilled.  

Consultation of the RSB (if applicable) 
The evaluation was selected for the scrutiny of the Regulatory Scrutiny Board. The evaluation 
staff working  document  was discussed at  the meeting on 28 February 2024.  The Regulatory 
Scrutiny Board issued a positive opinion with reservations on 29 February 2024. Table 2 below 
shows how this report took into account the RSB comments before launching the inter-service 
consultation. 

Evidence, sources and quality 

75 

 
 
 
 
 
 
 
 
The evaluation was based on the evidence collection and analysis carried out by external experts 
(contract with Ecorys to carry out the supporting study, see above). 
Annex  II  provides  additional  information  on  the  methodological  approach  taken.  To  ensure 
quality, the ISG provided feedback throughout all steps of the evaluation process, and the four 
Agencies evaluation staff was consulted on each study deliverable to ensure accuracy of the 
factual information and data.  

Table 1: Chronological overview of the evidence collection process 

Date 
27.04.2022 
18.05.2022 
16.06.2022 
17.10.2022 
23.10.2022 
5.12.2022 

2.03.2023- 
25.05.2023 
19.04.2023 
Q1-Q2 2023 
21.08.2023 
8.09.2023 
10.10.2023 
10.01.2024 
12.01.2024 

Type of activity 

ISG meeting to discuss the Tender specifications 
Launch Request for services 
Offers received 
Signature of Contract 
ISG meeting: kick-off meeting with external contractor 
ISG  Meeting:  Inception  external  evaluation  report  and  discussion  about  the 
consultation strategy and public consultation 
Publication of Call for Evidence and public consultation 

ISG Meeting: Interim external evaluation report 
Targeted consultations 
Draft final report 
ISG meeting: draft Final external evaluation report 
Revised draft Final external report 
ISG meeting on the draft Staff Working Document 
Receipt of final version of external report supporting the evaluation 

Table 2: How RSB comments have been addressed 

Opinion 29.02.2024 (RSB comments) 

How and where comments have been addressed 

76 

 
 
 
 
 
The SWD has been significantly amended in order to take 
on board the RSB comments contained in the 29 February 
opinion and in the quality checklist received from the RSB 
on 22 February. See below the main changes to address the 
points to improve. 

Summary of findings:  
The  Board  notes  the  additional  information 
provided and commitments to make changes to 
the  report.  However,  the  report  still  contains 
significant  shortcomings.  The  Board  gives  a 
positive  opinion  with  reservations  because  it 
expects the DG to rectify the following aspects: 
(i)  The  report  does  not  critically  assess  the 
usefulness, comparability, and robustness of the 
key  indicators  in  assessing  performance  and 
success in reaching the specific objectives. (ii) 
The analysis of effectiveness, efficiency and the 
EU  added  value  is  not transparent in  terms  of 
methodology  used 
sufficiently 
underpinned by evidence. (iii) The report does 
not  sufficiently  discuss 
the  potential  for 
simplification and cost reduction. 

and  not 

What to improve: 

(1)  

(1)  The report should better analyse and discuss 
to  what  extent  the  indicators  and  the 
delivered outputs contribute to the effective 
delivery  of 
the  general  and  specific 
objectives given that several indicators lack 
targets  and  are  presented  in  isolation 
two  benchmarks  or  a  clear 
(without 
context). It should provide the reader with 
clear evidence on how the agencies made a 
difference and added value by analysing the 
agencies’  impact  on  knowledge  creation 
and implementation of policies within their 
competence. 
 The  report  should  better  discuss  to  what 
extent  the  absence  of  SMART  objectives 
hinders  both  the  determination  and  the 
presentation of what success looks like. 

•  The SWD Section 4.1. on evaluation findings has been 
fine-tuned  by  adjusting  the  conclusions  about  the 
extent of success, in light of indicator limitations. 

•  Section 4.1.1. on effectiveness in particular Subsection 
4.1.1.1.  B)  on  achievement  of  results  presents  an 
improved narrative that better links agencies outputs to 
the  achievement  of  objectives  and  how  the  agencies 
made a difference. 

•  The revised SWD provides a more detailed analysis of 
the  agencies  monitoring  systems  and  their  KPI 
limitations, including a new Subsection 1.2.2.1. Annex 
X,  presents  the  KPIs  and  indicates  which  KPIs  set 
targets. 

•  Cases  where  indicators  were  presented  in  isolation 
have  been  reviewed  and  included  benchmark/context 
(e.g. see new Figure 1 in Section 2.1. Annex VIII). 

•  The SWD Section 1.2.2.2. on limitation now includes 
explicit reference to the absence of SMART objectives. 
The lessons learnt section also includes a clear message 
on  the  need  for  the  Agencies  to  address  these 
shortcomings. 

(2)  The  report  should  more  convincingly 
explain the scale of success using tangible 
evidence  and  transparent  methodology  to 

(2) 
•  On  ETF,  the  SWD  sections  on  methodology  (1.2.2.) 
and  effectiveness  (4.1.1)  explain  more  clearly  the 
difficulties  to  assess  ETF  success  and  the  Agency’s 

77 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
support  its  assessments.  For  example,  it 
should more clearly outline to what extent 
the  European  Training  Foundation  can  be 
concluded as a success and why.  
Similarly,  on  coherence  between 
the 
agencies,  the  report  should  specify  which 
inter-agency cooperation has increased and 
delivered tangible gains.  

Since all four agencies produce and manage 
different  data  sets  the  report  should  also 
evaluate  how  effectively  they  are  made 
available to other users avoiding overlaps or 
duplication.  

specificity as regards achievement of results. This ETF 
point is also reflected in the conclusions section. 

•  On  coherence  between  the  agencies,  SWD  Section 
4.1.3. has been simplified, and better explains what has 
changed in inter-agency cooperation compared to the 
previous evaluation period.  

•  On data management, SWD Section 5.2. now explains 
how  the  Agencies  interact  with  other  Commission 
services,  particularly  Eurostat,  to  avoid  duplications 
and exploit complementarities.   

It 

to 

improve 

evidence 

(3)  The  report  should  be  a  self-standing 
document in terms of its evidence base and 
the  methodologies  used. 
should 
systematically  make  better  use  of  the 
available 
the 
effectiveness  analysis.  It  should  bring  the 
key  evidence  to the  main  report (from  the 
to 
annexes  and 
substantiate the  upbeat  conclusions  on  the 
agencies’ achievements. It should be more 
explicit about addressing the limitations of 
the  analysis  related  to  the  benchmarking, 
for  example,  because  of  lack  of  the 
quantitative targets for certain areas.  

support 

study) 

the 

It  should  better  present  the  views  from 
different  stakeholder  groups  and  when 
using  the  staff  survey,  it  should  be  more 
explicit about the potential stakeholder bias.  

(3)  

•  The  SWD  has  been  revised  to  make  it  a  more  self-
standing  document.  Additional  evidence  from  the 
supporting  study  and  the  SWD  annexes  has  been 
included  in  the  main  report.  In  particular,  in  the 
effectiveness section: 
o  The  revised  SWD  presents  a  more  granular 
assessment  of  the  achievement  of  the  agencies’ 
specific objectives (Section 4.1.1.1. B.2), based on 
the  broader  evidence  included  in  the  supporting 
study.  

o  SWD  Section  4.1.1.1.  B.1  Use  of  the  Agencies’ 
outputs has been further developed using evidence 
from Annex VIII. 

o  A  new  Annex  XII  on  achievement  of  results  has 
been added, and its main findings are presented in 
the  main  text,  in  particular  with  an  improved 
narrative  to  better  reflect  the  limitations  of  the 
analysis. 

•  For  the  Efficiency  criterion,  the  revised  SWD  now 
includes additional evidence from the supporting study 
on cost-effectiveness analysis of key activities of each 
agency.  This  provides  more  robust  evidence  to 
underpin  the  efficiency  findings.  A  new  annex  is 
provided based on Annex 6 of the supporting study XI: 
Cost-effectiveness of key activities.  

•  Presentation  of 

stakeholders’  views  has  been 
developed  to  include  perceptions  from  different 
stakeholder groups where feasible. In particular, staff 
perceptions  on  efficiency  have  been  replaced  with 
those from the stakeholder survey (see new Figure 12 
Annex  VII,  on  cost-effectiveness  assessment  by 
agency stakeholders). When the staff survey is referred 
to, potential bias has been made explicit (e.g. Section 
4.1.3.3.) 

78 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
(4)  The  report  should  significantly  strengthen 
the EU added value assessment. Given that 
EU-added  value  arguments  are  primarily 
based on the agencies’ own assessment, the 
report  should  substantiate  the  claims  with 
the  most  relevant,  impartial,  and  robust 
evidence.  

(4) 
SWD Section 4.2. on added value has been fully redrafted. 
The structure has been simplified and the assessment of EU 
added value is underpinned by robust objective evidence, 
including external sources and stakeholder perceptions.  

in 

(5)  The  report  should  make  better  use  of 
quantified  estimates 
the  efficiency 
section.  It  should  develop  the  REFIT 
dimension  by  more  thoroughly  looking  at 
the  potential  for  simplification  and  cost 
reduction. It should provide the total costs 
of  the  governance  structures  and  indicate 
the  cost  saving  potential  that  could  result 
from 
enhancing 
measures.  

efficiency 

related 

______________________________________ 
(6)  The conclusions should be revised to reflect 
better  the  findings  and  be  consistent  with 
the  outcome  of  the  analysis.  They  should 
mirror  the  changes  made  in  line  with  the 
recommendations  from  this  opinion.  The 
lessons  learned  should  better  discuss  the 
need for an improved data management and 
monitoring system based on more SMART 
objectives. 

(5) 

•  The revised SWD includes in the main text the cost 
figures on governance savings from the supporting 
study (see new Table 5 in Section 4.1.2.5.).  

•  The  SWD  text  now  includes  figures  on  the 
potential cost savings from merging scenarios (see 
Section  4.1.3.7.).  These  figures  are  taken  from 
Annex IX. 

______________________________________________ 
(6) 
•  Conclusions  have  been  revised  to  better  reflect  the 
findings, finetuning the assessment of achievement of 
results and making explicit the ETF specificities in this 
regard. 

•  General lesson learnt n° 8, on the need to improve the 
agencies  monitoring  systems,  has  been  redrafted.  A 
clear  message  is  stated  on  the  need  to  improve 
monitoring systems, not just for simplification, but also 
to  improve  performance  assessment.  The  need  to  set 
concrete  targets  for  all  indicators,  including  result 
indicators  is  clearly  stated  and  SMART  objectives 
should be set for future evaluations. 

79 

 
 
 
 
 
 
 
 
 
 
  
 
 
 
ANNEX II:  METHODOLOGY AND ANALYTICAL MODELS USED 

1.  Overall approach to the evaluation work  

The main objective of this evaluation is to provide, for  the four Agencies: Eurofound, Cedefop, 
the  ETF  and  EU-OSHA,  an  assesment  of  their  effectiveness,  efficiency,  coherence  and  EU 
added  value  and  relevance,  for  the  period  2017-2022238.  The  evaluation  covers  both  an 
individual assessment of each Agency and a cross-cutting comparative perspective.  

DG  EMPL  took  a  mixed  approach  to  this  evaluation,  by  relying  on  the  work  of  external 
evaluators to: (i) collect and analyse the relevant evidence (including consultation work); (ii) 
provide initial answers to all evaluation questions; and (iii) present evidence-based conclusions 
and lessons-learned. The supporting study followed a mixed methods data collection approach, 
combining qualitative and quantitative research tools and sources to collect wide evidence on 
the four Agencies performance. The evaluation did not assess the performance of Agencies in 
each individual Member State and sector because of data and sampling constraints. Instead, it 
focused on evaluating the performance of the Agencies at a broader EU-level.  

The overarching methodological approach was based on each Agency’s Intervention Logic (see 
Annex VI), which formed the basis of the evaluation’s analytical framework (see Annex III), 
encompassing the evaluation inquiries. To conduct the evaluation, the supporting study team 
completed six tasks (see Figure 1).  

238 The legal basis of the named evaluation is set in the three tripartite Agencies (Eurofound, Cedefop, EU-OSHA) 

revised Founding Regulation and in Art. 24 (2) of ETF’s Founding Regulation. 

80 

 
 
 
Figure 1: Overview of methodology 

81 

       a  ing of theAgencies  activities,out uts and results       onsultationactivities       ost effectivenessanalysis       ross cuttinganalysis of the fouragencies       ase studies       alidation  ocus rou s            nterviews withAgencies            ublic  onsultation           Agency level costeffectivenessanalysis           Activity level costeffectivenessanalysis            ndicator frameworkto assess coherenceand efficiency           esk research Task               ualitative datareview            uantitative datareview            taff and stakeholdersurvey            onsultationactivities,   A  case studies Tasks  ,  ,               ross cutting casestudies           Analysis andverification            ase studiesselection            ase studiesim lementation           Agency s ecificworksho s            ross cutting issues all Agencies  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Task 1: Mapping of the Agencies’ activities, outputs and results 

The  aim  of  this  task  was  to  collect  qualitative  and  quantitative  information  regarding  the 
performance of the four EU Agencies, starting from 2016, which was the baseline year, up to 
2022.  The  findings  of  the  task  were  used  to  conduct  a  first  assessment  of  each  Agency’s 
performance, pre-populate the relevant Agency-specific reports (see supporting study Annexes 
1 to 4). The findings were also used to inform subsequent tasks, with a specific focus placed on 
finding out potential gaps in the evidence-base, to be addressed through primary data collection 
methods. 

Subtask 1.1 - Qualitative data review  

This subtask consisted in a desk-based review of analytical and documentary evidence relating 
to the Agencies’ operations. Evidence was gathered using the sources. 

•  EU-level documents governing the legal framework of EU decentralised agencies 
and Eurofound, Cedefop, EU-OSHA and the ETF as well as other reports produced 
by EU institutions. 

•  Previous external and internal evaluations, analytical studies and audit reports. 
•  Studies, documents and reports prepared by the Agencies. 

The planning, monitoring and reporting documentation produced by the Agencies as well as 
other  internal  administrative  documentation  related  to  the  Agencies’  operational  and 
administrative processes and governance structures. 

After the selection of relevant documents, the research team developed a coding framework to 
dissect the relevant information with the view to answering the evaluation questions. 

Subtask 1.2 - Quantitative data review  

This subtask consisted in a desk-based review of the Agencies’ monitoring data against their 
performance  indicators.  Information  was  gathered  using  planning,  monitoring  and  reporting 
documentation produced by the Agencies (e.g. in their annual activity reports, annual accounts, 
etc.) as well as other internal administrative documentation related to the Agencies’ operational 
and administrative processes and governance structures. The research team used a quantitative 
mapping framework to analyse the achievement of the indicators by Agency, year, expected 
target and level of achievement. 

Task 2: Consultation activities 

The consultation activities aimed at gathering the views of a broad range of stakeholders which 
were concerned by the Agency’s activities. 

Subtask 2.1. Interview programme 

The interviews aimed at collecting detailed information from stakeholders directly involved in 
the Agencies’ operations, their target groups, and stakeholders working in similar policy and 
thematic areas. 

The  team  conducted  a  total  of  157  semi-structured  interviews  as  part  of  these  targeted 
consultations. This type of interview had an interview guide that served as a checklist of topics 
to be covered and a default wording and order for the questions. However, neither the wording 
82 

 
 
nor the order of questions was fixed in stone, while additional questions were spontaneously 
included as a follow up to what the interviewees’ said. Two types of interviews were included 
in the targeted consultations. 

A total of 137 agency-specific interviews were carried out with key staff members, members 
of the executive and managing / governing boards, as well as relevant stakeholders, such as 
social partners. Most of these interviews were conducted in person, whereas a set of interviews 
were conducted online.  The purpose of these interviews  was to  gather detailed insights  that 
were not available in the literature review and were challenging to capture using the closed-
ended questions employed in the rest of the consultation activities. The number of interviews 
conducted were: 

o  29 interviews for Eurofound; 

o  46 interviews for Cedefop; 

o  28 interviews for ETF; and 

o  34 interviews for EU-OSHA. 

Twenty  high-level  interviews  were  carried  out  with  key  stakeholders  (DG  EMPL,  other 
Commission  departments  and  institutions,  authorities  or  committees),  and  international 
institutions representatives. The purpose of these interviews was to test the intervention logics 
of  the  evaluation  as  well  as  initial  insights  on  the  quality  and  relevance  of  the  Agencies’ 
activities. 

Topic guides were tailored to the various roles, governance levels, and areas of expertise of the 
stakeholders  consulted.  Researchers  took  detailed  notes  during  each  interview.  These  notes 
were written up in a structured format using a coding framework to  facilitate the analysis.  

Interpreting the interview data was an iterative process.  On the one hand, the research team 
compared the views of stakeholders to find areas of consensus, and disagreement. On the other 
hand, the research team also compared the views of stakeholders with the available evidence 
from Task 1 as well as from the rest of the consultation activities, to triangulate the evidence. 

Subtask 2.2. Public consultation 

The research team assisted the Commission in conducting the public consultation. The aim of 
this  consultation  was  to  gather  wider  feedback  on  the  performance  of  the  Agencies  from 
relevant stakeholders.  

The public consultation, in contrast to Subtask 2.3. Staff and Stakeholder survey, focused on 
the evaluation criteria that were more suited for respondents like the general public to provide 
input on. It therefore excluded questions around the Agencies’ efficiency.  

The consultation was made available in all  official EU languages and remained open for 12 
weeks,  from  1 March  to  25 May 2023,  in  line  with  the  Better  Regulation  guidelines.  The 
consultation’s questionnaire included both close- and open-ended questions. To disseminate it, 
the research team followed a snowball sampling approach and invited the Agencies as well as 
relevant organisations at EU-level to share it with their membership. Dissemination efforts by 
the Agencies included email campaigns, as well as sharing the news on the consultation in their 

83 

 
 
respective websites, newsletters, social media, and email signatures, for example, in the case of 
Cedefop.  In  turn,  dissemination  efforts  from  the  research  team  included  sharing  the 
consultation’s  questionnaire  with  the  Members  of  the  European  Parliament  Committee  on 
Employment and Social Affairs via email, reaching out to relevant organisations at EU-level, 
as well as making regular posts about it on corporate social media accounts, leveraging, where 
possible, existing networks, such as the European Alliance for Apprenticeships LinkedIn page.  

Examples  of  relevant  organisations  at  EU-level  that  were  contacted  included  the  European 
Trade Union Confederation, BusinessEurope, the European Vocational Training Association, 
the European Association of Institutes for Vocational Training, the Partnership for European 
Research in Occupational Safety and Health, IndustriAll Europe, the European Federation of 
Building  and  Woodworkers,  the  European  Transport  Workers’  Federation,  Eurocommerce, 
Industry4Europe,  the  European  Logistics  Association,  the  European  Safety  Federation  and 
others. Despite the continuous efforts from the research team and the Agencies, the total sample 
size after cleaning the dataset was relatively small, with 101 respondents.  

Figure 2 shows the distribution of responses across the Agencies. The number of responses 
was higher than the number of respondents,  as respondents could answer for more than one 
Agency. 

Figure 2: Number of responses to the public consultation 

44

41

31

32

50

40

30

20

10

0

Eurofound

Cedefop

ETF

EU-OSHA

Eight weeks after the conclusion of the public consultation, a factual summary was published 
on the Commission’s webpages,239 whereas the consultation’s analytical findings can be found 
in Annex 9to the final report. 

Subtask 2.3. Staff and stakeholder survey 

At  the  same  time  as  Subtask  2.2.  a  targeted  survey  was  sent  to  the  Agencies’  staff  and 
stakeholders. The survey was accessible for 9 weeks in English, German, and French. It was 
sent to the Agencies' contact points together with the public consultation. 

239 For more information see: https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/13400-EU-
Agencies-Eurofound-Cedefop-ETF-and-EU-OSHA-2024-evaluation/public-consultation_en.  Date  of  last 
access: 04 August 2023  

84 

 
 
 
 
 
The survey gathered data concerning all five evaluation criteria. It served a dual purpose for the 
research team. On the one hand, quantifying diverse aspects of the Agencies' performance based 
on these criteria, and on the other hand, gathering qualitative input to improve the evaluation 
process. 

Based on the evaluation framework, the survey questionnaire was designed as follows to have 
two levels of questions.  

•  Level  1  questions:  common  general  questions  on  the  performance  of  the  Agencies. 

These questions were designed to facilitate a comparison between the Agencies. 

•  Level  2  questions:  specific  questions  for  each  Agency,  focusing  on  the  relevance, 
effectiveness, efficiency, coherence, and EU added value. These questions also focused 
on the direct or indirect use of outputs and results of the Agencies by their stakeholders.  

The surveys gathered a total of 560 individual contributions. Figure 3 shows the breakdown of 
responses by Agency. Similar to the Subtask 2.2., the stakeholder survey enabled participants 
to provide answers for multiple Agencies. Therefore, the total number of responses exceeded 
the number of respondents. 

Figure 3: Breakdown of responses by Agency to the online survey 

200

180

160

140

120

100

80

60

40

20

0

174

65

51

75

48

91

52

45

Eurofound

EU-OSHA

ETF

Cedefop

Staff

Stakeholders

The survey results have been included in the individual Agency reports, while they are also 
available, together with the survey questionnaires, in Annex 13 of the final report. 

Task 3: Cost-effectiveness analysis 

A  comprehensive  cost-effectiveness  analysis  (CEA)  was  conducted  for  the  four  Agencies’ 
activities. The methods and results of the CEA are available in Annex 6 of the final report.  

A mixed-method approach was used to collect the information needed to carry out the CEA 
through Tasks 1, 2, 5 and 6 in order to evaluate the relationship between inputs and outputs 

85 

 
 
 
 
as well as the relationship between inputs and higher-level results and impacts. CEA was 
split in two respective subtasks, as shown below. 

Subtask 3.1. Agency-level cost-effectiveness analysis 

This  subtask  provided  a  CEA  at  the  Agency-level  for  2017  to  2022.  The  analysis  gathered 
information about the following: 

• 

• 

• 

• 

total expenditure per year;  

total staff number per year; 

total expenditure per staff member per year; 

types of expenditure (administrative, labour, operational); 

•  staff allocation; 

•  annual budget implementation rates; 

•  efficiency of governance structures. 

Subtask 3.2. Activity-level cost-effectiveness analysis 

This subtask provided a CEA for the Agencies’ activities. The activities examined have been 
identified  based  on  the  Agencies’  own  activity  based  management  (ABM)  approach.  For 
Eurofound and EU-OSHA, quantitative performance indicators corresponding directly to the 
ABM activities were available and examined as part of this CEA. For CEDEFOP and the ETF, 
performance  indicators  that  related  directly  to  the  ABM  activities  were  not  consistently 
available over the evaluation period. As a result, for these two Agencies,  overall performance 
indicators were used in assessing cost-effectiveness. On top of the ABM, interview data were 
also used to evidence this part of the CEA.   

Task 4: Cross-cutting analysis of the Agencies 

Task 4 aimed to assess coherence issues among the Agencies and assess issues of efficiency. It 
focused on coherence and synergies between and among the Agencies and therefore aimed to 
understand  the  extent  to  which  the  four  organisations  interacted,  complemented,  cooperated 
and mutually reinforced one another in 2017-2022.  

As  a  secondary  focus,  the  task  also  looked  at  efficiency  among  the  Agencies,  including  the 
extent to  which they  duplicated each other’s objectives,  activities, governance  arrangements 
and internal  organisation.  It  therefore looked into whether resources were being  used in  the 
most efficient way.  

The results of this task informed a cross-cutting report, which is available in Annex 5 of the 
final report.  

Subtask 4.1. Indicator framework to assess efficiency and coherence 

The first subtask to conduct the cross-cutting analysis was to develop an indicator framework 
to systematically assess issues related to coherence and efficiency. It includes, on the one hand, 
the  relevant  indicators  together  with  the  focus  of  the  evidence,  and  on  the  other  hand,  the 

86 

 
 
 
different  methods  from  which  the  cross-cutting  analysis  drew  on  to  collect  the  necessary 
evidence.  

Subtask 4.2. Desk research 

Desk-research (Task 1) was one key source of evidence for the cross-cutting assessment. During 
this task, the research team reviewed agreements between the Agencies, and in particular the 
Memoranda of Understanding, the Framework Agreements, and the action plans that were in 
place  during  the  evaluation  period.  Evidence  available  in  the  consolidated  annual  activity 
reports,  Agencies’  programming  documents  and  relevant  previous  evaluations  was  also 
considered. This subtask considered literature that was used to produce the reports for each of 
the Agencies (Annexes 1 to 4) within the scope of this evaluation.  

Subtask 4.3. Consultation activities, CEA and case studies 

Consultation  activities  (Task  2),  cost-effectiveness  (Task  3)  and  cross-cutting  case  studies 
(Task 5) served as supplementary sources of information for the cross-cutting report.  

In particular, the research team considered the findings of the consultation activities (Task 2) 
related to synergies, collaboration, and coherence between and among the Agencies as well as 
efficiency savings and potential areas of simplification. In addition, it considered views on the 
added value and quality of joint activities. 

Further to  the above, it considered the results of the  cost-effectiveness-analysis (Task 3) to 
assess the type of joint outputs developed and implemented, and to evaluate the use of shared 
or combined resources among the Agencies. 

The analysis also used the findings from the five cross-cutting case studies (Task 5) on social 
dialogue and industrial relations, COVID-19, Agencies’ contribution to the debate on digital 
and green skills and on the ageing society, as well as on surveys of employers and employees.  

Subtask 4.4. Analysis and verification 

The information from Subtask 4.2. and Subtask 4.3. Consultation activities, CEA & case studies 
was used to populate  

Subtask 4.1. Indicator framework to assess efficiency and coherence.  

The focus of this analysis was on identifying: 

•  where  synergies  and  cohesive  working  between  the  Agencies  was  most  and  least 
prevalent was well as finding out which type of cohesion was common and the collective 
positive and negative outcomes of this; 

•  areas  of  clear  overlap  between  the  Agencies  in  terms  of  their  objectives,  activities, 
governance and internal organisation. In particular, finding out areas of either positive 
overlap (for example, in relation to stronger outcomes, for instance tackling the same 
issue from different angles) or negative overlap (for example, leading to duplication, 
confusion, resource inefficiency); 

•  potential solutions to weak cohesion. In particular, finding out if there was anything that 
could be done at the staff, Agency and the Commission level to ensure stronger cohesion 
in the areas identified during the evaluation.      

87 

 
 
The subsequent cross-cutting analysis was presented and checked in the validation focus group 
focusing on cross-cutting issues (see Task 6). During the focus group, staff members had the 
opportunity to comment on the findings, as well as agree or disagree with its main messages. 
The focus group also served as an occasion to discuss in more detail the extent to which the 
2019  evaluation  recommendations  on  reinforcing  cooperation  have  been  implemented  and 
identify what further recommendations were required to cement cohesion across the Agencies. 
These recommendations informed the final evaluation report.  

Task 5: Case studies  

This  task  consisted  in  selecting  and  delivering  25  stand-alone  case  studies  to  inform  the 
individual and cross-cutting assessment of the Agencies. In particular, it included five specific 
case studies per Agency (see Tables 1 to 6).  The case studies results are in Annex 8 of the final 
report.  

The specific case studies per Agency enabled an analysis of causal links between an Agency’s 
activities and its deliverables, outputs and immediate and intermediate outcomes and impacts 
to  which  it  contributed,  providing  evidence  to  substantiate  the  Agencies  assesment  of  the 
different evaluation criteria.  In turn, the cross-cutting case studies focused on developing an 
in-depth  understanding  of  how  the  contributions  of  different  Agencies  may  have  reinforced 
each other and/or how their synergies could be exploited.  

Subtask 5.1. Case studies’ selection 

To  achieve  a  balanced  view  of  the  Agencies’  different  activities  while  covering  all  the 
evaluation  criteria  and  questions  set  in  the  analytical  framework,  the  supporting  study  team 
used the following criteria, as well as the Commission’s and other relevant stakeholders’ input 
during the inception phase of the project, to inform the selection of the case studies.  

•  New projects and approaches of the Agencies. 

•  Geographical coverage and functions of the implemented activities/projects. 

•  Contextual information on the functioning of the Agencies. 

•  The range and functioning of priorities and working areas of the Agencies. 

•  The policy development level for particular thematic working areas. 

•  Specific  impacts  of  external  developments  such  as  the  COVID-19  pandemic, 
digitalisation, the growth of artificial intelligence, and the growth of the green economy. 

•  Practical considerations relating to the feasibility of carrying out particular case studies. 

Subtask 5.2. Case studies’ implementation   

The case studies were mainly based on desk-based research and the findings of the consultation 
activities. Tables 1 to 4 show the topics selected for the Agency-specific case studies. In turn, 
Table 5 shows the topics selected for the cross-cutting case studies. As noted earlier, these case 
studies are in Annex 8 of the final report. 

88 

 
 
 
 
Table 1: Eurofound case studies 

EUROFOUND 

1.  Contribution to discussions and policy decisions related to the COVID-19 pandemic and Russia’s war of 

aggression against Ukraine.  

2.  Contributions to discussions and policy decisions in relation to digitalisation, new and non-standard forms 

of work, including the platform economy, and implications for working conditions.   

3.  Eurofound’s work on the minimum wage. 
4.  Eurofound’s work on young people. 
5.  Eurofound’s  work  in  terms  of  communicating  knowledge  and  organising  debate  with  its  tripartite 

stakeholders on working and living conditions, industrial relations and the job market. 

Table 2: Cedefop case studies 

CEDEFOP 

1.  Contributions  to  discussions  and  policy  decisions  in  relation  to  initiatives  and  support  related  to 

empowering adults through upskilling and reskilling. 

2.  Contribution of Cedefop VET policy monitoring to EU and national VET policy. 

3.  Contribution of Cedefop to increased transparency and recognition of qualifications. 

4.  Review of Cedefop’s online and data visualisation tools. 

5.  Cedefop’s  work  in  terms  of  communicating  knowledge  and  organising  debate  with  its  tripartite 

stakeholders on VET, skills, and qualifications. 

Table 3: ETF case studies 

ETF 

1.  ETF support to the African Union (PANAFRICA) on the African Continental Qualification Framework 

(ACQF).  
Studying ETF activities in skill demand anticipation (including the participatory Skills Lab approach). 

2. 

3.  ETF Virtual training 2021 on VET for all EUD. 

4.  The ETF’s work on skills for green transition. 

5.  Communication and stakeholder involvement in supporting the Torino Process from 2018-onwards. 

Table 4: EU-OSHA case studies 

EU-OSHA 

1.  Support for OSH management in the context of COVID-19. 
2.  The contribution of research outputs from the ESENER survey to the evidence base for OSH policymaking 

and research on how OSH is managed in the workplace. 

3.  The  contribution  of  EU-OSHA’s  work  to  raising  awareness  of  psychosocial  risks  and  contributing  to 

discussion about psychosocial risk management, including in the context of COVID-19. 
4.   The healthy workplaces campaigns (HWCs). The focus will be on the following two HWCs: 

  2020-2022: Healthy Workplaces Lighten the Load. 
  2018-2019: Healthy Workplaces Manage Dangerous Substances. 

5.   The contribution of EU-OSHA to the EU’s strategic frameworks on health and safety at work. 

Table 5: Cross-cutting case studies 
Cross-cutting case studies 

1.  Actions  to  support  the  social  dialogue  and  industrial  relations,  including  the 

All four Agencies 

capacity-building of the social partners. 

89 

 
 
 
 
 
 
 
 
 
 
2.  The contribution of the Agencies to managing the impact of COVID-19 and 

All four Agencies 

shaping the ‘new normal’. 

3.  Agencies’  contribution  to  the  debate  on  the  digital  and  green  skills  /  just 

All four Agencies 

transitions. 

4.  Agencies’ contribution to policy at EU level on how to develop responses to the 

ageing society. 

5.  Surveys of employers and employees. 

All  four  Agencies  (the 
ETF to a lesser extent) 
All four Agencies 

Task 6: Validation focus groups  

The validation focus groups aimed at validating the draft evaluation findings and at filling in 
the  remaining  knowledge  gaps.  Five  validation  focus  groups  were  conducted  on 
18 September 2023:  four  covering  each  of  the  Agencies,  and  one  covering  all  Agencies 
collectively.  The  latter  focused  on  issues  around  the  coherence  among  and  between  the 
Agencies, and within this, on issues around their efficiency. The focus groups took place in a 
single  online  event  in  September.  In  total,  59  participants  participated  in  the  focus  groups, 
excluding the members of the research team. The key findings of each focus group have been 
embedded in the final report. 

Limitations and robustness of the findings 

The research is subject to a number of limitations, mainly arising from the availability of data 
and the quality of data at hand. This section lists this limitations, together with the mitigation 
measures that were taken.  

Task 1: Mapping of the Agencies’ activities, outputs and results  

Overall,  the  desk-based  review  of  the  Agencies’  monitoring  data  has  shown  that  on  some 
ocassions  the  Agencies  did  not  set  specific  targets  for  their  performance  indicators  to 
benchmark their performance. On these ocassions, the research team reported on the trends over 
time.  Furthermore,  it  appeared  that  in  certain  instances,  the  Agencies  were  changing  their 
performance indicators or altering the methods used to gauge progress on these indicators. This 
limited to a certain extent, the ability of the research team to report on trends over time.  

Task 2: Consultation activities 

Both  the  public  consultation  and  the  stakeholder  survey  did  not  have  a  finite  population.  
Therefore,  the  sample  of  these  surveys  should  not  be  considered  as  representative  and  their 
results should not be generalised. While the same was not true for the staff survey, its response 
rate also rendered its findings as non-representative. At the same time, while the staff survey 
had a finite population, its sample size, albeit important, cannot equally be considered as fully 
representative.  

To  address  these  limitations,  the  results  of  the  different  consultations  were  cross-referenced 
with each other, as well as with findings from interviews and desk-based research. Moreover, 
the information incorporated in the final report underwent validation through validation focus 
groups (see Task 5). Consequently, even though the sample size of the above surveys might not 
have been fully representative, the evidence used in shaping the final report gained substantial 
reliability due to this triangulation approach.  

90 

 
 
 
Despite extensive dissemination efforts (for more information, see the specific annexes on the 
public consultation and the survey results), the sample size of the public consultation and the 
stakeholder  survey  was  relatively  small.  As  a  result,  it  was  not  always  possible  to  conduct 
subgroup analysis for different stakeholder groups, as often these were either consisting of a 
handful  of  participants  or  they  were  non-existent.  Despite  this,  to  the  extent  that  subgroup 
analyses were possible, they have been conducted.  

Task 3: Cost-effectiveness analysis 

Typically, a cost-effectiveness analysis (CEA) incorporates an assessment of how far activities 
have been implemented at a cost which is commensurate with the level of outputs and results 
achieved. However, for this evaluation, many elements of the Agencies’ activities  could not 
form  part  of  a  conventional  quantitative  CEA.  In  particular,  for  many  of  the  Agencies’ 
activities,  it  was  not  possible  to  identify  a  single  effect  or  indicator  that  could  be  easily 
quantified. Unit cost analysis at the level of outputs also presented challenges, as the Agencies 
were  all  unique,  with  their  own  distinctive  remits,  objectives,  and  wide  range  of  different 
activities, making it difficult to apply a set of comparable performance indicators across all four 
Agencies. As such, the Agencies’ performance had to be considered against multiple objectives 
rather than one output to be achieved by a given date. 

In recognition of these challenges, the CEA considered the total and/or unit costs of activities, 
along  with  factors  that  influenced  delivery,  effectiveness,  and  efficiency.  A  mixed-method 
approach has been conducted that collected  data through desk research, interviews, surveys, 
and case studies, to evaluate both production efficiency (i.e. the relationship between inputs and 
outputs) and allocation efficiency (i.e. the relationship between inputs and higher-level results 
and impacts) and how such efficiencies could be improved.   

91 

 
 
 
 
Tasks 

s
n
o
i
t
a
t
l
u
s
n
o
c

n
o
i
t
a
t
l
u
s
n
o
c

g
n
i
p
p
a

d
e
t
c
e
i
g
l
b
r
a
u
M
T
P
✔  ✔  ✔ 

s
i
s
y
l
a
n
a

s
e
i
d
u
t
s

s
s
e
n
e
v
i
t
c
e
f
f
e
-
e
t
s
s
o
a
C
C
  ✔ 

✔  ✔  ✔ 

✔ 

ANNEX III:  EVALUATION MATRIX  

Table 1: Evaluation matrix 

Sub-
questions 

Potential judgement criteria / indicators 

Evaluation 
criterion 

Main 
research 
question
s (as 
specified 
in the 
tender 
specifica
tions) 

Effectiveness  To what 

extent did 
the 
Agencies 
achieve 
their 
objectives? 

To what 
extent the 
objectives 
identified in 
the 
intervention 
logic for 
each Agency 
has been 
achieved? 

How do 
stakeholders 
and the 
wider public 
perceive the 
quality of 
the services 
provided by 
the four 
Agencies?  
How visible 
were their 
services and 
to which 
stakeholders
? 

To what 
extent 
were the 
services of 
the four 
Agencies 
actually 
used by 
their 
stakeholde
rs, 
including 
EU 
Institutions
, 
stakeholde
rs in the 
Member 
States, 
internation
al bodies 
and 

- Extent to which the main objectives set out in Annual Work 
Programmes/ Programming documents of the Agencies have been 
reached (% of objectives). 
- Extent to which expected outputs were achieved by the Agencies 
(number of publications, surveys, events delivered compared to the 
baseline period). 
- Extent to which results indicators were achieved by the Agencies 
(number of downloads, website visits, references in EU level 
documents, academic publications compared to the baseline period). 
- Stakeholder perception of the achievement of Agencies objectives as 
reported by stakeholder satisfaction surveys conducted by the 
Agencies during the evaluation period. 
- The extent to which expected outputs, results and impacts have been 
achieved according to the evaluations commissioned by the Agencies 
during the evaluation period. 
- Perception of the Agencies’ stakeholders and staff on the extent to 
which the objectives (identified in the intervention logic for each 
Agency) have been achieved. 
-Stakeholders’ perception of the quality of Agencies’ services (at least 
two thirds of all respondents are fully or largely satisfied). 
- Perception of the Agencies’ stakeholders and staff on what factors 
facilitated the achievement of objectives. 
- Agencies’ stakeholder and staff perceptions on the extent to which 
national networks (Eurofound’s Network of Correspondents, EU-
OSHA focal points, CEDEFOP’s ReferNet and ETF’s representatives 
from partner countries) contribute to achieving the objectives set out 
in the Agencies’ intervention logics. 
- Contribution of the Agencies’ performance to the observed changes 
(impacts). 
- Perception of the Agencies’ stakeholders and staff of the main 
reasons why any objectives were not achieved (if any). 
- Volume of use measured by number of PDF and HTML downloads 
(achievement of Agencies’ KPIs, % increase each year). 
- Use of Agencies’ expertise in key EU-policy documents 
(achievement of Agencies’ KPIs, % increase each year). 
- Number of references in peer-reviewed publications (% increase 
year on year) 
- Contribution to policy developments at priority events (achievement 
of Agencies’ KPIs, % increase each year, if available). 
- Contribution to events organised by other organisations. 
- Regularity of stakeholders’ use of Agency services (at least two 
thirds of all respondents indicate that they use Agencies’ services 
regularly). 
- Frequency of use (at least half the respondents have used Agencies’ 
services in the last six months). 
- Number of stakeholders reporting the use of Agencies’ services in 
preparing policy documents, workshops and informing national 
debates (at least half of respondents have used the Agencies’ services 
for their intended purpose). 
- References to the Agencies’ work in European media as reported in 
annual activity reports (if available). 

92 

 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
organisatio
ns and, in 
the case of 
ETF, 
partner 
countries?  
To what 
extent did 
Agencies 
focus their 
work on 
areas most 
in need 
and where 
they can 
have an 
impact? 

Were there 
other 
broader 
achieveme
nts/unexpe
cted 
impacts 
arising 
from the 
Agencies 
operations
? 
Which 
factors 
facilitated 
or hindered 
the 
effectivene
ss of the 
four 
Agencies’ 
operations
?  

To what 
degree 
have the 
four 
Agencies 
adapted to 
changes in 
EU policy, 
to 
Commissio
n political 
priorities 
over the 
evaluation 
period and 
to the 
political 
and socio-
economic 
situation in 
general? 
In 
particular, 

- Stakeholders’ perception on whether Agencies’ services meet their 
needs (at least-two thirds of respondents indicate their needs have 
been met). 
- Utility of services (at least-two thirds of all respondents find services 
Agencies provide ‘very useful’ or ‘useful’). 
- Stakeholders’ perceptions of the most useful services for their work. 
- Stakeholders’ perceptions of the areas where Agencies’ services 
would be needed to address their needs. 
- Process by which stakeholders’ needs are taken into account when 
identifying priorities, activities and services (existing mechanisms for 
stakeholder involvement in the decision-making process). 
- Stakeholders’ perceptions on the extent to which Agencies focus on 
providing evidence to policy makers. 
- Stakeholders’ perceptions on the extent to which Agencies are 
engaging in policymaking. 
- Stakeholders share examples of wider/unexpected impacts (if 
relevant).  
- Examples of wider unexpected impacts identified in annual reports, 
evaluations and stakeholder feedback surveys. 
- Stakeholders report the factors that facilitated the generation of 
unexpected impacts (if applicable). 

✔  ✔  ✔ 

  ✔ 

  ✔  ✔ 

  ✔ 

- Documentary evidence indicates the factors that facilitated and/or 
hindered achievement of the objectives. 
- Management/Governing Boards’ perceptions of factors that 
facilitated and hindered the achievement of Agencies’ objectives. 
- Agencies' staff reporting examples of factors that facilitated and 
hindered the achievement of the Agencies’ objectives. 
- Evidence on the fulfilment of Headquarters Agreements from the 
document analysis.  
- Staff perceptions on host Member States obligation fulfilment. 

✔  ✔  ✔ 

  ✔ 

- Evidence of any changes in Agencies’ activities in relation to 
changes in the EU policy priorities and broader changes in the 
political and socio-economic situation in the EU based on document 
analysis.  
- The extent to which these changes in Agencies’ activities have 
referenced changes in EU policy or the EU’s political and socio-
economic situation in their justification.  
- Stakeholders’ and staff perception of the changes in Agencies’ 
activities in relation to the broader changes in the EU.  

✔  ✔  ✔ 

  ✔ 

To what 
degree did 
host 
Members 
States fulfil 
their 
obligations 
as defined in 
the 
Headquarter
s 
Agreements 
between the 
Agency and 
Member 
State? 

- Documentary evidence of the new services developed in response to 
COVID-19 and Russia’s war of aggression against Ukraine. 

✔  ✔  ✔ 

  ✔ 

93 

 
 
 
 
 
 
Efficiency 

to what 
degree 
have the 
four 
Agencies 
provided 
useful 
support 
with regard 
to the 
unexpected 
and 
additional 
challenge 
of the 
COVID 
pandemic 
and 
Russia’s 
war of 
aggression 
against 
Ukraine, 
where 
pertinent? 
To what 
extent 
were the 
operations 
of the four 
Agencies 
cost-
effective? 

To what 
extent 
were staff 
resources 
and 
workload 
appropriate 
to fulfil 
efficiently 
and 
effectively 
the 
Agencies’ 
objectives 
and 
activities? 
To what 
extent 
were the 
internal 

- The extent to which Agencies adapted their cooperation patterns 
with other Agencies to respond to the needs caused by COVID-19 and 
Russia’s war of aggression against Ukraine.  
- Comparative analysis of the achievement of Agencies’ KPIs in 
2017-2019 and 2020-2022 to assess the effect of COVID-19 on the 
effectiveness of Agencies’ operations. 
- The uptake and results indicators (such as downloads) show the use 
of the outputs produced as a response to the COVID-19 pandemic and 
Russia’s war of aggression against Ukraine. 
- Stakeholders’ and staff perceptions of the quality of the services 
developed in response to COVID-19 and Russia’s war of aggression 
against Ukraine (at least two thirds of respondents are fully or largely 
satisfied). 

- Expenditure tracking by title and type of activity per Agency.  
- Total expenditure per staff member by Agency compared with other 
Agencies. 
- Operational efficiency (% of budget approved actually committed).  
- Staff allocation by type of role (administrative, neutral, operational) 
per Agency. 
- Ratio of operational and administrative expenditure. 
- Benchmarking Agencies’ performance to each other and other 
selected Agencies regarding cost-effectiveness.  
- Case-study evidence on the efficiency and effectiveness of specific 
Agencies’ activities.  
- Analysis of the output and result indicators in the context of the cost-
effectiveness of key activities. 
- Agencies’ evaluations conducted during the evaluation period 
assessment of the cost-effectiveness of Agencies’ activities (when 
available). 
- Comparative rating by stakeholders and staff with regard to 
Agencies’ cost-effectiveness.  
- Perceptions of stakeholders and staff whether any improvements 
need to be made to increase the cost-effectiveness of the Agencies’ 
activities.  
- Comparing country coefficient for each Agency and the extent to 
which it affects the resources allocated to each Agency. 
- Agencies stakeholders and staff perceptions on the extent to which 
survey methods used are efficient in collecting high quality data. Are 
there other survey methods that would be beneficial in certain 
circumstances? 
- Staff perception of workload balance.  
- Staff perception of human resources allocated to fulfil their duties.  
- Staff perceptions on the efficiency of the distribution of tasks inside 
the Agency. 
- Vacancy rate analysis. 
- Benchmarking Agencies’ performance to each other regarding 
adequacy of staff resources and workload.  

How 
balanced 
were the 
administrativ
e and 
operational 
budgets and 
why? 

✔  ✔  ✔  ✔  ✔ 

✔  ✔  ✔ 

  ✔ 

In particular, 
did 
digitalisation 

- Review of monitoring data in the performance measurement system 
and its links to specific monitoring, reporting and evaluation 
mechanisms.  

✔  ✔ 

  ✔ 

94 

 
 
 
 
mechanism
s for 
monitoring
, reporting 
and 
evaluating 
the 
Agencies 
adequate 
for 
ensuring 
accountabi
lity and 
appropriate 
assessment 
of the 
overall 
performan
ce of the 
Agencies, 
while 
minimising 
the 
administrat
ive 
burden?  
How 
efficient 
were the 
Agencies’ 
governanc
e 
structures?  

How well 
were the 
Agencies 
embedded 
in the 
broader 
EU policy 
governanc
e structures 
for their 
respective 
activity 
domains, 
and is 
there room 
for 
improving 
coherence 
and 
efficiency? 
Were there 
any 
potential 
areas 
and/or 
activities 
which 
could be 

play a role in 
the above? 

- Review of documentary evidence, particularly European Parliament 
discharge reports and audit reports on the Agencies’ internal 
monitoring, reporting and evaluation mechanisms. 
- Benchmarking Agencies’ performance to each other regarding 
adequacy of internal programming, monitoring, reporting and 
evaluating mechanisms. 
- Reviewing evidence on the changes to the monitoring, reporting and 
evaluation mechanisms. 
- Perceptions by Management/Governing Boards and Agencies’ staff 
about the ability of the mechanisms to ensure accountability of the 
Agencies.  
- Agencies’ staff perceptions of the overall administrative burden of 
the internal programming, monitoring, reporting and evaluating 
procedures.  

✔  ✔  ✔ 

  ✔ 

  ✔  ✔ 

  ✔ 

To what 
extent were 
the size and 
the 
composition 
of the 
Agencies’ 
management 
boards 
appropriate 
for the size 
and nature of 
the Agencies 
so as to 
ensure 
effective and 
efficient 
governance? 

- The extent to which the size and the composition of the Agencies’ 
management boards are in line with the other Decentralised Agencies. 
- Comparative analysis of the alternative tripartite governance models. 
- SWOT analysis of the alternative tripartite governance models. 
- Evidence on how the management boards contribute to the 
Agencies’ activities. 
- Evidence on the extent to which operations of the management 
board are appropriate to implement their functions (e.g. preparation 
and organisation of the meetings, decision-making process). 
- Stakeholder perception of the impact of management board size and 
composition on Agencies’ work.  
- Evidence from previous programming, monitoring, and reporting 
documents on the impact of the size and composition of management 
boards.  
- Evidence on the costs associated with the functioning of Agencies’ 
management boards.  

- EU-level stakeholders’ perceptions on the extent to which the 
Agencies are embedded in broader EU governance structures.  
- The extent to which the membership at the Agencies’ 
Management/Governing Boards and the EU-level policy/advisory 
committees are similar. 
- Agency leadership teams, EU-level stakeholders and social partners’ 
perceptions on the extent to which there are overlapping topics, 
mandates and membership between EU-level policy/advisory 
committees (e.g. ACVT, OSH Committee) and governance structures 
of the Agencies. 
- Agency leadership teams, EU level stakeholders and social partners’ 
perceptions on potential synergies/efficiency gains between the 
Agencies’ governance structures and the EU-level policy/advisory 
Committees (e.g. ACVT, OSH Committee). 
- Agencies’ staff perceptions on the extent to which the Agencies are 
embedded in the broader EU policy governance structures. 

  ✔  ✔ 

  ✔ 

- Examples of simplification measures introduced during the 
evaluation period based on document analysis and interviews. 
- Stakeholders reporting on the areas where further digitalisation 
would increase the efficiency of the Agencies’ operations. 
- Stakeholders reporting on potential efficiency gains due to cross-
Agency cooperations. 
- Agencies staff reporting examples of areas where the simplification 
would be possible. 

95 

 
 
 
 
- Mapping exercise to determine any impacts of new elements since 
the last evaluation. 
- Stakeholder views on changes in activities of Agencies since the last 
evaluation.  

  ✔  ✔ 

  ✔ 

Are there 
any other 
new 
elements that 
could play a 
role? 

subject to 
simplificati
on and/or 
administrat
ive burden 
reduction? 
To what 
extent do 
new 
elements 
(e.g. the 
ELA, 
updated 
mandates) 
change the 
conclusion
s of the 
previous 
evaluation 
regarding 
advantages 
/ 
disadvanta
ges of 
mergers 
between 
the 
Agencies? 

Coherence  To what 

extent 
were the 
mandates 
and 
activities 
of the four 
Agencies 
coherent 
among 
themselves
?  
To what 
extent do 
the four 
Agencies 
work 
cooperativ
ely with 
each other 
where 
needed?  

Were there 
any 
unnecessary 
overlaps or 
duplications
? 

To what 
extent have 
they 
reinforced 
cooperation 
among 
themselves 
during the 
current 
evaluation 
period? 

To what 
extent do 
they work 
cooperativel
y with DG 
EMPL and 
other 
Commission 
services? 

To what 
extent are 
the four 
Agencies’ 
mandates 
and 
activities 
coherent 
with DG 
EMPL 
policies 
and with 
other EU 

✔  ✔  ✔ 

  ✔ 

✔  ✔  ✔ 

  ✔ 

✔  ✔  ✔ 

  ✔ 

- Mapping and comparative analysis of the documentary evidence on 
the mandates, objectives, activities, target groups and themes of the 
four Agencies. 
- Examples of joint outputs delivered in cooperation between two or 
more Agencies related to the common themes identified. 
- Levels of duplication (mandates, objectives, activities, themes) –
none, minor, major. 
- Levels of complementarity (mandates, objectives, activities, themes) 
–none, minor, major. 
- Stakeholder perceptions of coherence of the mandates and activities 
of the Agencies among themselves. 

- Documentary evidence on the implementation of the 
recommendations from the previous evaluation in 2019 on reinforced 
cooperation, including cooperation regarding corporate functions 
(strategy, human resources, legal and financial management), 
performance management, mutual learning and sharing services, joint 
delivery, joint programming and planning. 
- Review of Agencies’ cooperation agreements and joint action plans 
and comparison to the previous evaluation period. 
- Review of joint publications, including survey reports. 
- Agencies management boards, stakeholders and staff report the areas 
where the cooperation was most successful and areas for 
improvement. 
- Agencies management boards, stakeholders and staff reporting on 
the factors that facilitated and hindered reinforced cooperation. 
- Agencies management boards, stakeholders and staff perceptions on 
the areas where further cooperation would have the greatest impact on 
the effectiveness and efficiency of the Agencies’ operations. 
- Processes are in place, and steps are taken to exploit synergies and 
minimise duplication – none, ad-hoc, systematic. 
- Documentary evidence/mapping of complementarities between the 
mandates and activities of the Agencies and relevant European 
Commission DGs and policies. 
- Documentary evidence of the uptake of the Agencies’ outputs in 
EU-level publications and events. 
- Mentions of the Agencies’ outputs in EU-level documents (when 
available). 
- EU-level stakeholders, Management/Governing Board members and 
staff reporting on the level of cooperation. 
- EU-level stakeholders, Management/Governing Board members and 
staff are reporting on the factors that facilitated/hindered cooperation. 

96 

 
 
policies 
(including, 
in the case 
of ETF, 
with the 
“Geopoliti
cal 
Commissio
n” 
approach 
and, in the 
case of 
OSHA, 
with the 
health in 
all policies 
approach, 
namely 
related to 
the 
prevention 
of 
occupation
al risks)?  
To what 
extent are 
the 
mandates 
and 
activities 
of the four 
Agencies 
coherent 
with those 
of other 
EU 
decentralis
ed 
Agencies 
(including 
the EUAN 
network), 
and, in 
particular, 
with the 
European 
Labour 
Authority?  
To what 
extent are 
the 
mandates 
and 
activities 
of the four 
Agencies 
coherent 
with those 
of other 
relevant 
organisatio
ns and 
stakeholde
rs (at EU, 
internation
al and 
national 
level)? 
What was 
the EU 
added 
value of 

EU added 
value 

- EU-level stakeholders, Management/Governing Board members and 
staff reporting on the areas where further complementarities would be 
beneficial to increase cooperation. 

✔  ✔  ✔ 

  ✔ 

To what 
extent do the 
four 
Agencies 
work 
cooperativel
y with those 
other 
Agencies? 

- Documentary evidence review and mapping of the mandates, 
objectives, activities, target groups and themes of the four Agencies 
and ELA.  
- Mapping existing cooperation agreements, joint action plans and 
Memoranda of Understanding between the four Agencies and ELA. 
- Documentary evidence review of potential complementarities with 
other decentralised Agencies (e.g. EIGE). 
- Agencies staff (including those that are not covered by this 
evaluation e.g. ELA) and stakeholders (including EUAN) report 
examples of mutual learning, sharing services, development of joint 
expertise and other examples of cooperation. 
- Agencies stakeholders and staff reported examples of coherence and 
cooperation in response to the common challenges e.g. COVID-19 
and Russia’s war of aggression against Ukraine (where relevant). 
- Stakeholders’ perceptions on the coherence of Agencies’ mandates 
with those of ELA. 
- Agencies stakeholders’ and staff perceptions on the factors that 
facilitated and hindered coherence and cooperation. 

To what 
extent do 
they work 
cooperativel
y with those 
organisation
s and 
stakeholders
?  
To what 
extent is 
such 
cooperation 
established 
with all 
relevant 
organisation
s and 
stakeholders
? 
To what 
extent did 
their 
achievement

- Documentary evidence of cooperation agreements with international 
organisations (OECD, UNESCO, ILO). 
- Documentary evidence of the examples of cooperation with 
international organisations and stakeholders at national level. 
- Stakeholders’ perceptions on the level and nature of cooperation. 

✔  ✔  ✔ 

  ✔ 

- Documentary evidence of the added value of the Agencies’ 
operations. 
- Stakeholders’ perceptions (including partners working in similar 
areas) whether the Agencies’ activities are unique and could not be 

✔  ✔  ✔ 

  ✔ 

97 

 
 
achieved through existing mechanisms at national, EU and 
international levels. 
- Stakeholders’ perceptions of the characteristics of the Agencies’ 
work they value the most. 

s add value 
in terms of 
volume, 
scope, 
process and 
role 
effects240, in 
particular in 
comparison 
to the 
previous 
evaluation 
period? 

- Documentary evidence on the intended target groups of Agencies’ 
operations. 
- Analysis of the indicators related to the use of Agencies’ services by 
intended beneficiaries collected during the evaluation of the 
effectiveness of Agencies’ operations. 
- Stakeholders’ perceptions of the groups of stakeholders directly 
affected by and using the Agencies’ services. 

✔  ✔  ✔ 

  ✔ 

✔  ✔  ✔ 

  ✔ 

- The extent to which outputs (based on the documentary analysis) 
were relevant to the EU policy needs. 
- The extent to which (based on interviews) high-level officials 
involved in the EU policy planning and Agencies’ 
Management/Governing Boards agree that the Agencies contribute to 
addressing EU policy needs. 
- Stakeholders’ perceptions regarding relevance of Agencies’ outputs 
and activities (at least two thirds of respondents considering 
Agencies’ work ‘Highly relevant’ or ‘Relevant’). 
- Stakeholders’ perceptions on the extent to which Agencies’ services 
have met their needs. 
- Stakeholders’ perceptions on the extent to which they feel they can 
inform Agencies’ activities. 

- Documentary evidence of the activities specifically developed to 
address the challenges posed by COVID-19 and Russia’s war of 
aggression against Ukraine. 
- Stakeholders’ perceptions on the relevance of the Agencies 
mandates and activities in responding to challenges posed by COVID-
19 pandemic and Russia’s war of aggression against Ukraine. 

  ✔  ✔ 

  ✔ 

Relevance 

the four 
Agencies 
during the 
evaluation 
period?  

To which 
groups of 
stakeholde
rs 
concretely 
did the 
Agencies 
work make 
a 
difference? 
To what 
extent did 
the four 
Agencies’ 
mandates, 
objectives 
and 
activities 
fulfil EU 
policy 
needs and 
those of 
relevant 
stakeholde
rs during 
the 
evaluation 
period? 
To what 
extent are 
the four 
Agencies’ 
mandates 
and 
activities 
still 
relevant, 
and has 
that been 
affected by 
the 
COVID-19 
pandemic 
and 
Russia’s 
war of 
aggression 
against 
Ukraine 

240 Volume effects: Agency action adds to existing action, either at EU, international or national level; Scope 
effects: Agency action broadens existing action by supporting groups or policy areas that would not 
otherwise receive support; Role effects: Agency action supports innovations that are taken up at national 
level or national innovative actions that are then mainstreamed; Process effects: Agency action influences 
Member-State administrations and organisations involved in the Agency actions.   

98 

 
 
 
 
 
 
 
 
If yes, what 
would be the 
financial 
implications
? 

- Documentary evidence identified/not identified on the need to 
amend Agencies’ mandates. 
- Stakeholders’ perceptions on the need to amend Agencies’ 
mandates. 
- Staff perceptions on the need to amend Agencies’ mandates. 

  ✔  ✔ 

  ✔ 

- Documentary analysis identifies the trends the Agencies need to 
consider to remain relevant. 
- Stakeholders’ perceptions on the impacts of future trends on the 
Agencies’ future relevance. 

  ✔  ✔ 

  ✔ 

- Rate (%) of recent audit recommendations implemented. 

✔  ✔  ✔ 

  ✔ 

(where 
pertinent)? 
To what 
extent is 
there a 
need to 
amend the 
mandate of 
the four 
Agencies?  
In terms of 
foresight, 
are there 
any future 
trends 
including 
megatrend
s (such as 
demograph
ic change, 
migration, 
etc.) that 
could 
affect the 
four 
Agencies’ 
future 
relevance 
and how? 
To what 
extent have 
recent 
audit 
recommen
dations 
been put 
into 
practice?  

99 

 
 
 
 
 
 
 
 
ANNEX IV:  OVERVIEW OF BENEFITS AND COSTS  

This Annex presents an summary of the cost and benefits examined  in the accompaning main 
report produced for the study supporting the evaluation of EU Agencies: Eurofound, Cedefop, 
the ETF and EU-OSHA. Values are presented in millions of euro (€) unless otherwise stated. 
The cost data are taken from the Agencies consolidated annual activity reports and the Agencies 
published  accounts  on  their  activity  based  management  (ABM)  activities.  Unless  otherwise 
stated, the figures presented in the benefits section relate to the stakeholder and staff interviews 
and  surveys  carried  out  as  part  of  this  evaluation.  Further  details  on  the  sources  used  are 
contained in the main report. 

The table presents direct costs and benefits identified through this evaluation. While the work 
of  the  Agencies  can  be  assumed  to  result  in  indirect  outcomes  as  well  as  additional  direct 
outcomes,  identifying  and  quantifying  these  is  highly  challenging  –  for  example,  it  is  very 
difficult  to  identify  specific  skills  or  health  outcomes  that  related  back  to  the  work  of  the 
Agencies,  although  many  of  the  Agencies  activities  are  designed  to  lead  to  longer-term 
downstream effects on a wide range of individuals or organisations. 

Table 1: Benefits and costs 

Total 
administrative 
costs 

Administrative 
cost: 
Administrative 
costs per staff 
member (€)  

20.7m 
20.9m 
21.2m 
22.6m 
23.6m 
23.6m 
1.9m 
2.7m 
2.6m 
1.6m 
1.7m 
1.7m 
16 376 
23 888 
21 315 
13 702 
14.865 
15 319 
0.15m 
0.15m 
0.21m 
0.15m 
0.14m 
0.06m 

EUROFOUND 
Direct costs (borne by the Agencies) 
2017 
Total 
2018 
expenditure of 
2019 
the Agency 
2020 
2021 
2022 
2017 
2018 
2019 
2020 
2021 
2022 
2017 
2018 
2019 
2020 
2021 
2022 
2017 
2018 
2019 
2020 
2021 
2022 
Survey management 
and development 
1.39m 
2017 
2.42m 
2018 
3.18m 
2019 
4.24m 
2020 
1.84m 
2021 
3.91m 
2022 

Administrative 
cost: Cost of 
governance 
meetings  

Key activities 
of the Agency 

CEDEFOP 

ETF 

EU-OSHA 

17.0m 
18.3m 
19.2m 
17.1m 
18.5m 
18.3m 
1.4m 
1.6m 
1.6m 
1.9m 
1.9m 
2.0m 
11 932 
13 456 
14 052 
16 208 
15 430 
18 364 
0.16m 
0.15m 
0.15m 
0.04m 
0.09m 
0.07m 

2017 
2018 
2019 
2020 
2021 
2022 
2017 
2018 
2019 
2020 
2021 
2022 
2017  
2018  
2019  
2020  
2021 
2022  
2017 
2018 
2019 
2020 
2021 
2022 
VET systems and 
institutions 
2017 
2018 
2019 
2020 
2021 
2022 

2.14m 
2.02m 
2.09m 
1.60m 
2.04m 
1.66m 

100 

20.2m 
20.1m 
20.1m 
21.2m 
20.6m 
21.2m 
1.9m 
2.0m 
1.7m 
2.3m 
1.7m 
1.6m 
13 841 
14 821 
12 512 
16 958 
12 405 
11 943 
0.15m 
0.12m 
0.09m 
0.01m 
0.08m 
0.12m 

2017 
2018 
2019 
2020 
2021 
2022 
2017 
2018 
2019 
2020 
2021 
2022 
2017 
2018 
2019 
2020 
2021 
2022 
2017 
2018 
2019 
2020 
2021 
2022 
VET provision and 
quality 
2017 
2018 
2019 
2020 
2021 
2022 

3.99m 
3.86m 
4.12m 
4.53m 
not 
available 
not 
available 

15.3m 
2017 
15.9m 
2018 
15.5m 
2019 
15.6m 
2020 
15.8m 
2021 
16.4m 
2022 
1.4m 
2017 
1.4m 
2018 
1.3m 
2019 
1.6m 
2020 
1.7m 
2021 
1.6m 
2022 
22 406 
2017  
20 846 
2018  
21 206 
2019  
24 810 
2020  
26 391 
2021  
24 483 
2022 
0.15m 
2017 
0.15m 
2018 
0.10m 
2019 
0.08m 
2020 
0.06m 
2021 
2022 
not available 
Awareness Raising and 
Communication 
2017  
2018  
2019  
2020  
2021  
2022  

5.48m 
4.25m 
4.19m 
4.13m 
3.61m 
4.00m 

 
 
 
Corporate 
communication and 
infrastructure 
2017 
2018 
2019 
2020 
2021 
2022 

4.71m 
4.69m 
4.34m 
5.06m 
6.54m 
5.82m 

Skills and labour market 
research 

Employment, skills, 
and employability 

Facts and Figures 

2017 
2018 
2019 
2020 
2021 
2022 

1.36m 
1.33m 
1.87m 
1.41m 
1.98m 
0.83m 

2017 
2018 
2019 
2020 
2021 
2022 

2017 
2018 
2019 
2020 
2021 
2022 

3.25m 
3.75m 
2.92m 
4.37m 
not 
available 
not 
available 

0.91m 
2.39m 
2.56m 
2.42m 
2.47m 
2.70m 

Quality of life/living 
conditions 

Transversal activities 

2017 
2018 
2019 
2020 
2021 
2022 

1.89m 
1.44m 
1.87m 
1.34m 
1.60m 
0.91m 

2017 
2018 
2019 
2020 
2021 
2022 

0.31m 
0.34m 
0.33m 
0.18m 
0.33m 
0.14m 

Employment and 
labour markets 
research 
2017 
2018 
2019 
2020 
2021 
2022 

3.96m 
2.85m 
3.03m 
3.79m 
1.84m 
1.04m 

Social dialogue and 
industrial relations 

2017 
2018 
2019 
2020 
2021 
2022 

1.28m 
1.47m 
1.62m 
2.02m 
3.61m 
3.57m 

Reacting to ad-hoc 
information requests 
2017 
2018 
2019 
2020 
2021 
2022 

0.4m 
0.67m 
0.38m 
1.82m 
0.11m 
0.23m 

Support to EU 
assistance in the 
context of EU 
external policies 
1.56m 
2017 
1.16m 
2018 
2.21m 
2019 
1.99m 
2020 
not 
2021 
available 
2022 
not 
available 
Policy analysis and 
system wide progress 
monitoring 
2017 
2018 
2019 
2020 
2021 
2022 

3.01m 
3.24m 
4.81m 
4.89m 
not 
available 
not 
available 

VET governance 

2017 
2018 
2019 
2020 
2021 
2022 

2.95m 
2.41m 
1.76m 
2.33m 
not 
available 
not 
available 
Qualifications and 
qualification systems 
2017 
2018 
2019 
2020 
2021 
2022 

3.60m 
3.62m 
2.54m 
2.10m 
not 
available 
not 
available 

Anticipating Change 

2017 
2018 
2019 
2020 
2021 
2022 

0.08m 
0.17m 
0.10m 
0.07m 
0.16m 
0.09m 

Networking Knowledge 

2017 
2018 
2019 
2020 
2021 
2022 

0.21m 
0.23m 
0.04m 
0.15m 
0.35m 
0.21m 

Strategic and 
Operational 
Activities/networking 

2017 
2018 
2019 
2020 
2021 
2022 

0.57m 
0.47m 
0.52m 
0.11m 
0.11m 
0.28m 

Tools for OSH 
management 
2017 
2018 
2019 
2020 
2021 
2022 

0.22m 
0.33m 
0.24m 
0.32m 
0.35m 
0.30m 

Entrepreneurial 
learning and 
enterprise skills 
1.77m 
2017 
2.10m 
2018 
2.18m 
2019 
0.71m 
2020 

101 

 
 
 
 
 
Direct benefits (to consumers241 of the Agencies outputs) 

2021 
2022  

not 
available 
not 
available 

Stakeholders rating of 
cost-effectiveness of 
the Agency 

Stakeholders rating of 
cost-effectiveness of the 
Agency 

Stakeholders rating 
of cost-effectiveness 
of the Agency 

Stakeholders rating of 
cost-effectiveness of the 
Agency 

54% of stakeholders 
rated the Agencies cost-
effectiveness as high, 
20% as medium, and 
5% as low (the 
remainder were unsure 
(N=65)).  

32% of stakeholders rated 
the Agencies cost-
effectiveness as high, 20% 
as medium, and 5% as low 
(the remainder were unsure 
(N=175)).  

34% of stakeholders 
rated the Agencies 
cost-effectiveness as 
high, 13% as medium, 
and 4% as low (the 
remainder were 
unsure (N=91)). 

56% of stakeholders rated 
the Agencies cost-
effectiveness as high and 
24% as medium (the 
remainder were unsure 
(N=75)).  

Surveys 

VET systems and 
institutions 

75% of staff regard 
surveys to be either 
cost-effective to a large 
extent (53%) or a 
moderate extent (22%), 
with no staff finding 
them to be not at all 
cost-effective (the 
remaining 25% felt 
unable to comment, 
N=51). 

The monitoring reports 
produced by Cedefop’s 
using its ReferNet network 
data are highly rated by 
stakeholders, with 72% 
rating them as very or 
rather high quality 
(N=174). These outputs 
are also highly rated by 
staff, with 67% rating them 
as cost-effective to a large 
(51%) or moderate extent 
(16%, N=45). 

Employment, skills, 
and employability 
research 

Interviewees noted 
that the Skills Lab 
Network of Experts 
has been beneficial in 
terms of allowing the 
ETF to establish high-
level contacts with 
partner countries’ 
statistical offices, 
which makes access 
to data easier and 
quicker. By December 
2022, Skills Lab had 
more than 160 
members, covering 
the majority of partner 
countries. 

Facts and figures 

81% of staff rated 
European surveys as cost-
effective to a large extent, 
8% to a moderate extent, 
2% to a small extent, with 
the remainder not 
answering (N=48). 

79% of staff rated 
Research reports as cost-
effective to a large extent, 
10% to a moderate extent, 
2% to a small extent, with 
the remainder not 
answering (N=48).  

EU-OSHA interviewees 
also noted that ESENER 
had become a tool of 
reference for cross-country 
comparisons and has 
proven to be a survey of 
high relevance for 
stakeholders at both 
national and EU levels. 
FOPs found ESENER to be 
a useful tool to re-think and 
re-design their OSH 
policies, as well as to 
encourage businesses and 
workers to take action in 
response to specific risks.  

EU-OSHA’s bi-annual 
stakeholder surveys 

241  Users  of  the  Agencies’  outputs  include  decentralised  EU  Agencies,  EEA/ETFA  country  governmental 
institutions/Agencies, employer organisations, EU candidate country governmental institution or Agency, EU 
Member  State  governmental  institutions/Agencies,  NGO’s,  research  or  consultancy  organisations,  Trade 
unions, and private companies. 

102 

 
 
 
 
Corporate 
communication and 
infrastructure 

The majority (65%) of 
Eurofound’s staff regard 
Communication and 
dissemination activities 
as cost effective to a 
large extent (47%) or to 
a moderate extent 
(18%), with 2% finding 
them cost-effective to a 
small extent and no staff 
finding them not at all 
cost-effective (the 
remaining 33% felt 
unable to comment, 
N=51). 

between 2016-2022242 
found that the majority of 
respondents were satisfied 
with ESENER (85% in 
2016, 90% in 2018, 94% in 
2020, 95% in 2022).  

Communication and 
dissemination 

Communication and 
dissemination 

Communication and 
dissemination  

46% of staff surveyed 
found these to be cost-
effective to a large 
extent, 33% to a 
moderate extent, and 
15% to a small extent 
(4% did not find them 
at all cost-effective 
and 2% did not 
provide an answer, 
N=52). 

The vast majority of staff 
(90%) consider EU-
OSHA’s activities to be 
either largely cost-effective 
(69%), or moderately cost-
effective (21%). A small 
minority (2%) believed 
they were cost-effective to 
a small extent only, while 
8% were not able to answer 
the question, N=48. 

The PPMI 2022 User 
satisfaction survey243 
found that information 
disseminated through 
publications, online tools 
and databases is perceived 
as being provided in an 
attractive form (27% 
strongly agree, 63% agree) 
and meets the most 
prominent needs of users 
(31% strongly agree, 65% 
agree (N=507)).  

Interviewed 
representatives from the 
Executive and 
management boards and 
the international/European 
organisations emphasised 
their high level of 
satisfaction with Cedefop’s 
outputs and the unique 
nature of the findings that 
the Agency presents. 

40% of staff surveyed 
found communication and 
dissemination activities to 
be cost effective to a large 
extent, 29% to a moderate 
extent, and 9% to a small 
extent (2% did not find 
them at all cost-effective 
and 20% did not provide 
an answer, N=45). 

Events 

Events 

Events 

Anticipating Change 

61% of staff view events 
as cost-effective (35% 
to a large extent, and 
25% to a moderate 
extent), although 31% 
felt unable to comment, 
2% found them to be not 
at all cost effective, and 
6% only found them 

Stakeholders’ assessment 
of the quality and 
usefulness of the Agency’s 
events (based on the 
Agency’s KPI reporting):  

2017 97% 

2018 94% 

2019 96% 

54% of staff surveyed 
found these to be cost 
effective to a large 
extent, 27% to a 
moderate extent, and 
13% to a small extent 
(4% did not find them 
at all cost-effective 
and 2% did not 

The majority of survey 
respondents found the 
Agency to be successful in 
delivering foresight 
information either to a 
large or moderate extent 
(96% of staff survey 
respondents (N=48), 95% 
of stakeholder survey 
respondents (N=75), and 

242 Available to download here: https://osha.europa.eu/en/publications.  
243 PPMI (2022), User satisfaction survey 2022. 

103 

 
 
 
93% of OPC respondents, 
(N=41) during the 
evaluation period). 

Interviewees confirmed 
that foresight activity was 
used to inform national 
OSH strategies and 
national policy debates. 
The foresight reports 
themselves were also 
considered to be cost-
effective to a large extent 
by the majority of staff 
members surveyed (79%, 
N=48). 

cost-effective to a small 
extent (N=51). 

2020 94% 

2021 98% 

2022 95% 

provide an answer, 
N=52). 

53% of staff surveyed 
found the Agencies events 
to be cost effective to a 
large extent, 22% to a 
moderate extent, and 4% to 
a small extent (20% did 
not provide an answer, 
N=45). 

49% of staff surveyed 
found the Agencies peer 
learning activities to be 
cost-effective to a large 
extent, 18% to a moderate 
extent, and 4% to a small 
extent (29% did not 
provide an answer, N=45). 

Quality of life research  Outputs related to 
implementation of 
transparency and 
recognition tools 

The 2020 Eurofound 
user survey244 reported 
that 64% of respondents 
noted that this activity 
was relevant to their 
work (N=544). 

64% of staff surveyed 
found these to be cost 
effective to a large (42%) 
or moderate (18%) extent, 
and 4% to a small extent 
(36% did not provide an 
answer, N=45). 

Qualifications and 
qualification systems 

Tools for OSH 
Management 

A key initiative under 
this activity area is the 
Torino Process (TRP),  

In a 2018 PPMI 
evaluation245 
interviewees 
highlighted that the 
TRP provides 
valuable insights, 
contributes to policy 
dialogue, and informs 
the Agency on partner 
country needs. The 
process was also 
praised by partner 
country governments, 
EU institutions and a 
partner country civil 
society organisation, 
with a majority of all 
three stakeholder 
types finding it 
relevant to a large or 
to some extent (83%, 

73% of staff surveyed 
indicated that the tools 
were largely cost-effective, 
with 19% finding them 
moderately cost-effective 
and no staff finding them 
to be of little or not at all 
cost-effectiveness (8% felt 
unable to respond, N=48). 

The 2020 midterm 
evaluation of OiRA246 
found this to be highly 
efficient in terms of 
achieving its outputs and 
outcomes and that it had 
saved costs at the national 
level by providing readily 
available online risk 
assessment tools, thereby 
eliminating the need to 
develop such tools from 
scratch. EU-OSHA’s 
stakeholders’ 
assessments247 confirm the 

244 ICF (2021), User Feedback Survey 2020. 
245 PPMI (2018), Evaluation of the ETF Functions: Policy Delphi Results 1st round: June/July 2018. 
246 Ipsos (2020), Mid-Term evaluation of the OiRA activity - Final Report. A total of 40 interviews were conducted 
for  the  OiRA  evaluation,  covering  EU-OSHA  staff  and  contractors,  management  board  Members,  OiRA 
national  partners,  IRAT  partners,  EU  social  partners,  national  social  partners  and  European  Commission 
stakeholders.  

247 Bi-annual stakeholder survey 2016-2022. Satisfaction with OiRA over time  excludes do not know / cannot 

answer responses. 

104 

 
 
 
 
Employment and 
labour markets  

Reports 

81% and 77%, 
respectively, N=263). 
In 2019, 78% of 
partner countries were 
using the TRP for 
policy dialogue. 

growing satisfaction and 
popularity of OiRA, with 
satisfaction rates at 83% in 
2018, 92% in 2020, and 
93% in 2022.    

Expertise to EU 
project and 
programming cycle 

Promoting networking 
and coordination 

89% of survey 
respondents felt that 
Eurofound had achieved 
its objectives under this 
activity over 2017-2022 
to a large or moderate 
extent (N=116). 

The 2020 Eurofound 
user survey found that 
79% of respondents 
found this activity 
relevant for their work 
(N=544). 

Of staff surveyed, 73% 
found skills forecasting 
reports to be cost effective 
to a large or moderate 
extent, with this figure at 
76% for thematic research 
reports, 67% for 
monitoring report, 78% for 
country reports, and 76% 
for EU-wide study reports, 
N=45. 

37% of staff surveyed 
found these to be cost-
effective to a large 
extent, 40% to a 
moderate extent, and 
12% to a small extent 
(12% did not provide 
an answer, N=52). 

Social dialogue and 
industrial relations 

Outputs related to 
upskilling and reskilling 

89% of stakeholders 
rated this as very or 
rather good (N=65) 

53% of staff surveyed 
found the Agencies events 
to be cost-effective to a 
large extent, 22% to a 
moderate extent, 4% to a 
small extent (20% did not 
provide an answer, N=45). 

Supporting capacity 
building in partner 
countries 

33% of staff surveyed 
found these to be cost-
effective to a large 
extent, 37% to a 
moderate extent, and 
17% to a small extent 
(2% did not find them 
at all cost-effective 
and 12% did not 
provide an answer, 
N=52). 

81% of EU-OSHA staff 
survey respondents felt that 
the Agency was able to 
meet this objective to a 
‘very large extent’ (N=48), 
with approximately half of 
respondents in the 
stakeholder survey (52%, 
N=75) and the OPC (51%) 
reporting the same thing 
(N=41). 

Staff perceive events are 
being largely cost-effective 
(67%) or cost-effective to a 
moderate extent (23%). 2% 
found them cost-effective 
to a small extent and 8% 
were unsure (N=48). 

Networking knowledge 

84% of staff judge the cost-
effectiveness of EU-
OSHA’s online database as 
either largely cost-effective 
(52%) or as moderately 
cost-effective (31%). The 
remaining 17% felt unable 
to judge cost-effectiveness 
(N=48).  

Respondents to the 
stakeholder survey from 
Trade Unions indicated 
that one of the main 
achievements of the 
Agency in their eyes 
during 2017-2022 was the 
underpinning of EU 
Strategic Framework on 
Health and Safety at Work, 
which attests to the impact 
of EU-OSHA’s knowledge 
outputs (N=75).  

The provision of quality 
work and of updated 
information of health and 
safety topics was also 
mentioned in the 

105 

 
 
 
stakeholder survey as a key 
achievement of the 
Agency. The activity also 
exceeded its set targets 
from 2019 to 2022 in terms 
of usefulness, relevance, 
EU-added value and 
impact, based on the 
assessment of stakeholders. 
The assessment of 
usefulness rose from 83% 
to 94% over 2019 to 2022, 
while the other indicators 
remained stable. 

MB members and FOPs 
assessment of 
performance 

(KPI target 80%) 

Peer learning activities 

Policy advice to 
partner countries 

49% of staff surveyed 
found these to be cost-
effective to a large extent, 
18% to a moderate extent, 
and 4% to a small extent 
(29% did not provide an 
answer, N=45). 

42% of staff surveyed 
found these to be cost-
effective to a large 
extent, 35% to a 
moderate extent, and 
13% to a small extent 
(10% did not provide 
an answer, N=52). 

2017 

2018 

2019 

2020 

2021 

2022 

89% 

93% 

87% 

96% 

96% 

98% 

Skills and labour market  Monitoring and 

diagnostics reports 

Cedefop developed a tool 
for online vacancy analysis 
(Skills-OVATE) over the 
evaluation period. An ex-
post evaluation248 revealed 
that stakeholders greatly 
appreciate quick access to 
the data, the ability to 
implement comparisons of 
countries, and effective 
visualisations 

38% of staff surveyed 
found these to be cost-
effective to a large 
extent, 38% to a 
moderate extent, and 
10% to a small extent 
(2% did not find them 
at all cost-effective 
and 12% did not 
provide an answer, 
N=52). 

248 Intellera Consulting (2022). Real-time Labour Market Information on Skills Requirements: Setting up the EU 
system for online vacancy analysis Final Report. Cedefop Ex-Post Evaluation 2021 ROC/CTF/19/ETF/0001-
02/2021 

106 

 
 
 
 
 
 
 
 
 
ANNEX V:  STAKEHOLDER CONSULTATION – SYNOPSIS REPORT  

Chapter 1: Introduction 

This report provides an overview of all consultation activities and the input received through 
the  call  for  evidence  that  supported  the  evaluation  study  of  the  EU  agencies  Eurofound, 
Cedefop, ETF  and EU-OSHA (‘the Agencies’)  covering the period from 2017 to 2022. The 
legal basis of the evaluation is set out in the revised Founding Regulation of the three tripartite 
Agencies (Eurofound, Cedefop and EU-OSHA) and in Article 24(2) of the ETF’s Founding 
Regulation. 

The report is structured as follows:  

•  Chapter 2 presents an overview of the consultation strategy, including the distribution 

of stakeholder groups across the various consultation activities; 

•  Chapter 3 provides information on the consultation activities, including the approach 
taken for dissemination, as well as the sample size, and the strengths and limitations of 
each consultation; and 

•  Chapter 4 consolidates and discusses the findings of the consultation activities across 
the  five  evaluation  criteria  of  effectiveness,  efficiency,  coherence,  relevance  and  EU 
added value. 

Chapter 2: Overview of consultation strategy 

Several stakeholder groups were consulted to inform the study’s findings. Table 1 shows an 
overview of the consulted stakeholder groups across the various consultation activities.  

Table 1: Consulted stakeholder groups across the consultation activities 

Stakeholder group  

High-level 
interviews 

Interviews  Public 

consultation 

Stakeholder 
survey 

Staff 
survey 

Validation 
focus 
groups 

Have 
your 
say 
portal 

European Commission Directorates-General 

EU agencies, authorities or committees  

EU institutions / experts excluding the Commission 

International institutions (OECD, World Bank, etc.) 

✓ 

✓ 

✓ 

✓ 

Agencies’ staff members 

Agencies’ management / governing board  

Agencies’ executive board members / directors 

Agencies’ stakeholders 

General public 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

107 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
As  Table  2  shows,  the  study  incorporated  900  stakeholder  contributions  across  the  various 
consultation activities249.  

Table 2: Number of stakeholders consulted 

High-level 

Interviews 

Public 

Call for 

Stake-

Staff 

Validation 

Total 

interviews 

consul-

tation 

evidence 

holder 

survey 

focus 

survey 

groups 

Number 

22 

143 

101 

15 

364 

196 

59 

900 

Chapter 3: Consultation activities 

This  section  provides  a  detailed  account  of  the  objectives,  the  dissemination  methods,  the 
sample  size  and  composition,  as  well  as  the  strengths  and  limitations  of  all  consultation 
activities.  

High-level interviews 

A total of 22 high-level interviews were conducted between March and July 2023 involving 
representatives from the following stakeholder groups: 

•  European  Commission  Directorates-General,  including  DG  Budget,  DG  Education, 
Youth,  Sport  and  Culture,  DG  Employment,  Social  Affairs  and  Inclusion,  Eurostat, 
and 
DG Human  Resources 
DG Neighbourhood and Enlargement Negotiations; 

International  Partnerships 

and  Security,  DG 

•  EU  institutions  excluding  the  European  Commission  –  these  include  the  European 
Economic and Social Committee, as well  as the  European  Parliament Committee on 
Employment and Social Affairs or experts appointed by the European Parliament; 
•  EU decentralised agencies, including the European Centre for Disease Prevention and 
Control,  the  European  Environment  Agency,  the  European  Economic  and  Social 
Committee, the European Institute for Gender Equality, the European Labour Authority 
and the EU Agency for Fundamental Rights; and 
international  organisations  including  the  EU  Agencies  Network,  the  International 
Labour Organization, the Organisation for Economic Co-operation and Development, 
UNESCO, UNICEF and the United Nations Industrial Development Organization. 

• 

The  aim  of  these  interviews  was  to  sense-check  the  intervention  logic,  gain  initial  insights 
regarding the merit and relevance of the Agencies’ activities, and identify the primary areas of 
their impact. The selection of representatives within the named stakeholder groups was made 
in  collaboration with  Unit G5 of DG  Employment, Social Affairs and  Inclusion, which was 
responsible for the study. 

249 Some of these contributions might not be unique, as the same stakeholders might have been consulted through 

multiple activities. 

108 

 
 
 
 
 
Interviews 

A  total  of  143  interviews  were  conducted  with  the  Agencies’  staff  members  and  their 
stakeholders between February and June 2023. The purpose of these interviews was to add to 
the desk-based research by providing in-depth insights into the Agencies’ performance across 
the  five  evaluation  criteria.  Interviews  were  well-suited  to  this  purpose  as  their  qualitative 
information was difficult to capture in the desk-based research or in the close-ended questions 
used in the rest of the consultation activities.  

While  most  interviews  were  conducted  on  the  Agencies’  premises,  some  were  conducted 
online. Specifically, 30 interviews were conducted for Eurofound, 46 interviews for Cedefop, 
33 interviews for the ETF, and 34 interviews for EU-OSHA. The number of interviews was 
subject to staff and stakeholders’ availability. 

Interpreting the interview data was an iterative process. On the one hand, interview notes from 
different staff members were compared to find areas of consensus or disagreement. On the other 
hand, interview notes were compared with the available evidence from the desk-based research 
and the other consultation activities, to assess the weight of the evidence. After this process, 
interview findings were blended in the evaluation report. 

Public consultation 

A public consultation was available in all official EU languages for a period of 12 weeks, from 
2 March 2023  to  25 May 2023.  The  consultation,  in  contrast  to  the  staff  and  stakeholder 
surveys,  focused  on  the  evaluation  criteria  that  were  more  suitable  to  seek  input  on  from 
respondents like the general public. It therefore excluded questions about the efficiency of the 
Agencies. 

For  the  consultation’s  dissemination,  ‘snowball  sampling’  was  used.  The  four  EU  Agencies 
used email campaigns and shared the consultation through their websites, newsletters, social 
media, and email signatures. Similarly, the research team disseminated the public consultation 
to the Members of the European Parliament Committee on Employment and Social Affairs, to 
relevant EU-level associations and to social media.  

The total sample size after cleaning the dataset according to the European Commission’s rules 
for feedback and suggestions was 101 responses. As a result, 44 responses were received for 
Cedefop, 41 for EU-OSHA, 32 for the ETF, and 31 for Eurofound (Figure 1). One limitation of 
this dataset was that it did not enable subgroup analysis. In addition, it should be noted that the 
public consultation was not based on random sampling from a finite population and therefore 
its results were not representative of the EU population.  

109 

 
 
Figure 1: Number of public consultation responses per agency250 

Number of responses per Agency

44

41

31

32

a
t
a
d

t
n
u
o
C

50
45
40
35
30
25
20
15
10
5
0

Eurofound

Cedefop

ETF

EU-OSHA

Most respondents (81 out of 101) were from within the EU, from 24 different Member States. 
The highest number of responses came from Spain (12 out of 101) and Belgium (10 out of 101). 
A total of 20 responses (out of 101) came from outside the EU, from 14 non-EU countries. The 
non-EU country with the most responses was Türkiye with 4, followed by the UK with 3, and 
Armenia with 2.  

Figure 2: Number of public consultation respondents per country 

Country distribution (n=101) 

6

6

5

5

4

3

3

3

3

3

3

2

2

2

2

11

7

4

3

2

12

10

14
12
10
8
6
4
2
0

250 The number of responses was larger than the number of respondents, as respondents could answer for more 

than one agency. 

110 

 
 
 
 
 
 
Figure 3: Types of respondents per stakeholder group 

Types of Respondents Distribution (n=101)

24

21

10

9

8

8

11

6

4

A wide range of stakeholders replied to the consultation. The highest number of responses (24 
out of 101) came from individual EU citizens, followed by those working for public authorities 
(21 out of 101). When combined, businesses and business associations represented the third-
largest  category of respondents  (14 out  of 101). The remaining  respondents  (42 out  of 101) 
included  trade  unions,  academic/research  institutions,  NGOs,  non-EU  citizens,  and  others 
(Figure 3).  

Call for evidence 

Simultaneously with the public consultation, a call for evidence was published on the European 
Commission’s  ‘Have  your  say’  web  portal.  The  call  for  evidence  gathered  15  unique 
contributions from business associations, academic/research institutions, trade unions, NGOs, 
EU and non-EU citizens, and others. The feedback from these organisations and individuals has 
been incorporated into Chapter 4 of this report.  

Stakeholder and staff survey 

Simultaneously  with  the  public  consultation,  a  targeted  survey  was  administered  to  the 
Agencies’ staff and stakeholders. The survey was accessible in English, German and French. It 
was  sent  to  the  Agencies’  contact  points  together  with  the  public  consultation  for  of 
dissemination.  The  survey  questionnaire  was  distributed  via  a  unique  link  that  included  a 
separate route for the staff members and stakeholders of each of the four Agencies, resulting in 
a total of eight surveys. 

The survey gathered data concerning all five evaluation criteria. It served a dual purpose: on 
the  one  hand,  quantifying  diverse  aspects  of  the  Agencies’  performance  in  relation  to  the 

111 

 
 
 
evaluation  criteria,  and,  on  the  other  hand,  gathering  qualitative  input  to  strengthen  the 
evaluation process. 

The surveys gathered a total of 560 unique contributions.  Figure  4 shows the breakdown of 
responses by agency whereas Table 3 shows the sample composition of the stakeholder survey. 
The sample composition of the stakeholder survey presented similar challenges for subgroup 
analysis as seen in the public consultation’s sample composition. In addition, similarly to the 
public  consultation,  the  stakeholder  survey  did  not  rely  on  random  sampling  from  a  finite 
population. Therefore, its results should not be generalised. The same was not true for the staff 
survey, but because of its low response rate, its findings were not representative either. 

Figure 4: Number of responses to the stakeholder/staff surveys 

Table 3: Sample composition of stakeholder survey 

Stakeholder group 

Eurofound 

Cedefop 

Business organisation / private company 

Decentralised EU agency 

EEA/EFTA country governmental 
institution or agency 

Employer organisation 

EU candidate country governmental 
institution or agency 
EU Member State governmental 
institution or agency 

European Commission 

International  non-EU  organisation  (e.g. 
OECD, ILO, UNESCO, etc.) 

NGO 

Other (please specify) 

4 
(6%) 
1 
(2%) 

1 
(2%) 

14 
(22%) 
1 
(2%) 
13 
(20%) 

5 
(8%) 
1 
(2%) 

3 
(5%) 
5 
(8%) 

13 
(7%) 
1 
(1%) 

5 
(3%) 

10 
(6%) 
3 
(2%) 
27 
(16%) 

1 
(1%) 
2 
(1%) 

11 
(6%) 
42 
(24%) 

112 

ETF 

3 
(3%) 
1 
(1%) 

5 
(5%) 

2 
(2%) 
4 
(4%) 
3 
(3%) 

13 
(14%) 
14 
(15%) 

2 
(2%) 
13 
(14%) 

EU-OSHA 

3 
(4%) 
2 
(3%) 

2 
(3%) 

10 
(13%) 
2 
(3%) 
27 
(36%) 

0 
(0%) 
0 
(0%) 

2 
(3%) 
5 
(7%) 

 
 
 
 
 
Other  EU  institution  (e.g.  CoR,  EESC, 
etc.) 
Research or consultancy organisation 

Trade union 

1 
(2%) 
11 
(17%) 
5 
(8%) 

Vocational education and training provider  0 

Total 

(0%) 
65 
(100%) 

0 
(0%) 
30 
(17%) 
9 
(5%) 
20 
(11%) 
174 
(100%) 

14 
(15%) 
8 
(9%) 
1 
(1%) 
8 
(9%) 
91 
(100%) 

0 
(0%) 
2 
(3%) 
20 
(27%) 
0 
(0%) 
75 
(100%) 

Validation focus groups 

The validation focus groups aimed to validate the draft evaluation findings and to fill in the 
remaining knowledge gaps. Five validation focus groups were conducted: four covering each 
of the Agencies,  and one covering all Agencies  collectively.  The  one covering all Agencies 
collectively  focused  on  issues  around  the  coherence  within  and  between  the  Agencies,  and 
within this, on issues around their efficiency. The focus groups took place as a single online 
event on 18 September 2023. In total, 59 participants participated in the focus groups, excluding 
the members of the research team.  

Chapter 4: Summary of results 

This section brings together the findings of the evaluation’s consultation activities, as presented 
in the previous chapter. Overall, the different consultation activities were cross-referenced with 
each other, and their findings underwent further validation through the validation focus groups. 
By  employing  this  triangulation  approach,  the  findings  presented  in  this  section  acquired 
significant  reliability,  even  in  light  of  the  non-representative  nature  of  the  samples  across 
various consultation activities. 

Effectiveness 

Across all Agencies, responses to effectiveness-related questions in the public consultation 
were predominantly positive. Negative responses were rare, ranging from 0% to a maximum 
of 20% such as in  the  case of assessing  effectiveness  against Russia’s  war against Ukraine. 
However,  many  respondents,  across  questions  and  Agencies,  often  opted  for  moderate 
responses, suggesting room for improvement in various effectiveness dimensions.  

In summary, respondents broadly found the Agencies’ services to have been of high quality. 
For Eurofound, the services that were found to have the highest quality were its European-level 
surveys and its research outputs. For Cedefop, services such as its EU-wide study reports, its 
events and its monitoring and diagnostics reports were appraised. As regards the ETF, its role 
in  supporting  capacity  building  in  partner  countries  and  offering  policy  advice  received 
recognition.  Lastly,  for  EU-OSHA,  services  involving  communication  and  dissemination 
activities  and  research  reports  were  highlighted  as  areas  of  excellence.  These  findings  were 
consistent  across  the  public  consultation  and  the  stakeholder  survey  and  were  corroborated 
during the interviews with the Agencies’ staff members and their relevant stakeholders. 

113 

 
 
 
 
According to the public consultation, albeit with individual differences, the main purposes for 
which respondents were using the Agencies’ services were: (i) to familiarise themselves with 
developments  in  the  Agencies’  respective  areas:  (ii)  for  peer  learning  and  identifying  good 
practices; and (iii) on some occasions, to inform policy development at the national level. The 
survey results were consistent with the public consultation. They also highlighted a strong use 
of the Agencies’ guidance documents and tools by respondents in order to inform their work. 
This use was confirmed by the interview data, with stakeholders referencing examples such as 
Eurofound’s European working conditions surveys, Cedefop’s network of independent experts 
in  lifelong  career  guidance  and  development,  the  ETF’s  rapid  education  diagnosis  and 
EU-OSHA’s healthy workplaces campaigns.  

The  corresponding  activities  and  outputs  of  the  Agencies’  services  also  appear  to  have 
effectively addressed the respondents’ needs. Some Agencies seem to have fared slightly better 
compared to others in this regard. The same pattern held true for the Agencies’ objectives, as 
the  majority  of  respondents  believed  they  had  been  achieved  to  a  significant  extent.  The 
contributions to the call for evidence also confirmed that the Agencies’ services met the needs 
of their stakeholders. As an illustration, a national academic/research institution in the field of 
medicine from Sweden reported to have frequently used statistics and reports from Eurofound 
and  EU-OSHA  and  to  have  received  these  Agencies’  support  in  organising  national-level 
events. EU-OSHA has also received praise from Danish, Spanish and French trade unions for 
producing  balanced  and  relevant  scientific  knowledge  on  occupational  safety  and  health. 
However, some suggestions have been made, indicating that there was potential for EU-OSHA 
to further assist Member States in improving the data they provide, through capacity-building 
activities and direct assistance. 

114 

 
 
 
Figure 5: To what extent did the Agencies’ overall activities and / or outputs meet your 
needs? 

The  public  consultation  also  revealed  patterns  in  the  characteristics  of  the  Agencies  that 
respondents  highly  valued.  Despite  individual  variations,  the  Agencies’  European  coverage, 
quality  of  information,  and  specific  thematic  knowledge  were  consistently  reported  as  the 
aspects most valued by respondents. These traits were followed by the quality of the Agencies’ 
methodologies and their tripartite nature. As Figure 5 shows, the findings of the stakeholder 
survey  were  consistent  with  the  public  consultation  findings,  while  similar  views  were 
expressed in the rest of the consultation activities. In particular, stakeholders often commented 
on  the  high  added  value  of  the  Agencies’  European  coverage,  quality  of  information  and 
specific thematic knowledge (Figure 6). In addition, they highlighted the effectiveness of the 
Agencies’  tripartite  nature  in  terms  of  consolidating  diverse  perspectives  and  increasing  the 
acceptability of the Agencies’ research at national level.  

115 

 
 
 
 
Figure 6: Which characteristics of the Agencies’ work did you value the most? 

Overall, respondents  to  the public consultation  expressed positive views  on the focus  of the 
Agencies,  although  to  varying  degrees.  For  instance,  a  greater  proportion  of  respondents 
reported that Eurofound and Cedefop were directing their efforts towards areas with the greatest 
need  and  potential  impact,  in  comparison  to  the  ETF  and  EU-OSHA,  although  for  all  four 
Agencies, most respondents expressed positive views. 

Finally, regarding the Agencies’ effectiveness in terms of addressing the needs that emerged 
from  the  two  big  crises  during  the  evaluation  period,  i.e.  COVID-19  and  Russia’s  war  of 
aggression  against  Ukraine,  respondents’  views  did  not  make  it  possible  to  draw  definite 
conclusions, although the overall picture appeared to be broadly positive. 

On the one hand, the Agencies were broadly successful in terms of providing useful support to 
address the challenges posed by COVID-19, with consulted stakeholders highlighting several 
examples of the Agencies’ work that helped in this direction. On the other hand, a sizeable share 
of responses suggested that the Agencies’ work has only moderately addressed the issues that 
emerged from COVID-19.  

116 

     324143Tripartite governance/ specific thematicknowledgeEuropean coverageQuality of informationEurofound (n=65)8598124Specific thematicknowledgeEuropean coverageQuality of informationCedefop (n=174)454956Quality ofmethodologiesSpecific thematicknowledgeQuality of informationETF (n=91)444446Awareness raisingQuality of informationTripartite governanceEU-OSHA (n=75) 
 
 
Figure 7: To what extent were the Agencies’ responsive to pressures arising from 
COVID-19?251 

Positive examples that were mentioned during the interviews included Eurofound’s COVID-19 
Policy Watch, which collated information on the responses of governments and social partners 
to the COVID-19 crisis and Russia’s war against Ukraine; Cedefop’s information hub on how 
countries and stakeholders were coping with vocational education and training (VET) during 
COVID-19;  and  the  ETF’s  partnership  with  Eurofound  on  Eurofound’s  living,  working  and 
COVID-19 survey. 

As  regards  the  Agencies’  effectiveness  in  terms  of  addressing  the  needs  that  emerged  from 
Russia’s  war  of  aggression  against  Ukraine,  respondents  have  been  somewhat  more  modest 
across  all  consultation  activities.  This  is  possibly  due  to  the  low  affinity  of  the  Agencies’ 
mandate  with  those  needs.  Despite  this,  stakeholders  also  pointed  to  positive  examples  of 
support. All in all, it appears that the ETF has fared better than the other Agencies in this aspect, 
closely followed by Eurofound and Cedefop, while EU-OSHA ranked last.  

251 The figure excludes respondents who selected ‘not applicable’. 

117 

 
 
 
 
Figure 8: To what extent were the Agencies’ responsive to pressures arising from 
Russia’s war of aggression against Ukraine?252 

Efficiency 

The  prevailing  opinion  among  stakeholders  was  that  the  Agencies  have  operated  in  a 
cost-effective manner (Figure 9). Interview and focus group data suggested that the Agencies 
were continuously seeking to increase efficiency in delivering their services, e.g. by performing 
joint  actions  between  them  and  with  other  institutions,  by  streamlining  monitoring  and 
reporting, and other measures. However, it became evident that the Agencies were often under 
pressure to deliver their work programmes and that further resources could have helped them 
maximise  their  impact  on  the  ground,  e.g.  for  EU-OSHA,  by  increasing  their  missions  in 
countries that have less developed occupational safety and health (OSH) structures and driving 
the  EU  OSH  agenda  in  these  countries.  Figure  10  also  points  to  an  issue  around  the 
appropriateness of available resources, as it demonstrates that the allocation of human resources 
was perceived as too low among staff members across all four Agencies. Importantly, almost 
no staff members reported that this allocation was too high. In this context, it was noted that 
compensating for the lack of resources by continuously improving efficiencies might not be 
sustainable  in  the  long  run.  In  addition,  such  an  approach  might  undermine  the  Agencies’ 
capacity to strategically plan their future actions.  

252 The figure excludes respondents who selected ‘not applicable’. 

118 

 
 
 
 
Figure 9: Stakeholders’ perception of the Agencies’ efficiency: 

How do you rate the cost-effectiveness of the Agencies’ operations in 2017-2022?253 

Figure 10: How do you perceive the amount of human resources allocated to fulfil the 
functions of your department or unit in the period 2017-2022?254 

Despite the resource constraints, staff members of all four Agencies considered that there were 
adequate  mechanisms  in  place  to  ensure  accountability  towards  stakeholders,  maintain 

253 Replies to the evaluation stakeholder survey. The sample size excludes respondents who selected ‘do not know’ 

or who did not answer. 
254 Replies to the staff survey. 

119 

 
 
 
 
 
 
transparency with stakeholders and the general public, and conduct suitable assessments of their 
performance. However, respondents did not select extreme values when making these claims, 
suggesting that there might have been some potential for improvement. 

Finally, Figure 11 shows that staff members were also overall positive about the appropriateness 
of the size and composition of the Agencies’ management or governing boards. Specifically, 
while it was recognised that the tripartite nature of the Agencies entails a certain administrative 
burden, it was noted that this tripartism’s role in synthesising diverse viewpoints and rendering 
the  Agencies’  research  more  acceptable  to  national-level  stakeholders  counterbalanced  this 
drawback. 

Figure 11: How would you rate the appropriateness of the size and composition of the 
management / governing board? 

Coherence 

Similar to effectiveness, the distribution of the public consultation responses for all Agencies 
leaned  towards  positive  values  across  all  questions  on  coherence.  Instances  of  responses 
suggesting that the Agencies’ work has been either incoherent or very  incoherent have been 
marginal, ranging from as low as 0% to a maximum of 6%.  

120 

 
 
 
 
 
Figure 12: To what extent do you think that the Agencies’ mandates and activities are 
coherent with other relevant EU programmes/initiatives? 

60%

50%

40%

30%

20%

10%

0%

51%

32%

32%

28%

45%

45%

47%

39%

2%

0%

0%

0%

0%

0%

6%

2%

Very coherent

Coherent

Incoherent

Very incoherent

Eurofound
N = 31

Cedefop
N = 44

ETF
N = 32

EU-OSHA
N = 41

Nevertheless,  a  notable  share  of  respondents  opted  for  moderately  positive  values  (e.g. 
coherent) instead of  extreme positive values. (e.g. very coherent),  suggesting  that there was 
some  room  for  further  improving  different  dimensions  of  coherence.  At  the  same  time,  a 
significant proportion of respondents was unable to provide an answer (from 32 to 61%). This 
can be explained by the technical nature of these questions. 

Despite this, the overall picture that emerged from the public consultation and the interviews 
was that during the evaluation period, the Agencies were often seeking opportunities to tap into 
shared opportunities to increase their effectiveness and efficiency, and, conversely, to decrease 
overlap. Participants in the focus groups confirmed this observation and pointed to the visibility, 
complementarity, and outreach gains the cooperation brought. However, they also stressed that 
cooperation is not necessarily associated with efficiency gains and that in practice, they work 
better  when  they  come  in  a  bottom-up  way,  via projects,  compared  to  a  formalised  way.  In 
addition to seeking collaboration opportunities among themselves, the four Agencies were also 
exploring similar opportunities with other agencies. For example, in the call for evidence, the 
European Centre for Disease Control expressed a positive view on its collaboration with the 
four Agencies through the work of the EU agencies sub-network on scientific advice as well as 
through bilateral cooperation on issues of mutual concern. 

All in all, joint publications, joint data collection and joint information- or expertise-sharing 
received positive ratings across various stakeholder groups across both the public consultation 
and  the  stakeholder  survey.  Sub-group  analysis  on  these  surveys  did  not  yield  conclusive 
patterns when it came to the quality of these joint activities as perceived by different stakeholder 
groups. 

121 

 
 
 
Relevance 

All in all, for the four Agencies, most respondents suggested that they have been on the ‘right 
track’ in terms of addressing issues that were relevant to their stakeholders. This finding was 
consistent across the public consultation and the stakeholder survey. 

Figure 13: How relevant, if at all, were the Agency’s overall activities and/or outputs to 
your work in 2017-2022? 

Interview accounts and contributions to the call for evidence corroborated this observation, with 
stakeholders providing numerous examples of how different activities and their corresponding 
outputs  were  relevant  to  their  needs.  For  example,  non-EU  citizens  have  reported  using  the 
Agencies as points of reference for various matters. They have mentioned Cedefop as a point 
of reference for VET-oriented initiatives, and Eurofound as a point of reference for collecting 
EU-level  information,  e.g.  through  the  European  working  conditions  telephone  survey. 
Furthermore, a trade union from Denmark reported that the Agencies had clear relevance  to 
their work during the evaluation period. 

Nevertheless, when grouping respondents who viewed the overall activities and/or outputs of 
the Agencies as moderately, slightly or not at all relevant to their work, it becomes evident that 
there was noticeable room for improvement. In the public consultation, this subset accounted 
for 22% of respondents for EU-OSHA, 32% for Eurofound, 49% for Cedefop and 53% for the 
ETF. In this context, one contribution suggested that Cedefop could improve its relevance by 
further  promoting  some  strands  of  its  work  (e.g.  Europass,  ECVET,  ESCO,  EQAVET, 
EQF/NQFs).  According  to  interview  data,  the  same  observation  held  true  for  the  other 
Agencies. 

Similar  trends  emerged  in  terms  of  the  relevance  of  the  services,  activities  and  outputs 
introduced by the Agencies in response to COVID-19. In this context, for EU-OSHA (54%) 

122 

 
 
 
 
and  Eurofound  (31%),  most  respondents  to  the  public  consultation  found  these  offerings  to 
align with their needs. This was less pronounced for Cedefop (16%) and the ETF (13%), which 
might be due to the closer alignment of the mandates of EU-OSHA and Eurofound with the 
emerging needs from COVID-19. Similar patterns were identified in the stakeholders' survey, 
although respondents in this survey generally displayed a more positive outlook. Despite these 
observations, a notable share of respondents across all Agencies indicated that such services, 
activities or outputs moderately met their needs, indicating that despite the numerous positive 
examples, there might have been some room for some improvement. 

Regarding both crises, the Agencies’ staff maintained the view that there was no need to amend 
the Agencies’ mandates. 

Regarding the relevance of the services, activities and outputs introduced by the Agencies in 
response to Russia’s war of aggression against Ukraine, patterns were less distinct, with a large 
share of respondents – ranging from a minimum of 32% to  a maximum of 63% in both the 
public  consultation  and  the  stakeholder  survey  –  opting  not  to  provide  an  answer,  therefore 
limiting  the  conclusions  drawn  to  a  small  number  of  participants.  Nevertheless,  overall, 
respondents  expressed  more  reserved  opinions  on  the  Agencies’  relevance  in  this  context, 
which, again, might be due to the low affinity of the Agencies’ mandates with the needs that 
emerged.  Despite  this,  positive  examples  of  support  were  cited  for  all  Agencies,  including 
Cedefop’s  lifelong  guidance  policy  for  Ukrainian  refugees  and  the  ETF’s  development  of 
ESCO in Ukrainian. 

Finally, regarding trends that might affect  the future relevance of the Agencies, the findings 
were once again consistent across the public consultation and the stakeholder survey. In both, 
most respondents considered that digitalisation and technological development, climate change, 
as well as demographic change and migration were the key trends likely to affect the Agencies’ 
work in the future. In addition to these trends, the focus groups also pointed to re-globalisation, 
geopolitics and external crisis as some megatrends that might affect the work of the Agencies. 

EU added value  

The four Agencies have helped bring about changes that would have been difficult to achieve 
otherwise. Specifically, most respondents in the public consultation and the stakeholder survey 
held the view that national-level organisations were not equipped to take over the role of the 
Agencies. This sentiment was more pronounced for Eurofound and EU-OSHA, compared to 
Cedefop and the ETF. Respondents also expressed similar, albeit less strong views regarding 
the  ability  of  European-  and  international-level  organisations  to  substitute  the  work  of  the 
Agencies. Similar views were also shared during the focus groups. 

In this context, the Agencies’ work has had a clear added value in Member States that lacked 
resources or had no equivalent national competences. For stakeholders in these Member States, 
the  Agencies’  outputs  frequently  served  as  the  only  available  reference  points.  At  the  same 
time, the Agencies’ outputs also proved valuable to stakeholders in countries with equivalent 
national  competences  in  place.  These  stakeholders  were  able  to  supplement  their  national 
research  by  exploring  a  topic  from  a  different  angle,  but  also  to  include a  comparative,  EU 
dimension to their research.  

123 

 
 
Overall,  the  Agencies’  unique  ability  to  produce  work  encompassing  all  EU  countries  and 
provide time-series data emerged as clear sources of added value across several consultation 
activities,  including  interviews.  Other  key  strengths  that  have  been  highlighted  included  the 
Agencies’ focus on EU policy needs, the development of repositories and tools that were shared 
or were made accessible across Europe, as well as the set-up of pan-European networks.  

Overall, across various stakeholder consultation activities, the work of the Agencies has been 
reported to provide the highest added value to the European Commission and to EU Member 
State public authorities. In most cases, organisations representing employers and employees, 
academic institutions as well as education and training providers took similar positions on the 
perceived benefits they derived from the Agencies. These results might hint that the Agencies 
could perhaps strengthen their visibility to the general public, a point emphasised in multiple 
consultation  activities,  including  the  call  for  evidence.  Discussions  in  the  focus  groups 
highlighted that presenting the Agencies’ work at local level could help in this direction. 

Finally,  respondents  cited several  examples of activities that  have brought  added value.  For 
Eurofound,  these  included  its  pan-European  surveys  including  the  European  working 
conditions surveys, the European company surveys, and the European quality of life surveys; 
for Cedefop, these included its expert networks as well as its reports focusing on forecasting 
skills and future labour market demands, as well as its reports on the future of  VET; for the 
ETF,  these  included  its  capacity-building  activities  in  partner  countries;  and  for  EU-OSHA, 
these included the healthy workplaces campaigns, ESENER, the Online Interactive Assessment 
Tools (OiRA) and others. 

124 

 
 
 
 
ANNEX VI: AGENCIES INTERVENTION LOGICS 

1.  Eurofound 

The intervention logic of Eurofound is presented in Figure 1. It was produced by reviewing the 
Agency’s key strategic documents and external evaluation reports, listed below:  

•  Revised Founding Regulation (Regulation (EU) 2019/127)  

•  Eurofound Programming document 2017–2020: work programme 2020  

•  Eurofound Programming document 2021–2024: Towards recovery and resilience 

•  Eurofound consolidated annual activity report 2020 

•  Evaluation of the EU Agencies under the remit of DG Employment: Eurofound, Cedefop, 

ETF and EU-OSHA (European Commission, 2018)  

2.  Cedefop 

The intervention logic of Cedefop is presented in Figure 2. It was produced by reviewing the Agency’s 
key strategic documents and an external evaluation report. The key documents reviewed are listed 
below:  

•  Revised Founding Regulation (Regulation (EU) 2019/128) 

•  Evaluation of the EU Agencies under the remit of DG Employment: Eurofound, Cedefop, 

ETF and EU-OSHA (European Commission, 2018)  

•  Programming documents: 

o  Single Programming Document 2022-24  

o  Programming document 2021-2023  

o  Programming document 2020-2022  

o  Programming document 2019-2021  

o  Programming document 2018-2020 

3.  ETF 

The intervention logic of the ETF is presented in Figure 3. It was produced by reviewing the Agency’s 
key strategic documents and an external evaluation.  

•  Council Regulation (EC) No 1360/90, amended by the recast ETF Regulation (1339/2008) 

•  Evaluation of the EU Agencies under the remit of DG Employment: Eurofound, Cedefop, 

ETF and EU-OSHA (European Commission, 2018)  

•  Programming documents: 

o  Single Programming Document 2022-24 - Work Programme 2022 

o  Single Programming Document 2021-23 - Work Programme 2021 

o  Single Programming Document 2020-22 - Work Programme 2020 rev. 2 

o  Single Programming Document 2017-2020 - Annual Work Programme 2019 

125 

 
 
o  Single Programming Document 2017-2020 - Annual Work Programme 2018 

o  Single Programming Document 2017-2020 (Includes WP 2017 and first outline of 

WP 2018) 

4.  EU-OSHA 

The intervention logic of EU-OSHA is presented in Figure 4. It was produced by reviewing the 
Agency’s key strategic documents and an external evaluation.  

•  Revised Founding Regulation (Regulation (EU) 2019/126) 

•  EU Strategic Framework for Health and Safety at Work 2021-2027 

•  EU-OSHA Multiannual Strategic Programme 2014-2020 

•  Evaluation of the EU Agencies under the remit of DG Employment: Eurofound, Cedefop, 

ETF and EU-OSHA (European Commission, 2018)  

•  EU-OSHA’s corporate strategy 2018–2023 / EU-OSHA Multiannual Strategic Programme 

2018-2023 

•  Single Programming Document 2021-2023 

5.  Cross-cutting activities intervention logic 

The intervention logic of cross-cutting activities is presented in Figure 5. It was produced by 
reviewing the above mentioned strategic documents and the previous evaluation of the four Agencies. 
The main documents are also listed below: 

•  Regulation (EU) No 2019/127 of the European Parliament and of the Council of 

16 January 2019, OJ L 30 of 16 January 2019 

•  Regulation (EU) 2019/128 of the European Parliament and of the Council of 16 January 2019 
establishing a European Centre for the Development of Vocational Training (Cedefop) and 
repealing Council Regulation (EEC) No 337/75 

•  Council Regulation (EC) No 1360/90, amended by the recast ETF Regulation (1339/2008) 

•  Regulation (EU) 2019/126 of the European Parliament and of the Council of 16 January 2019 
establishing the European Agency for Safety and Health at Work (EU-OSHA), and repealing 
Council Regulation (EC) No 2062/94. OJ L 30, 31.1.2019  

•  Evaluation of the EU Agencies under the remit of DG Employment: Eurofound, Cedefop, 

ETF and EU-OSHA (European Commission, 2018)  

126 

 
 
 
 
 
Figure 1: Intervention logic of Eurofound 

127 

 
 
 
Figure 2: Intervention logic of Cedefop 

128 

 
 
 
 
Figure 3: Intervention logic of the ETF 

129 

 
 
 
 
 
 
Figure 4: Intervention logic of EU-OSHA 

130 

 
 
 
 
 
 
Figure 5: Intervention logic of cross-cutting activities 

131 

 
 
ANNEX VII: FIGURES AND TABLES  

Figure 1: Changes in staff numbers in the four Agencies, 2016-2022255 

 Source: Agencies consolidated annual activity reports, 2016–2022. 

The staff numbers employed by all four Agencies remained largely stable during the evaluation 
period. EU-OSHA is the smallest Agency with 65 staff members in 2022 and the ETF was the 
largest with 134 staff members employed at the end of the evaluation down from 141 during 
the baseline period in 2016. Cedefop staff numbers reduced slightly from 121 in 2016 to 116 
in 2022, while staff numbers at Eurofound remained largely the same at 110 in 2016 and 108 
in 2022 but were slightly higher during the evaluation period. 

Table 1: Total staff numbers by organisation and year256 

Agency 

2016 

2017 

2018 

2019 

2020 

2021 

2022 

Eurofound 

CEDEFOP 

ETF 

110 

121 

141 

116 

116 

134 

113 

116 

135 

120 

114 

139 

117 

116 

133 

116 

120 

137 

108 

116 

134 

255 All figures presented in the table represent the state of employment as of 31 December of the respective year. 
The staff count includes members outside the establishment plan, such as contract agents, seconded national 
experts, and structural service providers. 

256 All figures presented in the table represent the state of employment as of 31 December of the respective year. 
The staff count includes members outside the establishment plan, such as contract agents, seconded national 
experts, and structural service providers. 

132 

 
 
 
 
 
EU-OSHA 

65 

64 

65 

63 

63 

64 

65 

Source: Consolidated Annual Activity Reports and Programming Documents, 2016–2022. 

Box 1: ETF Extra-subsidy projects in 2022 

•  Service level agreement (SLA) with DG EMPL to support the international dimension 

of Centres of Vocational Excellence (CoVEs) in 2022 and 2023257; 

•  Contribution agreement with the Delegation of the EU to the Republic of Kazakhstan 
for  the  project  Dialogue  and  Action  for  Resourceful  Youth  in  Central  Asia 
(DARYA)258; 

•  Agreement  with  DG  INTPA  for  the  provision  of  the  ETF’s  expertise  in  the 
implementation of the Africa Continental Qualifications Framework (ACQF)259.  
•  Human Capital Development (HCD) training for EU delegations in Africa and Asia - 

contract signed in October 2022 with DG INTPA. 

Figure 2: Work programme delivery: Eurofound, ETF and EU-OSHA260 (%) 

97
94

84

93
90
89

93
91

83

91
91

87

96
96
93

98
97

93

98
96

87

97

84

71

95

83

74

100

95

95

90

85

80

75

70

65

60

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022

Eurofound

ETF

EU-OSHA

Source: Consolidated Annual Activity Reports for Eurofound, ETF and EU-OSHA 2016-2022 

257 Second phase (SLA2) signed in autumn 2022; Amount: EUR 1 million – DG EMPL. 
258 Proposed duration: (2022-2027). Contract signed in Q2 2022 – Amount: EUR 10 million –DG INTPA. 
259 Proposed Action intends to contribute to the Multi-Annual Indicative Programme for Sub-Saharan Africa 2021-
2027 Priority area 1 – Human Development: Education.   Amount: EUR 5 million – DG INTPA – start date 
January 2023. 

260 Cedefop does not use quantitative indicators for monitoring work programme delivery. 

133 

 
 
 
 
 
 
Figure 3: Staff perceptions on administrative burden, % of respondents 

Source: Staff Survey (ETF N=52, Cedefop N=45, Eurofound N=51, EU-OSHA N=48) question: Do you think 
that administrative tasks related to programming, monitoring, reporting and evaluation hindered the 
implementation of your primary tasks in the period 2017-2022? 

Table 3: Simplification/cost-savings measures implemented by the four Agencies 

Eurofound 

Collaboration  and  sharing  services:  Joint  procurement  (through  EUAN),  recruitment  (sharing 
reserve  lists  with  other  decentralised  Agencies)  and  developing  joint  outputs  with  the  other 
EMPL Agencies and international organisations. 

Automation/digitisation:  

1) Digitalisation of the Agency’s documents and processes 

2) Exploration of more cost-efficient methodologies, particularly computer-assisted telephone 

interviewing (CATI) as opposed to face-to-face data collection 

       3) Transitioning meetings to online platforms, alleviating the missions and travels budget 

Streamlining:  Implementation  of  the  Activity  Based  (budget)  Management  (ABM)  approach, 
extending cooperation with Brussels Liaison Office for more efficient contact with stakeholders 
at EU level. 

Cedefop 

Collaboration  and  sharing  services:  Joint  procurement  (through  EUAN),  recruitment  (sharing 
reserve  lists  with  sister  Agencies),  sharing  staff  (joint  legal  support  with  ENISA)  and 
developing joint outputs with other Agencies and international organisations. 

Automation/digitisation: 

1)  Switching to e-tendering and e-submission for its open procedures  

2)  Digitalisation of the Agency’s documents and processes 

3)  Transitioning meetings to online platforms 

Streamlining:  

1)  Setting up new management approaches, such as portfolio review and negative priorities 

2)  Transformation of annual reports to short summaries that do not duplicate CAARs 

ETF 

Collaboration  and  sharing  services:  Joint  procurement  (through  EUAN),  recruitment  (sharing 
reserve lists with other decentralised Agencies), sharing staff (joint accounting officer with EU-
OSHA) and developing joint outputs with other Agencies and international organisations. 

134 

 
 
 
Automation/digitisation:   

1)  Digitalisation  of  the  Agency’s  documents  and  processes  (HR,  financial  workflows, 

procurement) 

2)  Establishment  of  online  collaboration  platforms  with  stakeholders  (e.g.  Skills  Lab 
Network)  facilitating  knowledge  exchange,  collaboration,  and  storage  of  information 
without costs on on-site meetings 

3)  Collaboration with statistical offices in partner countries by sharing data, providing remote 

assistance for data calculations, or offering calculation scripts and guidance 

Streamlining: The new Project Management Unit set up in 2021 allowed for a streamlined approach 

to the management of the framework contract and ensured timely payments. 

EU-OSHA 

Collaboration  and  sharing  services:  recruitment  (sharing  reserve  lists  with  other  decentralised 
Agencies), sharing staff (joint accounting officer with EU-OSHA) and developing joint outputs 
with other Agencies and international organisations. 

Automation/digitisation:  

1) 

Introduction of a paperless process and digital signatures  

2)  Transitioning meetings to online platforms 

3)  Using machine translations of research outputs, particularly for shorter material 

Streamlining:  

1) 

Introduction of new tools for reporting on tasks 

2) 

Improving efficiency of campaigns organisation through introduction of the Focal Points 
Assistance Tool (FAST)261. 

261 FAST is a tool supporting FOPs to run and promote campaigns at the national level, e.g. through contracting a 

media partner in national Member States. 

135 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 4: Stakeholders’ perceptions of the appropriateness of the size of the 
Governing/management board, % of respondents 

Stakeholders' perceptions of the appropriateness of the size of the 
Governing/Management Board, % of respondents

d
n
u
o
f
o
r
u
E

p
o
f
e
d
e
C

F
T
E

A
H
S
O
U
E

-

Others (n = 31)

26%

10%

6%

3%

55%

MB members (n = 33)

64%

24%

6%

6%

Others (n = 142)

15%

11%

4%

1%

MB members (n = 31)

61%

Others (n = 84)

18%

10%

2%

70%

70%

26%

3%

10%

MB members (n = 5)

80%

20%

Others (n = 33)

48%

27%

3%

21%

MB members (n = 41)

71%

22%

5%

2%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

To a large extent

To a moderate extent

To a small extent

Not at all

Do not know

Figure 5: Stakeholders’ perceptions of the appropriateness of the composition of the 
Governing/management board, % of respondents 

Stakeholders' perceptions of the appropriateness of the composition of the 
Governing/Management Board, % of respondents

d
n
u
o
f
o
r
u
E

p
o
f
e
d
e
C

F
T
E

A
H
S
O
U
E

-

Others (n = 31)

29%

10%

3%

58%

MB members (n = 33)

Others (n = 142)

14%

11%

5%

1%

MB members (n = 31)

Others (n = 84)

17%

11%

2%

73%

73%

15%

3%

9%

15%

3%

9%

70%

70%

MB members (n = 5)

100%

Others (n = 33)

52%

27%

21%

MB members (n = 41)

68%

20%

10%

2%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

To a large extent

To a moderate extent

To a small extent

Not at all

Do not know

136 

 
 
 
 
Source: Stakeholder Survey (ETF N=91, Cedefop N=174, Eurofound N=65, EU-OSHA N=75) question: In 
2017-2022, to what extent were the following aspects of Agencies’ governance structures appropriate for its size 
and nature 

Figure 6: Percentage of Agency staff indicating that they were very satisfied or satisfied 
with the fulfilment of the Headquarter Agreement in three areas. 

71

67

53

46

31

88

63

54

45

47

34

100

90

90

80

70

60

50

40

30

20

10

0

EU-OSHA

ETF

Eurofound

Cedefop

Overall Headquarter Agreement fulfilment

Multilingual and European-oriented schooling

Appropriate transport connections

Source: Agency staff survey responses to the following question: Please, indicate your level of satisfaction with 
the fulfilment of Headquarter Agreement regarding the following obligations: Overall Headquarter Agreement 
fulfilment; Multilingual and European-oriented schooling; and Appropriate transport connections. N = EU-
OSHA 48, ETF 52, Eurofound 51, Cedefop 45. 

Figure 7: Stakeholder views on the characteristics of the Agencies’ work they value 
most.  

137 

 
 
 
 
Source: Supporting study Stakeholder survey for each of the four Agencies, 2023. N= Eurofound 65; EU-OSHA 
75; ETF 91; Cedefop 174. Multiple answers possible. 

Table 4 – ‘Hypothetically, would termination of Agency activities affect your work?' (% 
of each Agency’s stakeholders replies) 

Cedefop 
stakeholders 
66 
18 
16 

ETF 
stakeholders 

Eurofound 
stakeholders 

EU-OSHA 
stakeholders 

62 
21 
18 

72 
9 
18 

85 
7 
8 

Yes 
No 
Don’t 
know/cannot 
answer 

Source: Supporting study stakeholders survey, 2023.  Cedefop n=174; ETF n=91; Eurofound n=65; EU-OSHA 
n=75 

Figure 8: Assessment of the ETF functions by stakeholder groups 

Source: compiled by research team based on the Policy Delphi data, responses include those who rated the 
functions as successful to a large and to some extent (n=263) 

138 

 
 
 
 
 
 
 
Figure 9: Public Consultation responses on the extent to which the Agencies focus their 
work on areas most in need and where they can have an impact 

Source: Public consultation, 2023. Question: To what extent did the Agencies focus their work on areas most in 
need and where they can have an impact? N = Cedefop 44; ETF 32; EU-OSHA 41; Eurofound 31. 

Figure 10: Responses to the staff survey: 'In your opinion, to what extent do the 
Agency's stakeholders take full advantage of the information, resources and services 
provided by the Agency?' 

Source: Staff survey for each of the four Agencies, 2023. Question: In your opinion, to what extent do the 
Agency's stakeholders take full advantage of the information, resources and services provided by the Agency?  
N = Cedefop 45; ETF 51; EU-OSHA 48; Eurofound 51. 

139 

 
 
 
Table 5: Alignment between EU policy priorities and Agencies’ objectives262 

Source: Supporting study (ECORYS) 

EURO- 
FOUND 

CEDEFOP 

EU-
OSHA 

ETF 

Juncker Commission priorities 2014-2019 

A new boost for jobs, growth and investment 

✔ 

✔ 

✔ 

✔ 

A connected digital single market 

A resilient Energy Union with a forward-looking 
climate change policy 

A deeper and fairer internal market with a 
strengthened industrial base 

A deeper and fairer Economic and Monetary 
Union (EMU) 
A reasonable and balanced free trade agreement 
with the United States 
An area of Justice and Fundamental Rights based 
on mutual trust 

Towards a new policy on migration 

Europe as a stronger global actor 

A Union of democratic change 

Von der Leyen Commission priorities 2019-2024 

A European Green Deal 
A Europe fit for the digital age 
An economy that works for people 
A stronger Europe in the world 
Promoting our European way of life 
A new push for European democracy 

✔ 

✔ 

✔ 
✔ 
✔ 
✔ 
✔ 
✔ 

✔ 

✔ 

✔ 

✔ 

✔ 
✔ 
✔ 

✔ 
✔ 
✔ 

✔ 

✔ 

✔ 

✔ 

✔ 

✔ 
✔ 
✔ 
✔ 
✔ 
✔ 

Table 6: Examples of the Agencies’ responses to new EU needs in the evaluation period 

Agency 
Eurofound 

Example of a response 
•  Eurofound’s strategic area anticipating and managing the impact of change replaces 
the digital age SAI from the previous programming period to cover a much broader 
spectrum of societal and economical changes, covering the ‘twin transitions’ (i.e. 
climate change and digitalisation). 

262  European  Parliament  (2014),  Setting  EU  priorities,  2014-2019:  The  ten  points  of  Jean-Claude  Juncker’s 
political guidelines, available at: https://www.europarl.europa.eu/EPRS/EPRS-Briefing-538963-Setting-EU-
Priorities-2014-19-FINAL.pdf., and European Commission [online], The European Commission’s priorities: 
for  2019-24,  available  at:  https://commission.europa.eu/strategy-and-
6  Commission  priorities 
policy/priorities-2019-2024_en. 

140 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
EU-OSHA 

• 

• 

In 2018 there was a strong policy focus on the future of work and digitalisation, 
especially the area of platform work263, specifically through the production of a 
platform economy web repository alongside the comparative EU-wide study of 
employment and working conditions of selected types of platform work264, which 
fell under Eurofound’s Strategic Area of Intervention of the digital age. 
In 2019, Eurofound’s work on monitoring convergence was of use in contributing to 
the European Union’s policy agenda and in the Assessment of the Europe 2020 
strategy265. 

•  Eurofound adopted an environmental policy in 2021266. Adopting such a policy that 
impacts on how the Agency is run is directly linked to the European Green Deal and 
the requirement for EU institutions, as well as many other organisations, to work 
towards lessening their carbon footprint. 

•  Across 2017 and 2018, EU-OSHA conducted a two-year study “Foresight on new 
and emerging OSH risks associated with digitalisation by 2025”267-268, which was 
followed up by a major OSH Overview research project on policies and practices 
with regard to digitalisation, as well as work on ‘Regulating the OSH impact of the 
online platform economy’269, which directly echoed the Juncker Commission 
priority related to the Digital Single Market. 
In recent years, EU-OSHA has also intensified its work related to the green and 
digital transitions, as demonstrated by a growing number of thematic publications on 
green jobs and digitalisation. 

• 

•  Thematic publications on green jobs include foresight studies aiming at anticipating 

OSH challenges in relation to green jobs and the circular economy. 

•  EU-OSHA has digitalised a number of work processes reducing its environmental 

impact and uses a scoreboard to monitor its environmental impact. 

Cedefop 

•  Cedefop’s work on digitalisation focuses on the impact it might have on EU 

employment and skill needs. The data sources that the Agency employs are the 
Cedefop European skills and jobs surveys, online job advertisements, based on Big 
Data analysis, skills forecasts, Cedefop survey on platform work, and Cedefop work 
on key competences with a focus on digital competences. 

•  The focus of Cedefop’s second European Skills and Jobs Survey is on the 

relationship between technological change and digitalisation, changing job-skill 
requirements and skill mismatch of EU adult workers and their adaptiveness to such 
trends via vocational education and training. 

•  Overall, during the evaluation period, Cedefop produced 11 publications under the 
umbrella of the Cedefop Green Observatory, exploring the impact of the green 
economy on VET and skills in sectors and occupations270.  

•  Among Cedefop’s most relevant initiatives within this scope are the sectoral skills 
foresight analyses, which aim to understand which occupational profiles will foster 

263 Eurofound (2018), Consolidated annual activity report of the Authorising Officer for the year 2018, p. 3, 
available at: https://www.eurofound.europa.eu/publications/annual-report/2019/consolidated-annual-
activity-report-of-the-authorising-officer-for-the-year-2018. 

264 Ibid. p.8. 
265 Eurofound (2019), Consolidated Annual Activity Report 2019, p. 3. 
266 Eurofound (2021), Environmental Policy, available at: 

https://www.eurofound.europa.eu/sites/default/files/page/field_ef_documents/eurofound_environmental_pol
icy_signed.pdf. Eurofound has since certified for EMAS, verification in Q4 2022. 

267 EU-OSHA (2018), Annual Activity Report 2017, available at: https://osha.europa.eu/en/publications/annual-

activity-report-2017/view-0  

268 EU-OSHA (2019), Annual Activity Report 2018, available at: https://osha.europa.eu/en/publications/annual-

activity-report-2018  

269 EU-OSHA (2018), Annual Activity Report 2017, available at: https://osha.europa.eu/en/publications/annual-

activity-report-2017/view-0 

270 Cedefop (2022), Consolidated Annual Activity Report 2021, p.10, available at: 

https://www.cedefop.europa.eu/files/rb202200697_caar_2021_0.pdf. 

141 

 
 
 
the transition of smart and green cities, waste management, agri-food and circular 
economy towards a "greener" future271.  

•  The green employment and skills transformation report272 provides insights into the 
possible effect of the European Green Deal on different sectors and occupations and 
identifies opportunities and challenges for upskilling and reskilling within this 
changing context. 

•  Cedefop has started reducing its energy consumption, diminishing its carbon 

footprint and digitalising workflows to become paperless273. Further, the emissions 
generated by the Agency have been gradually reducing, with an especially notable 
reduction since the pandemic.  

Table 7: Agencies’ joint outputs relevant to common themes 

EUROFOUND 

CEDEFOP 

ETF  

EU-OSHA 

Eurofound 

- 

Cedefop 

See above 

European company 
survey 2019 and 
the accompanying 
‘Workplace 
practices 
unlocking 
employee 
potential’ report; 
‘Skills forecast: 
trends and 
challenges to 
2030’ report274. 
Knowledge 
exchange and 
seminars (e.g. 
Skills and quality 
jobs as drivers of a 
just green 
transition)275. 
- 

‘Towards age-
friendly work in 
Europe: a life-
course perspective 
on work and 
ageing from EU 
Agencies’278. 

Living, working and 
COVID-19 survey 
covered EU 
Member States and 
10 EU neighbouring 
countries276. New 
Working Life 
Country Profile for 
Ukraine was 
published in 
November 2022 by 
Eurofound and the 
ETF277.  

Joint outputs related 
to VET policy 

Common thematic 
areas were not 

271 Cedefop (2021), Sectoral skills foresight, available at: 

https://www.cedefop.europa.eu/files/cedefop_sectoral_skills_foresight_methodology_and_experts.pdf  
272 Cedefop (2021), The green employment and skills transformation. Insights from a European Green Deal skills 

forecast scenario, available at: https://www.cedefop.europa.eu/en/publications/4206 

273 European Parliament (2021), Discharge report 2019, p.8, at: https://www.cedefop.europa.eu/en/content/2019-

discharge-european-centre-development-vocational-training-cedefop.  

274 Cedefop, Eurofound (2018), ‘Skills forecast: trends and challenges to 2030’, Luxembourg: Publications 

Office, Cedefop reference series No 108, available at: http://data.europa.eu/doi/10.2801/4492.  

275 EU-OSHA. Cedefop-Eurofound joint knowledge seminar on Skills and Job Quality as drivers of a just green 

transition, available at:  https://osha.europa.eu/en/oshevents/cedefop-eurofound-joint-knowledge-seminar-
skills-and-job-quality-drivers-just-green-transition 

276 Eurofound-ETF (2022), Living, working and COVID-19 in the European Union and 10 EU neighbouring 

countries, Publications Office of the European Union, Luxembourg, available at: 
https://www.eurofound.europa.eu/sites/default/files/ef_publication/field_ef_document/ef21065en.pdf. 

277 See: https://www.etf.europa.eu/en/news-and-events/news/understanding-working-life-ukraine-joint-

endeavour-eurofound-and-etf.  

278 EU-OSHA, Cedefop, Eurofound, EIGE (2017), ‘Towards age-friendly work in Europe: a life-course 

perspective on work and ageing from EU Agencies’, available at: 
https://www.eurofound.europa.eu/publications/report/2017/towards-age-friendly-work-in-europe-a-life-
course-perspective-on-work-and-ageing-from-eu-Agencies.  

142 

 
 
 
 
reporting as part of 
the implementation 
and monitoring of 
Riga medium-term 
deliverables, 
qualification 
systems, including 
validation of non-
formal learning, 
work-based learning 
and work-placed 
learning, skills 
matching, 
anticipation, and 
identification of 
skills needs279. 
- 

identified. Joint 
activities and 
outputs were 
identified in the 
context of 
multilateral 
activities e.g. 
‘Employment, 
skills and social 
policies that work 
for young 
Europeans in times 
of uncertainly’ 
event. 

Common thematic 
areas not 
identified. 
Cooperation was 
based on 
multilateral 
activities 
mentioned above. 

ETF 

See above 

See above 

Table 8: Agencies’ outputs corresponding to the European Commission’s 2019-2024 
strategic priorities 

A 
European 
Green Deal 

EUROFOUND 
'Energy scenario: 
Employment 
implications of the 
Paris Climate 
Agreement' report, 
2019 
'Distributional 
impacts of climate 
policies in Europe' 
report, 2021 

A Europe 
Fit for the 
Digital Age 

'Digitisation in the 
workplace' report, 
2021 
'New forms of 
employment: 2020 
update' report, 2020 

279 Ibid. 

CEDEFOP 
'Work-based 
learning and the 
green transition' 
collaborative leaflet, 
2022 
'The green 
employment and 
skills 
transformation: 
Insights from a 
European Green 
Deal skills forecast 
scenario' report, 
2021 
'Skill development 
in the platform 
economy' research 
paper, 2021 
'Understanding 
technological 
change and skill 
needs: skills surveys 
and skills 

143 

EU-OSHA 
Foresight study on the 
circular economy and 
its impacts on OSH 

ETF 
'Building a resilient 
generation in Central 
Asia and Europe' - 
Youth views on 
lifelong learning, 
inclusion and the 
green transition', 
2021 
'Skilling for the 
green transition' 
policy brief, 2023 

'The future of work - 
New forms of 
employment in the 
Eastern Partnership 
countries: Platform 
work', 2021 
'Scaling up and 
integrating the 
SELFIE tool for 

OSH overview 
research project 2020-
2023 including 
research on: advanced 
robotics and artificial 
intelligence; Worker 
management through 
artificial intelligence 

 
 
 
 
 
 
An 
Economy 
that Works 
for People 

A Stronger 
Europe in 
the World 

Promoting 
our 
European 
Way of 
Life 

A New 
Push for 
European 
Democracy 

'Minimum wages in 
the EU' publication 
series, 2015-2023 
'COVID-19 
pandemic and the 
gender divide at 
work and home' 
report, 2022 

'Casual work: 
Characteristics and 
implications' report, 
2019 
'The cost-of-living 
crisis and energy 
poverty in the EU: 
Social impact and 
policy responses - 
Background paper', 
2022 
'Challenges and 
prospects in the 
EU: Quality of life 
and public services' 
report, 2019 
'Tackling labour 
shortages in EU 
Member States' 
report, 2021 

'European quality 
of life surveys' 
publication series, 
2003-2018 
'Maintaining trust 
during the COVID-
19 pandemic' 
report, 2022 

forecasting' 
practical guide, 
2021 
'Digital gap during 
COVID-19 for VET 
learners at risk in 
Europe' report, 2020 
'Economic and 
gender equality: are 
skills systems the 
answer?' blog 
article, 2020 
'Lifelong guidance 
policy for Ukrainian 
refugees in the EU' 
flyers from 
CareersNet network, 
2023 
'Mobilising skills 
for business 
performance' 
webinar, 2023 

'European skills and 
jobs' survey and 
online tool, last 
entry 2021 
'Relocation 2.0: 
tying adult refugee 
skills to labour 
market demand' 
policy brief, 2022 

'European company 
survey 2019: 
Workplace practices 
unlocking employee 
potential' report, 
2020 

school's digital 
capacity in education 
and training, 2021 
'Developing national 
career development 
support systems', 
ILO-ETF report, 
2021 
'Unlocking youth 
potential in Soth 
Eastern Europe and 
Turkey', 2020 
'How do migration, 
human capital and 
the labour market 
interact in the 
Western Balkans', 
2022 
'The future of skills: 
A case study of the 
automotive sector in 
Turkey', 2021 

'Skills mismatch 
measurement in ETF 
partner countries', 
2022 
'Building resilient 
societies with young 
people in the era of 
COVID-19', 2020 - 
UNICEF and ETF 
report 
'Civil society 
organisations' role in 
Human Capital 
Development' 
country reports, 2022 

Expert article on OSH 
as a key factor to 
attract new personnel; 
Expert article on 
workers with mental 
health condition in a 
digitalised world 

Foresight study on 
circular economy and 
digitalisation 
'Casual work: 
Characteristics and 
implications' report, 
2019 

ESENER-3 
Psychosocial risks - 
main overview report 
Expert article on 
'workforce diversity 
and MSDs: COVID-19 
risks and migrants' 

Different outputs that 
can provide workers 
with useful 
information about their 
rights at the workplace 
regarding health and 
safety 

Source: compiled support study contractor based on consolidated annual activity reports 

144 

 
 
Figure 11: Quality of collaboration between the four Agencies and ELA 

Source: Staff surveys (Eurofound, n=140; Cedefop, n=146; ETF, n=142; EU-OSHA, n=142; ELA, n=190), 
question: How would you rate the collaboration and joint activities between your Agency and the following 
Agencies / Authorities: 

Figure 12: Cost-effectiveness assessment by Agency stakeholders 

d
n
u
o
f
o
r
u
E

p
o
f
e
d
e
C

F
T
E

A
H
S
O
U
E

-

Others (n = 31)

45%

16%

6%

32%

MB members (n = 33)

61%

24%

3%

12%

Others (n = 141)

26%

20%

6%

49%

MB members (n = 30)

63%

20%

17%

Others (n = 82)

34%

15%

4%

48%

MB members (n = 41)

60%

Others (n = 33)

39%

18%

MB members (n = 40)

70%

40%

42%

30%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

High

Medium

Low

Don't know

Source: supporting study Stakeholders survey; Q: “How do you rate the cost-effectiveness of EU-OSHA's 

operations during 2017-2022?” 

145 

 
 
 
Figure 13: Stakeholders’ perception of “megatrends” that will impact the future work of 
the Agencies 

Future trends affecting Agencies' relevance

21%

18%

22%

21%

18%

17%

28%

30%

5%

5%

8%

6%

40%

30%

20%

10%

0%

Climate change

Demographic
change

Urbanisation

Migration

Digitalisation and
technological
development

Future pandemics

Survey

PC

Source: Combined data from Stakeholder survey (n=1111) and public consultation (n=267), question: ‘Are there 
any future trends that could affect the Agency's / Agencies' future relevance?’ 

146 

 
 
  
 
 
 
 
ANNEX VIII: TECHNICAL ANNEX 

1.  DELIVERING OUTPUTS (MEETING OPERATIONAL OBJECTIVES) 

The  four  Agencies  have  been  delivering  their  activities  according  to  their  operational 
objectives280 (see Agencies’ intervention logics in Annex VI). 

1.1. Eurofound 

Eurofound’s monitoring activities relied on three observatories – EurWORK, EMCC and 
EurLIFE – to which the newly created COVID-19 EU PolicyWatch database was added281. 
EurWORK  has  been  the  most  active  of  these  observatories,  publishing  regular  reports  and 
receiving quarterly inputs from the Network of Eurofound Correspondents (NEC) on working 
life developments, whereas the COVID-19 EU PolicyWatch database, created in 2020, is one 
of the flagship outputs of the evaluation period. This database has since been transformed into 
the  EU  PolicyWatch  database  and  is  now  a  general  social  and  employment  policy-related 
database used across all SAIs. 

Research  and  data  collection  by  Eurofound  are  underpinned  by  its  pan-European 
surveys282. The latest European working conditions survey (EWCS) was conducted in 2020, as 
planned, and covered 37 countries (two more compared to the previous one delivered in 2015). 
In addition, a follow-up telephone survey was conducted in 2021, which made it possible to 
assess  the  impact  of  COVID-19  on  job  quality.  The  European  company  survey  (ECS)  was 
delivered  in  2019-2020  in  cooperation  with  Cedefop.  The  survey  represented  the  first 
large-scale, cross-national survey to use a ‘push-to-web’ approach283. Eurofound’s decision to 
conduct surveys using a computer-assisted telephone interviewing (CATI) methodology rather 
than face-to-face interviewing arose as a necessity of COVID-19, but prompted Eurofound to 
make an assessment of methods other than face-to-face interviewing for collecting data in the 
longer term,  in  order to  save costs284.  The next  round of the European  quality of life survey 
(EQLS) was not intended to take place during the evaluation period. The focus was on analysing 
the results and disseminating the latest edition of the survey conducted in 2016.285  

1.2. Cedefop 

280 Each agency collects and monitors the data on the outputs produced per year, but the level of detail, type of 
outputs and definitions vary across the Agencies. See Annexes 1 to 4 to the supporting study for a detailed 
analysis of each agency’s performance in achieving its operational objectives.  

281  European  Observatory  of  Working  Life  (EurWORK),  European  Restructuring  Monitor  (ERM),  European 

Monitoring Centre on Change (EMCC), and European Observatory on Quality of Life (EurLIFE). 

282  European  working  conditions  survey  (EWCS),  European  company  survey  (ECS),  European  quality  of  life 
survey (EQLS), and living, working and COVID-19 e-survey. The share of expenditure allocated to survey 
management  and  development  increased  considerably  during  this  evaluation  period,  rising  from  7%  of 
expenditure in 2017 (EUR 1.39 million) to 18% in 2022 (EUR 3.91 million). 

283 Where businesses were contacted via telephone to identify a management and an employee respondent, who 

were then asked to fill out the survey questionnaire online; see: 
https://www.eurofound.europa.eu/surveys/2019/european-company-survey-2019.  

284 Eurofound (2022), Consolidated annual activity report 2021, p. 3. 
285 See: https://www.eurofound.europa.eu/observatories/eurlife.  

147 

 
 
 
 
 
Cedefop has eight operational objectives on  vocational education and training (VET), skills 
and qualifications: (i) to carry out research; (ii) to contribute to EU and national policies; (iii) 
to compile relevant data; (iv) to monitor VET policy developments, systems and policies; (v) 
to develop and implement European tools to support VET, skills and qualifications policies; 
(vi)  to  carry  out  activities  with  other  agencies/organisations;  (vii)  to  contribute  to  capacity 
building; and (viii) to disseminate knowledge, evidence and data.  

During the current evaluation period, Cedefop delivered the expected outputs across all these 
objectives, with a strong focus on research and policy analysis. Compared with the 2016 
baseline, the number of publications slightly decreased during the current evaluation period, 
which was due to Cedefop’s decision to shift from traditional publications to online content, 
making the agency’s research and analyses available in online interactive tools and databases 
(see below). 

In  line  with  the  objective  to  compile  relevant  VET,  skills  and  qualifications  data,  one 
important achievement of the period is the development of online tools and databases, to 
increase  interaction  and  the  user-friendliness  of  the  collected  data286.  Except  Skills  Forecast 
(developed in 2015), the other tools were designed and implemented in 2016-2022, including 
the European database on apprenticeship  schemes,  the database on validation of non-formal 
and informal learning, the European skills index and the European VET policy dashboard. 

Throughout  the evaluation period,  Cedefop also  contributed to capacity building through 
events and policy learning activities, although to a more limited extent compared with the 
baseline. The number of events had already started to decline in the previous evaluation period; 
by 2014, it had halved287. The number stayed relatively stable, with minor declines, until 2020, 
when  it  dropped  due  to  the  COVID-19  pandemic,  as  Cedefop  correspondingly  reduced  the 
budget dedicated to missions and events. 

Cedefop has also achieved the objective ‘Supporting the development and implementation 
of European tools to support VET, skills and qualifications policies’.  It has produced the 
expected outputs in relation to supporting EQF, ECVET and EQAVET, and has played a 
crucial role in transitioning to the new Europass. 

1.3. EU-OSHA  

The agency has fully delivered on its six operational objectives, with very high levels of 
timely delivery, above 90% for most of the activities288: (i) develop foresight activities; (ii) 
provide occupational  safety  and health  (OSH) facts  and figures; (iii) develop  tools  for OSH 
management; (iv) raise awareness of and communicate on OSH issues; (v) network knowledge 
on OSH; and (vi) maintain and develop strategic and operational networks. 

Compared to the baseline, the number of outputs related to facts and figures almost tripled and 
there has been a substantial  increase in  outputs related to  networking knowledge during  the 

286  According  to  the  2022  user  satisfaction  survey,  they  are  Cedefop’s  second  most  appreciated  outputs,  after 

publications.  

287 From 150 in 2012, and 135 in 2013, to 69 in 2014. Cedefop’s annual reports for 2012-2014. 
288 See Table 6 of the main report on the supporting study. 

148 

 
 
 
second half of the evaluation period (2020-2022). The number of outputs across other areas was 
similar to the baseline year of 2016. 

Table 1: Number of EU-OSHA outputs delivered on time under each activity, 2016-2022 

EU-OSHA 
activity 
(including all 
corresponding 
outputs) 

A1: Anticipating 
change 
A2: Facts and 
figures 
A3: Tools for 
OSH 
management 
A4: Raising 
awareness 
A5: Networking 
knowledge 
A6: Strategic and 
operational 
networking 

2016 

2017 

2018 

2019 

2020 

2021 

2022 

0 

14 

4 

22 

3 

22 

5 

18 

8 

33 

8 

21 

10 

32 

15 

35 

9 

12 

12 

20 

12 

42 

7 

24 

4 

23 

16 

34 

14 

28 

12 

23 

16 

32 

23 

24 

3 

41 

4 

37 

24 

23 

% of timely 
delivered 
outputs vs 
number of 
committed 
outputs 
(2017-2022) 

92% 

72% 

93% 

98% 

79% 

92% 

Source: Ecorys supporting report based on EU-OSHA’s annual activity reports for 2016–2022. 

The key outputs falling under facts and figures were the European survey of enterprises on new 
and  emerging  risks  (ESENER),  musculoskeletal  disorders  (MSDs),  EU-OSH  info  systems, 
OSH and digitalisation, and support to help SMEs achieve compliance. According to the key 
performance  indicators  from  the  2021  annual  activity  report,  OSH  and  digitalisation  and 
support to help SMEs achieve compliance were the most impactful activities (93%), with the 
ESENER coming in. 

The ‘tools for OSH management’ objective was increasingly important during the current 
evaluation  period.  The  agency  has  developed  practical  tools  and  guidance  that  enable 
workplaces to easily assess and manage their  OSH risks. Over 72 000 new risk assessments 
were performed in 2022, which is a figure much higher than the target of 30 000 and the baseline 
(in 2016, only 13 281 new risk assessments were performed)289. 

In  terms  of  raising  awareness  of  and  communicating  on  OSH  issues,  the  agency  has 
worked  extensively  on  the  healthy  workplaces  campaigns  (HWC)290.  Through  these 
campaigns,  EU-OSHA  was  able to  engage with various stakeholders at  national  level,  from 
small  to  medium-sized  enterprises  to  larger  corporations,  as  well  as  national  bodies,  raising 
awareness  of  risks  from  dangerous  substances  in  the  workplace  and  on  prevention  of 
work-related MSDs. See EU-OSHA Case Study 4 in Annex 8 to the supporting study for details.  

289 EU-OSHA (2016), annual activity report for 2016. 
290 See EU-OSHA Case Study 4 in Annex 8 to the supporting study. 

149 

 
 
 
 
For the HWC on MSDs, EU-OSHA had 100 official campaign partners, 29 media partners, 
performed  1 322  promotional  actions,  and  organised  2 217  campaign  activities.  These 
included  activities  of  official  campaign  partners,  media  partners,  focal  points  and  European 
Enterprise Network (EEN) OSH ambassadors. For the HWCs generally, EU-OSHA delivered 
100%  of  its  planned  outputs  for  its  2016-2017  and  2018-2019  campaigns,  and  96%  of  its 
planned outputs for its 2020-2022 campaign, with the only missing output for the 2020-2022 
campaign being a communication output related to the Napo291.  

Finally, the agency delivered 91% (21 out of 23) of its outputs related to its ‘maintaining and 
developing strategic and operational networks’ activity on time, a trend that continued in 
2018 and 2019. 2020 saw some setbacks, with cancellations affecting the delivery of outputs. 
Specifically, four actions and one event were cancelled as they were impossible to implement 
due to the pandemic. On a positive note, in 2021, most outputs were delivered on time, although 
three outputs related to coordinating relations with key stakeholders had to be cancelled due to 
travel restrictions during the pandemic. 

1.4. ETF 

During  the  evaluation  period,  the  ETF  delivered  on  its  five  operational  objectives  oriented 
towards its partner countries in the context of EU external relations: (i) provide information, 
policy  analysis  and  advice;  (ii)  support  capacity  building  and  networking;  (iii)  monitor 
system-wide progress; (iv) support knowledge dissemination; and (v) support the EU project 
and programming cycle in EU external relations. 

To provide information and policy analysis, the ETF published  reports, policy briefings, guides 
and toolkits, which provided stakeholders with insights about human capital development in its 
partner countries and were assessed positively by its stakeholders (see Section 4.1.1 of the main 
staff working document).  

In relation to the ETF’s support on capacity building and networking, the number of events 
held  by  the  ETF  increased  dramatically  over  the  evaluation  period  and  compared  to  the 
baseline292, with only 24 events held in 2017 compared with 143 events held in 2021. 

To monitor system-wide progress, the corresponding output is the Torino Process assessment 
reports,  one  key  activity  used  by  the  ETF  to  ensure  that  its  work  meets  the  needs  of 
partner  countries293.  The  fifth  round  of  the  Torino  Process  (2018-2020)  covered  100%  of 

291 Napo has become the main ambassador for EU-OSHA’s HWCs: he is the hero in a series of animated films, 
providing light-hearted, discussion-provoking introductions to OSH topics and risk prevention. He is a typical 
worker in any industry or sector, who is frequently exposed to common hazards and risky situations in the 
workplace. However, he is actively involved in identifying risks and providing practical solutions. For more 
information, see: https://www.napofilm.net/en. 

292  Data  for  2015  and  2016  shows  that  the  number  of  events  organised  was  20  and  29,  respectively  (ETF’s 

consolidated annual activity reports for 2015 and 2016). 

293 Conducted for the first time in 2010-2011, the TRP is a biannual review of the status and progress of VET in 
ETF partner countries, which is carried out by stakeholders in these countries and summarised in country 
reviews. 

150 

 
 
 
partner countries294, while there was a decrease in the number of countries in the sixth round 
(2022-2024), as participation in the in-depth review became voluntary.  

In  terms  of  supporting  the  EU  project  and  programming  cycle,  the  ETF  was  consistently 
requested to provide support in the majority of its partner countries, and it has also provided 
advice and expertise by responding to requests from EU stakeholders, including policy dialogue 
inputs to EU institutions and agencies295 and updates on the impact of Russia’s war of aggression 
against Ukraine on refugees and their needs for skills development and validation296. However, 
the number of requests from EU services has decreased substantially since 2020297, although the 
ETF has reported that recent years have seen a smaller number of more substantial requests.  

Finally,  the  ETF  has  made  progress  in  supporting  knowledge  dissemination  through 
strengthened external communication, following the conclusions from the previous evaluation, 
which  found  that  there  was  ‘room  for  improvement  in  the  ETF’s  communication  efforts  to 
ensure that stakeholders, particularly from partner countries, fully understand its role, work and 
objectives’298. The main developments include the creation of a dedicated online platform for 
the Torino Process in 2017, the launch of a new website for the agency in 2018, the launch of 
the Open Space knowledge-sharing platform in 2019299, and the greater emphasis that has 
been placed on social media, so as to better reach stakeholders. Each of the agency’s social 
media accounts – Facebook, LinkedIn, Twitter (now X), and Instagram – grew throughout the 
evaluation period. Facebook has seen the largest growth: 11 539 in 2017 compared to 25 010 
in 2022. 

2.  USE OF THE OUTPUTS BY THE AGENCIES’ MAIN STAKEHOLDERS 

The  four  Agencies’  services  and  outputs  have  been  used  by  their  key  stakeholders,  with  a 
tendency for usage by EU institutions to be both more prominent and more widespread than 
usage by stakeholders at national (or international) level, in line with the use patterns shown 
during the previous evaluation period. 

2.1. EU-level, policy-oriented stakeholders are among the main target groups for each agency. 
They  made  increased  use  of  the  agencies’  services  and  outputs,  particularly  in  the  case  of 
Eurofound, where the number of references to its work in key EU-level policy documents300 
increased  over  the  evaluation  period  and  compared  to  the  2016  baseline301.  The  European 

5.2.2.   

5.2.1.   

294 There was a 41% increase in stakeholder participation and the  agency reported that 2 100 participants were 

reached through events (ETF (2019); consolidated annual activity reports for 2018 and 2021). 

295 For example work with the European Institute for Gender Equality on gender gaps in employment in 2021. 
296 ETF (2023), consolidated annual activity reports for 2021-2022. 
297 They remained stable between 2016 and 2019 with between 105 and 116 requests, and then dropped to 86 in 

2020, 50 in 2021, and 40 in 2022, mainly explained by the impact of the COVID-19 pandemic. 

298  European  Commission,  Evaluation of  the  EU  Commission  Agencies  working  in  the  employment  and  social 

affairs policy field: EUROFOUND, CEDEFOP, ETF and EU-OSHA (SWD(2019) 160 final).  

299 ETF’s consolidated annual activity reports for 2018, 2019 and 2021. 
300 Key EU policy documents are defined as those that initiate policy processes, are of a consultative/advising 

nature or are reports with comprehensive uptake of Eurofound’s knowledge.  

301 Although the total number of EU policy documents referencing Eurofound’s work decreased. 

151 

 
 
 
 
Commission is consistently the institution that refers to Eurofound knowledge most302 in its 
key policy documents303, closely followed by the European Parliament, which saw a jump in 
the  number  of  mentions  in  key  policy  documents  in  2021  and  2022304.  Eurofound’s 
COVID-19-related outputs  consistently  featured  among the most-referenced outputs in  key 
EU policy documents between 2020 and 2022.  

Table 2: References in EU policy documents, 2016-2022 

2016 
Cedefop305   167 EU-

level 
documents 
Eurofound  78 key out 

of 315  

2017 
144 EU-
level 
documents 
99 key out 
of 315  

2018 
200 EU-
level 
documents 
81 key out 
of 272  

2019 
167 EU-
level 
documents 
52 key out 
of 185  

2020 
165 EU-
level 
documents 
95 key out 
of 274  

2021 
189 EU-
level 
documents 
152 key 
out of 206 

2022 
124 EU-
level 
documents 
163 key 
out of 198  

Source: annual activity reports. 

As  in  the  previous  period306,  for  Cedefop,  the  number  of  EU-level  documents  containing 
references to  its  work fluctuated, with  a low of  124 documents in  2022 and a high  of 189 
documents in 2021, with, overall, higher figures than the 2016 baseline (see Table 2)307. The 
European Commission is the stakeholder most frequently citing the Agency’s work, followed 
by the European Parliament, the ETF, and the Joint Research Centre. The thematic areas of 
Cedefop’s  work  most  referenced  in  EU  policy  documents  were  skills  anticipation  and 
matching,  and  monitoring  and  analysing  VET  policy  developments  (with  524  and  449 
references, respectively; much higher figures than for the other areas, see Figure 1)308. 

5.2.9.   5.2.10.  

302 As shown in Table 23 of the supporting study. 
303 With 41 references in key EU-level policy documents in 2017, 39 in 2018, 18 in 2019, 35 in 2020, 36 in 2021 

and 61 in 2022.  

304 With 63 references in key EU-level policy documents in 2021 and 67 in 2022, compared to only 27 in 2017, 15 

in 2018, 9 in 2019 and 18 in 2020. See Eurofound annual activity reports. 

305 Cedefop reports on this indicator differently than Eurofound, as it did not report the number of references in 

key EU-level documents for any of the years. 

306 Between 2011-2016, the lowest number of references to Cedefop’s work in  EU-level documents was 97 in 

2014, the highest number was 173 in 2013. 

307 This indicator is not reported on by the ETF and EU-OSHA. 
308 Cedefop’s consolidated annual activity reports for 2016-2022. 

152 

 
 
 
 
 
 
 
Figure 1 Number of citations of Cedefop’s outputs: top 5 thematic areas with most references in 
EU and international policy documents in 2016-2021 

Skills anticipation and matching

Monitoring and analysing VET policy developments

Supporting the development and use of European tools
and principles
Towards LMSI 2.0 – reshaping skills trends analysis and 
intelligence 

Promoting access to and attractiveness of VET

147

160

106

154

162

140

369

524

228

449

0

100

200

300

400

500

600

References to Cedefop's work in international policy documents

References to Cedefop's work in EU policy documents

Source: Cedefop’s consolidated annual activity reports for 2016-2021. 

2.2.  Research  and  academia  stakeholders  more  actively  used  Cedefop’s  and  Eurofound’s 
work compared to the previous evaluation period, as evidenced by the year-on-year increase in 
the  number  of  citations  of  both  Agencies’  outputs  in  academic  journals,  reaching  a  peak  of 
1 030 academic papers citing Cedefop’s work in 2022 and of 1 220 citations of Eurofound’s 
work in 2021 (see Table 3)309. Eurofound has intensified collaboration with universities across 
Europe310.  In  2020,  it  received  around  50  requests  from  universities,  research  institutes  and 
other EU-level stakeholders, including the European Trade Union Institute (ETUI), the  Joint 
Research Centre and the OECD311. Participation numbers at Cedefop events show the use of 
the Agency’s services by a wider range of its stakeholders. Although at least 50% of participants 
are policymakers at EU or national level, the share of researchers attending Cedefop’s events 
increased throughout the latter half of the evaluation period, from 11% in 2018 to 20% in 2021.  

Table 3: Number of citations of each agency’s work in academic literature, 2016-2022312 

Cedefop 
Eurofound 

2016 
591 
627 

2017 
545 
663 

2018 
611 
720 

2019 
838 
880 

2020 
956 
1 034 

2021 
988 
1 220 

2022 
1 030 
1 134 

2.3. Use at national level 

The  use  of  the  Agencies’  services  by  stakeholders  at  Member-State  level  is  less  well 
documented  than  the  use  by  European-level  stakeholders313.  Nevertheless,  the  evaluation 

309 No data is available on the uptake of EU-OSHA’s or ETF’s work. 
310 For example, in 2021, Eurofound signed a cooperation agreement with the European University Institute in 

Florence. 

311 For example to access the microdata under the quality of life and public services activity. 
312 Annual activity reports for 2016-2022. 
313 Partly because European-level stakeholders are the primary stakeholders. 

153 

 
 
 
 
 
 
 
provides wide evidence of effective use by national stakeholders, to varying extents given the 
Agencies’ different remits and activities.  

EU-OSHA’s outputs and tools  were used intensively  by national  stakeholders, with  outputs 
such as OiRA314, OSH-wiki, and the healthy workplaces campaigns (HWCs) increasing their 
popularity during the evaluation period, compared to 2016. The HWCs have been increasingly 
growing in terms of partnerships, events and stakeholder participation. Examples of the use of 
Eurofound’s  work  by  national  stakeholders  include  its  work  on  minimum  wages,  as 
highlighted in a case study315. Eurofound’s minimum wage publication series is continuously 
referred to by national wage-setting institutes, including governments, trade unions and low pay 
commissions316,  in their policy reports. Cedefop’s focus on national stakeholders decreased 
during the current period compared to the previous one. Its performance measurement system 
shows that ‘direct support to Member States’ (e.g. by way of thematic country reviews, policy 
learning forums, and events in Member States to support the implementation of policies) has 
decreased, following the Agency’s  decision to  prioritise EU level  policy  support in  view of 
intense budget constraints317. 

2.4.  The  only  agencies  considering  international  actors  and  organisations  as  being  key 
stakeholders  are  Cedefop  and  the  ETF  (though  Eurofound’s  outputs  are  also  used  by 
international  actors).  In  2017,  2018  and  2020,  Cedefop’s  work  on  skills  anticipation  and 
matching was the most referenced output by international organisations. In 2019, this was the 
agency’s work on monitoring VET policy developments, and in 2021, its work on reshaping 
skills  trends  analysis  and  intelligence.  These  outputs  were  the  most  frequently  referenced 
Cedefop outputs in EU policy documents as well. As regards the ETF, a 2023 ETF stakeholder 
survey  to  monitor  the  uptake  of  its  outputs  showed  that  the  thematic  areas  in  which  the 
knowledge was used the most were innovative teaching and learning, skills demand analysis 
and  vocational  excellence  provision  models,  whereas  the  outputs  on  engaging  businesses  in 
skills  development,  quality  assurance  and  governance  mechanisms,  and  upskilling  and 
reskilling were used less frequently318.  

2.5. Use and dissemination of outputs through websites and media 

5.2.17.   5.2.18.  

The  use  of  the  Agencies’  outputs  measured  by  the  number  of  downloads,  showed  varying 
trends across Agencies.  The number of downloads of  Eurofound’s publications was higher 
than the 2016 baseline figure of 155 943 throughout the evaluation period, with a peak in 2020 
of 188 556 PDF downloads from its website, supported by an increasing number of social media 
users; EU-OSHA, which had a relatively low baseline number of downloads in 2016 (82 558) 

314 10 of the 15 OiRA national partners made an explicit reference to the OiRA tool in their national OSH strategy 

or legislation. 

315 See Annex 8 to the supporting study. 
316 See Irish Low Pay Commission, Recommendations for the National Minimum Wage, July 2022, available at: 

https://assets.gov.ie/234304/074a6944-2f9e-4443-8e6d-a4f85be432d2.pdf. 

317 There were 46 outputs at the start of the evaluation period in 2017, but this decreased to only 5 in 2021. 
318  Ockham,  IPS  (2023),  Development  of  Methodology  to  Monitor  Uptake  and  Use  of  ETF  Knowledge 

Products/Activities. 

154 

 
 
 
saw319 a huge leap in 2019 and a peak of 2 259 137 in 2020320, followed by a significant drop in 
2022321. Cedefop’s downloads have been higher than the 2016 baseline (383 600) since 2020, 
when  downloads  peaked  at  465 000,  linked  to  the  shift  to  fully  digital  dissemination  of 
publications.  The  number  of  downloads  of  the  ETF’s  publications  saw  a  dramatic  decrease 
from 1 181 462 in 2016 to 351 219 in 2018322 (see Table 4). 

Table 4: Number of downloads of each agency’s publications, 2016-2022 

2016 

2017 

2018 

2019 

2020 

2021 

2022 

Cedefop323 
(publications) 

383 600 

313 850  290 000  352 000 

465 000 

416 000 

440 000 

Cedefop 
notes) 

ETF324 

(briefing 

88 400 

63 150 

55 000 

72 000 

88 000 

82 000 

N/A 

1 181 462  837 236  351 219 

EU-OSHA325 

82 558 

N/A 

N/A 

1 701 161  2 259 137  2 132 816  1 477.674 

Eurofound326 

155 943 

178 406  161 974  175 761 

188 556 

174 878 

160 233 

The importance of social  media channels  for the visibility, uptake and use of the Agencies’ 
outputs increased during the current evaluation period. The data available for Cedefop and the 
ETF shows a year-on-year growth of social media followers across all channels used by both 
Agencies327, which increased the range of the channels used (the ETF introducing Instagram in 
2018 and Cedefop LinkedIn in 2021). The use of social media channels is particularly important 
for the ETF as many stakeholders in partner countries rely on information shared through these 
channels.  Comparable  data  are  not  published  by  EU-OSHA328  and  Eurofound;  however,  the 
importance of social media channels and the increase in followers is reported in the Eurofound’s 
consolidated annual activity report for 2022329.   

319 1 701 161 downloads in 2019 and a peak of 2 259 137 in 2020, followed by a drop to 1 477 674 in 2022. 
320 This can be explained by the agency’s COVID-19 guidance documents, which were the most downloaded and 

accessed publications in the agency’s history.   

321  In  September  2021,  the  agency  migrated  its  websites,  and  from  then  to  mid-2022,  some  issues  with  the 
redirection of publications were reported, which may have negatively affected the number of downloads. 

322 Data on the number of downloads of the ETF’s publications is only available for 2016-2018. 
323 Cedefop splits this indicator into: (i) downloads of publications; and (ii) briefing notes (apart from in 2022). 
324  ‘Number of downloads of publications’. The ETF did not collect any data on this indicator after 2018. 
325  ‘Number of downloads on EU-OSHA’s corporate and campaign websites’. 
326 ‘Uptake of and engagement with Eurofound’s knowledge through its website and other corporate platforms – 

PDF downloads’. 

327 During the evaluation period, the number of followers at least doubled across most channels for both Agencies. 
328 EU-OSHA does not publish data on social media followers. 
329 Eurofound (2023), Consolidated Annual Activity Report 2022, p. 23. 

155 

 
 
 
 
 
 
 
 
  
 
156 

 
 
 
 
ANNEX IX:  TECHNICAL ANNEX (II) 

This annex presents additional detailed evidence on the theme of coherence (points 1 to 4) and 
relevance (point 5).  

1.  Agencies’ mandates, common thematic areas and activities 
The technical  specifications for the evaluation asked the following  question:  ‘1.14. To what 
extent  do  new  elements  (e.g.  the  ELA,  updated  mandates)  change  the  conclusions  of  the 
previous evaluation regarding advantages / disadvantages of mergers between the agencies?’ 

The Agencies’ mandates identify broad policy domains in which they operate.  

Table 1: Thematic coverage of the Agencies 

Eurofound 

Cedefop 

ETF 

EU-OSHA 

Thematic coverage according to the mandate of each agency 
To  provide  the  Commission,  other  Union  institutions,  bodies  and  Agencies,  the 
Member  States  and  the  social  partners  with  support  for  the  purpose  of  shaping  and 
implementing policies concerning the improvement of living and working conditions, 
devising employment policies, and promoting the dialogue between management 
and labour. To that end, Eurofound shall enhance and disseminate knowledge, provide 
evidence  and  services  for  the  purpose  of  policy  making,  including  research-based 
conclusions,  and  shall  facilitate  knowledge  sharing  among  and  between  Union  and 
national actors. 
To support the promotion, development and implementation of EU policies in the field 
of  vocational  education  and  training  (VET)  as  well  as  skills  and  qualifications 
policies by working together with the Commission, Member States and social partners. 
Contributing to human capital development in the context of EU external relations 
policies and aiming to promote the development of VET systems in the EU’s partner 
countries. 
EU-OSHA's  objective  shall  be  to  provide  the  Union  Institutions  and  bodies,  the 
Member States, the social partners, and other actors involved in the field of safety and 
health  at  work  with  relevant  technical,  scientific  and  economic  information  and 
qualified  expertise  in  that  field  in  order  to  improve  the  working  environment  as 
regards the protection of the safety and health of workers. 

Source: Compiled by research team based on the basis of Founding Regulations of the four Agencies. 

If one combines the policy scope of the Agencies as set out in the mandates (in the table above) 
with  an  analysis  of  the  strategic  objectives  that  appear  in  the  Agencies’  programming 
documents, one finds that two (or more) Agencies sometimes work in the same broad policy 
area. The most significant cases are described below. 

(i) 

Cedefop  and  the  ETF  worked  on  VET,  in  particular,  qualification  systems, 
modernisation of VET, lifelong learning, anticipation of skills for the future of work, 
labour market intelligence, quality assurance and teacher training.  

Cedefop and the ETF both worked on VET, particularly on qualification frameworks, 
quality  assurance  and  teacher  training330.  During  the  current  evaluation  period,  both 

330 PPMI, ECORYS (2018), Evaluation of the EU Agencies under the remit of DG Employment: EUROFOUND, 

CEDEFOP, ETF and EU-OSHA, available at: 
https://ec.europa.eu/social/BlobServlet?docId=21074&langId=en. 

157 

 
 
 
 
Agencies  worked  on  the  modernisation  of  VET  and  qualifications  systems,  lifelong 
learning  policies,  anticipation  of  skills  for  the  future  of  work,  and  labour  market 
intelligence. However, the actual focus of work of the two Agencies in this area was 
clearly distinguished: the ETF focused on capacity building and policy implementation 
in  partner  countries  outside  the  EU,  while  Cedefop  focused  on  research  and 
policymaking in the EU. 

(ii) 

Cedefop  and  Eurofound  worked  on  the  intersection  of  VET,  skills  mismatches, 
working and living conditions and labour market policies.  

Eurofound’s research activity on working conditions crosses over into Cedefop’s work 
on vocational skills development. The two Agencies also complement each other’s work 
in  terms  of  Cedefop’s  focus  on  skills  development  and  Eurofound’s  research  on 
employment change and the changing nature of work.  

However, the two Agencies approach these topics from different perspectives. Cedefop 
focuses on the supply side of skills mismatches and the perspective of VET providers, 
while  Eurofound  focuses  on  the  demand  side.  The  Agencies  also  have  different 
approaches to the production of research outputs and dissemination: Cedefop tends to 
produce  shorter  and  more  regular  reports  and  bulletins,  while  Eurofound  generally 
provides longer and more detailed reports.  

(iii) 

Eurofound and EU-OSHA worked on occupational health and safety.  

Eurofound and EU-OSHA both worked on occupational health and safety as this area 
can be considered part of working conditions, which is part of Eurofound’s mandate and 
objectives. For example, Eurofound’s activities include work on health and safety in the 
form of the European working conditions survey (ECWS) and the related ‘Health and 
well-being at work’ analysis. However, EU-OSHA was created when Eurofound was 
already in place and its Founding Regulation set out a very specific focus on providing 
highly technical specialised support on protecting the safety and health of workers. The 
tasks of EU-OSHA require such a high level of specialisation that a possible merger 
with another agency would clearly jeopardise its effectiveness. 

The  analysis  above  shows  that  each  agency  approached  these  broad  policy  themes  from 
different  perspectives.  For  example,  tackling  the  labour  force  from  both  the  supply  and 
demand  angles,  the  working  environment  from  both  the  employer  and  employee  angle,  and 
skills shortages from the provider and employer standpoint. 

The Agencies’ mandates cover broad, complex and interconnected themes, which are the 
basis for cooperation between the Agencies. These themes could also be the basis for exploring 
the potential for merging some of the Agencies, as was done in the previous 2019 cross-cutting 
evaluation (see next section).   

Focusing  on  cooperation,  the  common  thematic  areas  are  well  understood  by  most 
policymakers and agency staff, and thematic overlaps in the mandates do not necessarily 
mean duplication of work. In fact, common thematic areas are both accepted and used as a 
basis for inter-agency cooperation, ensuring coherence. 

158 

 
 
The current evaluation has found no instances of duplication of work among the Agencies. 
It also found that staff from all Agencies are much more aware of each other’s work compared 
to  the  previous  evaluation  and  are  therefore  able  to  complement  rather  than  duplicate  each 
other’s work. Each agency has its own unique area of expertise when addressing the broad 
themes that are common among the four Agencies.  

The examples of joint outputs and activities delivered by the agency pairs related to common 
thematic areas are presented in Table 2. 

Table 2: Agencies’ joint outputs relevant to common themes 

Eurofound 

Cedefop 

ETF 

EU-OSHA 

Eurofound 

- 

Cedefop 

- 

European company 
survey 2019 and the 
accompanying 
‘Workplace practices 
unlocking employee 
potential’ report; 
‘Skills forecast: 
trends and challenges 
to 2030’ report331. 
knowledge exchange 
and seminars (e.g. 
Skills and quality 
jobs as drivers of a 
just green 
transition)332. 
- 

‘Towards age-
friendly work in 
Europe: a life-course 
perspective on work 
and ageing from EU 
Agencies’335. 

Common thematic 
areas were not 
identified. Joint 
activities and outputs 
were identified in the 
context of 
multilateral 
activities, e.g. 
‘Employment, skills 
and social policies 
that work for young 
Europeans in times 
of uncertainly’ event. 

Living, working and 
COVID-19 survey 
covered EU Member 
States and 10 EU 
neighbouring 
countries333. New 
Working Life 
Country Profile for 
Ukraine was 
published in 
November 2022 by 
Eurofound and the 
ETF334.  

Joint outputs related 
to VET policy 
reporting as part of 
the implementation 
and monitoring of 
Riga medium-term 
deliverables, 
qualification 
systems, including 
validation of non-
formal learning, 
work-based learning 
and work-placed 
learning, skills 
matching, 
anticipation, and 

331 Cedefop, Eurofound (2018), ‘Skills forecast: trends and challenges to 2030’, Luxembourg.  
332 EU-OSHA. Cedefop-Eurofound joint knowledge seminar on Skills and Job Quality as drivers of a just green 

transition.  

333 Eurofound-ETF (2022), Living, working and COVID-19 in the European Union and 10 EU neighbouring 

countries, Publications Office of the European Union, Luxembourg. 

334 See: https://www.etf.europa.eu/en/news-and-events/news/understanding-working-life-ukraine-joint-

endeavour-eurofound-and-etf.  

335 EU-OSHA, Cedefop, Eurofound, EIGE (2017), ‘Towards age-friendly work in Europe: a life-course 

perspective on work and ageing from EU Agencies’.  
159 

 
 
 
 
ETF 

- 

- 

identification of 
skills needs336. 
- 

Common thematic 
areas not identified. 
Cooperation was 
based on multilateral 
activities mentioned 
above. 

Source: Compiled by research team 

2.  The European Labour Authority 
The evaluation question also identified the European Labour Authority (ELA) as a potential 
new element that could lead to revising the analysis of merger scenarios. 

However, the Agencies’ potential to duplicate the work of the ELA is limited. EU-OSHA, 
Cedefop, and Eurofound are predominantly research-focused and do not have an operational 
and cross-border focus as ELA does. The potential for a merger with the ETF is limited as it 
only operates in non-EU countries, mainly in the EU’s neighbourhood.  

The ELA can, nevertheless, use existing research resources developed by the Agencies in their 
respective  areas of competence;  this could  also  lead to  pooling resources  or producing  joint 
reports on issues of common concern, e.g. on restructuring or respecting occupational health 
and safety rules in mobility situations, including posting, in the context of risk assessment.  

Cooperation between the four Agencies and the ELA is at an early stage due to the ELA having 
only been recently created. Nevertheless, the three tripartite Agencies have begun to cooperate 
with ELA on  certain issues, for example, on a seasonal  workers campaign (EU-OSHA) and 
skills intelligence and job shortages (Eurofound and Cedefop). Cooperation on digital labour 
mobility  with  the  ETF  has  been  identified  as  a  subject  for  future  cooperation.  Cooperation 
across  all  five  agencies  took  place  when  ELA  participated  in  the  September  2022  event 
‘Employment, skills and social policies that work for young Europeans in times of uncertainly’. 

3.  Potential for mergers: previous evaluation findings and current evaluation assessment 
Based on the assessment of the Agencies’ mandates and common thematic areas, the previous 
2019  cross-cutting  evaluation  identified  which  of  them  could  have  a  higher  potential  for 
merging. The evaluation also assessed the benefits and risks of each merger scenario. 

While the Cedefop/ETF and Cedefop/Eurofound merger options could be feasible, they were 
high-risk scenarios and there were difficulties in balancing the positive and negative effects. A 
Eurofound/EU-OSHA  merger  was  also  considered,  but  it  was  concluded  that  it  would  face 
significant  ‘stumbling  blocks’  as  EU-OSHA  is  highly  specialised.  Overall,  as  individual 
agencies  were  shown  to  be  overall  efficient  and  effective,  it  was  noted  that  one  of  the  key 
reasons for mergers – joining a ‘weaker’ organisation to a ‘stronger’ one – was absent. The 
previous  evaluation  also  concluded  that  there  was  no  straightforward  scenario  for 
mergers  and  that,  to  improve  efficiency,  there  was  rather  room  for  strengthening 
cooperation.  

336 Ibid. 

160 

 
 
 
The current evaluation supporting study confirmed these previous conclusions as there 
were no substantial changes to the mandates and activities during the current evaluation 
period.  The  2019  revision  of  the  tripartite  Agencies’  Founding  Regulations  did  not 
change/expand the mandates of the Agencies beyond their existing scope (see details on this 
analysis in Section 4.1.2.7 of the supporting study). 

Although the evaluation has not identified any grounds for mergers, scope for further synergies 
and cooperation has been identified. 

The table below summarises the benefits and costs of each merger scenario. The middle column 
provides  the  analysis  from  the  previous  evaluation.  The  right  column  provides  additional 
comments, in particular if there are new elements that could change the assessment from the 
previous evaluation. 

161 

 
 
 
Table 3: Overview of benefits and risks of agency merger scenarios 

Previous evaluation findings 

Current evaluation assessment 

Eurofound and Cedefop merger  
 (Common area: intersection of VET, skills mismatches, working and living conditions and labour 
market policies – Cedefop supply side / Eurofound demand side) 

Benefits 

Cost savings are possible for research roles 
(estimated at up to EUR 1.5 million) and 
back-office roles (estimated at up to EUR 1.5 
million) 

Potential synergies could be exploited 
through collaboration in areas such as long-
term unemployment, youth unemployment, 
skills, employment and migration. 

Potential to develop more efficient 
research processes, e.g. by pooling 
knowledge and working closer on 
methodological development and data 
collection. 

Cost savings are possible due to a 
reduction in both administrative 
roles and operational roles (e.g. 
some management roles). The cost 
of living crisis, inflationary pressure 
on salaries, and ever-increasing 
pressure on Agencies' budgets 
increase the weight of this benefit.  
Differences in salary coefficients 
make a merger with Eurofound 
more expensive than vice versa. A 
merger with Eurofound is more 
likely as Greece already has another 
decentralised agency, while Ireland 
does not.  

It is likely that more efficient 
processes for research and 
collaboration could increase the 
cost-effectiveness of the different 
outputs produced. The overall level 
of outputs produced by the two 
Agencies (as combined now) would 
more likely decrease due to an 
overall reduction in staff.  

Merging tripartite Agencies would likely 
be easier than merging a tripartite and a 
non-tripartite Agency. 

No change in governance; 
therefore, this is still valid. 

Risks 

Very broad thematic scope, risk of reducing 
or limiting priority to some fields or 
issues, such as VET 

No significant changes in 
thematic areas; therefore, this is 
still valid. 

There are likely to be higher costs related 
to additional governance and management 

Still valid. 

162 

 
 
 
 
costs associated with increased 
organisational complexity. 

Disruption in the short term through staff 
losses due to the need for relocation. 

Still valid. 

Potential deterioration in specific and 
unique thematic knowledge in some areas, 
along with weakened cooperation with EU 
institutions/agencies, Member States, and 
national and EU-level stakeholders. 

Closure costs are high, around EUR  2-3 
million per agency. 

No significant changes in 
thematic areas; therefore, this is 
still valid. 

Closure costs remain a risk and 
would increase in proportion to the 
growth of the agency’s budget.  

Cedefop and ETF merger  
(Common area: VET – Cedefop within the EU / ETF outside the EU) 

Benefits 

Cost savings are possible for research roles 
(estimated at up to EUR 2 million) and back-
office roles (estimated at up to EUR 1.5 
million) 

Practical difficulties in collaborating on 
capacity-building tools should be overcome, 
leading to a more efficient production of 
tools. 

Potential synergies could be exploited in 
developing methodological expertise, e.g. in 
skills anticipation and matching. 

163 

Cost savings are still likely due to a 
reduction in both administrative 
roles and operational roles (e.g. 
some management roles). 
Savings on operational roles are still 
likely to be higher than in the 
Cedefop/Eurofound merger due to 
more overlapping thematic 
expertise.  
The cost of living crisis, 
inflationary pressure on salaries, 
and ever-increasing pressure on 
Agencies’ budgets increased the 
weight of this benefit.  
Differences in salary coefficients 
would make the merger with the 
ETF more expensive than vice 
versa. 

The current level of collaboration 
on common tools between Cedefop 
and ETF is already assessed as high 
in this evaluation, as the Agencies 
cooperate on key tools where there 
are synergies. It is likely that more 
efficient processes for research and 

 
 
Potential to develop more efficient 
research processes, e.g. by pooling 
knowledge and working closer on 
methodological development and data 
collection. 

Closer fit between Cedefop and the ETF 
than between Cedefop and Eurofound 

Risks 

Some activities are unlikely to be 
continued due to restructuring. 

collaboration could increase the 
cost-effectiveness of the different 
outputs produced. The overall level 
of outputs produced by the two 
Agencies (as combined now) would 
more likely decrease due to an 
overall reduction in staff.  

No significant changes in 
mandates, therefore this still holds 
true. 

No significant changes in 
thematic areas; therefore, this is 
still valid. 

There are likely to be higher costs related to 
additional governance and management 
costs associated with increased 
organisational complexity. 

Still valid. 

Disruption in the short term through staff 
losses due to the need for relocation 

Still valid. 

The ETF’s ability to deliver advice 
perceived as independent may be hindered 

Different mandates and intervention 
logics of the Agencies. 

Different governance structures (Cedefop 
tripartite, ETF not) would need resolution: 
Member States might object if social partners 
have a voice in EU external action. 

Closure costs are high, around EUR 2-3 
million per agency. 

No significant changes in 
mandates/activities; therefore, this 
is still valid. 

No significant changes in 
mandates; therefore this is still 
valid. 

As governance structures remain 
different, this risk is still valid. 

Closure costs remain a risk and 
would increase in proportion to the 
growth of the agency budget. 

Loss of specialist knowledge in relation to 
working in partner countries (particularly if 
the ETF relocates). 

Still valid. 

Cedefop might emerge as the dominant 
partner, prioritising EU Member States  

Still valid. 

164 

 
 
Potential deterioration in specific and 
unique thematic knowledge in some areas, 
along with weakened cooperation with EU 
institutions/agencies, Member States, and 
national and European-level stakeholders. 

Still valid. 

Eurofound and EU-OSHA merger  
(Common area: occupational health and safety – Eurofound specificity / EU-OSHA specificity) 

Benefits 

Cost savings are possible for research roles 
(estimated at up to EUR 1 million) and back-
office roles (estimated at up to EUR 0.5-1 
million) 

Potential synergies could be created by 
pooling methodological expertise in 
researching working conditions and safety 
and health issues. 

Potential to develop more efficient 
research processes, e.g. by pooling 
knowledge and working closer on 
methodological development and data 
collection. 

Risks 

Closure costs are high, around EUR 2-3 
million per agency. 

Cost savings are still likely due to a 
reduction in both administrative 
roles and operational roles (e.g. 
some management roles). 
The cost of living crisis, 
inflationary pressure on salaries, 
and ever-increasing pressure on 
Agencies’ budgets increased the 
weight of this benefit of the merger.  

It is likely that more efficient 
processes for research and 
collaboration could increase the 
cost-effectiveness of the different 
outputs produced. The overall level 
of outputs produced by the two 
Agencies (as combined now) would 
more likely decrease due to an 
overall reduction in staff. 

Closure costs remain a risk and 
would increase in proportion with 
the growth in the agency’s budget. 

There are likely to be higher costs related to 
additional governance and management 
costs associated with increased 
organisational complexity. 

Still valid. 

Some activities are unlikely to be 
continued due to restructuring. 

Still valid. 

The risk that Eurofound becomes the 
dominant partner, and safety and health 
policy issues are given insufficient priority 
relative to other issues, reducing their 
visibility at European level. 

As Eurofound is still the larger 
agency, this risk is still valid. 

165 

 
 
Potential deterioration in specific and 
unique thematic knowledge in some areas, 
along with weakened cooperation with EU 
institutions/agencies, Member States, and 
national and EU-level stakeholders. 

Still valid. 

4.  Detailed coherence evidence 
(i) Joint activities, events and sharing data collection 

The already well-developed cooperation between Cedefop and the ETF, specifically on VET 
policy monitoring, was further developed as set out below. 

(a) Joint support for monitoring the deliverables of the Riga Conclusions in the field of VET. 
In  2019,  the  jointly  launched  Riga  questionnaire  resulted  in  the  final  report  Enhancing 
European cooperation in VET: outcomes of the Riga cycle337, which was presented at meetings 
of the Advisory Committee on Vocational Training and the Directors-General for Vocational 
Training  (2020)338.  The  two  agencies  were  also  entrusted  with  the  annual  monitoring  of  the 
implementation of the Council Recommendation on VET and the Osnabrück Declaration for 
2021-2030.  

(b) Regular joint contributions to work on national and regional qualifications frameworks in 
cooperation  with  UNESCO  (2017-2018,  2020)339.  Cedefop  contributed  to  the  Eastern 
Partnership conference on qualifications and mobility in 2019. In 2020, Cedefop published an 
article on  European Qualifications Framework  qualifications, to  which the  ETF contributed. 
The agencies also developed joint publications: the European inventory on validation of non-
formal and informal learning, and, together with  UNESCO, the global inventory of national 
qualifications frameworks. 

(c) The agencies are jointly implementing the European skills and jobs survey. The focus is on 
the  energy  and  crafts  sector  in  selected  ETF  partner  countries,  while  Cedefop  covers  EU 
countries  (2021-2023).  The  two  agencies  also  jointly  monitor  the  implementation  of  the 
Osnabrück Declaration, supporting digital skills and inclusion in VET (from 2020)340.  

(d) Cedefop shared its expertise to develop a tool for validating refugees’ work skills in Türkiye 
and identifying best practice for the Eastern Partnership event in October 2017. In addition, the 

337 Cedefop (2020), Enhancing European cooperation in VET: outcomes of the Riga cycle: progress in common 

priorities for 2015-20, available at: https://www.cedefop.europa.eu/en/publications/3084.  

338 Cedefop and ETF (2020), Enhancing European cooperation in VET: outcomes of the Riga cycle, available at: 

https://www.cedefop.europa.eu/files/3084_en.pdf.  

339 See, for example: ETF, Cedefop and UNESCO (2019), Global inventory of regional and national 

qualifications frameworks 2019: Volume I: Thematic chapters, available at: 
https://www.cedefop.europa.eu/files/2224_en_0.pdf. 

340 See, for example: ETF (2022), Osnabrück Declaration: supporting digital skills and inclusion in VET, 

available at: https://www.etf.europa.eu/en/news-and-events/news/osnabruck-declaration-supporting-digital-
skills-and-inclusion-vet.  

166 

 
 
 
 
ETF contributed to the Cedefop-OECD Experts Forum on upskilling, reskilling and employing 
adult refugees, and shared its experiences from Jordan and Lebanon. 

Cooperation between Eurofound and the ETF increased during 2017-2022. For the first time, 
the agencies cooperated on a joint survey. Typically, they cooperate on platform work, young 
people  not  in  employment,  education  or  training  (NEETs)  and  capacity  building  of  social 
partners. In particular, the two agencies cooperated on the below topics. 

(a) Social partnership: The two agencies increasingly on social partnership and dialogue and 
public-private partnerships in VET, with a particular focus on EU-level social dialogue. Joint 
events supported this, such as the ETF-Eurofound working seminar on social dialogue and skills 
development  (2021),  joint  participation  in  the  expert  workshop  on  capacity  building  for 
effective social dialogue (2021), and peer review of publications. 

(b) Digital economy: The two  agencies participated in or jointly organised seminars on the 
digital economy, platform work (2019, 2020, 2021), and the impact of COVID-19 on businesses 
(2021). The ETF  also supported the distribution of the online survey on living and working 
during COVID-19 among 10 partner countries.  

(c) Skills shortages and skills mismatch: Eurofound participated in the advisory board of the 
ETF’s initiative on skills demand in partner countries (2020). 

Cooperation  between  Cedefop  and  Eurofound  became  more  extensive  during  2017-2022. 
Prior  cooperation  emphasised  information  exchange  and  participation  in  events  rather  than 
developing joint outputs and activities. A notable example of cooperation was the joint EU-
wide  European  company  survey,  which  was  launched  in  2019  and  led  to  joint  analysis  and 
reporting. The two agencies also started working together on skills forecasting and analysis to 
integrate  Eurofound’s  job  tasks  approach  into  the  forecasting  analysis  for  the  EU  Skills 
Panorama (2017). The two agencies published the Skills forecast: trends and challenges to 2030 
report341.  Other  joint  publications  included  a  report  entitled  Workplace  practices  unlocking 
employee  potential342.  Eurofound  led  an  event  on  the  future  of  manufacturing,  and  the 
publication of a related study, to which Cedefop contributed with employment projections343.  

There is a longstanding cooperation between Eurofound and EU-OSHA and the two agencies 
had  previously  published  joint  reports  on  psychosocial  risks  at  work344.  The  two  agencies 
focused on themes such as psychological risks, mental health, self-employment, working time 
patterns, the issue of burn-out, older workers, and foresight345. In 2018, a foresight publication 

341 Cedefop, Eurofound (2018), ‘Skills forecast: trends and challenges to 2030’, Cedefop reference series No 

108, Publications Office, Luxembourg, available at: http://data.europa.eu/doi/10.2801/4492.  

342 https://www.cedefop.europa.eu/en/publications/2228  
343 Eurofound (2019), The future of Manufacturing in Europe, 

https://www.eurofound.europa.eu/observatories/emcc/fome.  

344 Evaluation of EU agencies in the field of employment and social affairs, 

https://ec.europa.eu/social/main.jsp?langId=en&catId=85&furtherNews=yes&newsId=9348. 

345 Eurofound (2017), Consolidated annual activity report of the Authorising Officer for the year 2017, p. 25, 
available at: https://www.eurofound.europa.eu/publications/annual-report/2018/consolidated-annual-
activity-report-of-the-authorising-officer-for-the-year-2017. 

167 

 
 
 
New  and  emerging  OSH  risks  associated  with  Information  and  Communication  Technology 
(ICT)  by  2025  was  published346.  In  May  2020,  the  European  Parliament  Committee  on 
Employment and Social Affairs held an exchange of views with then Eurofound Director Juan 
Menéndez-Valdés and the Director of EU-OSHA on the social and employment impacts of the 
COVID-19 pandemic347. 

(ii) Pan-European surveys 

There  was  increased  cooperation  between  the  Agencies  on  pan-European  surveys.  Such 
survey cooperation led to  wider benefits  such as cost  sharing  (particularly for the European 
company survey), aligning definitions and tools, facilitating data and best practice sharing, and 
overcoming methodological difficulties.  

-European company survey, 2019. Eurofound in cooperation with Cedefop, leading to a publication on 
the subject of workplace practices unlocking employee potential in 2020. 

-Career guidance policy and practice in the pandemic, 2020. Cedefop in cooperation with the ETF, the 
European Commission, the International Centre for Career Development and Public Policy, the ILO, 
the OECD and UNESCO, leading to a publication of the same name in 2020. 

-Living, working and COVID-19 survey, 2020-2022. Eurofound in cooperation with the ETF, leading 
to  a  publication  entitled  Living,  Working,  and  COVID-19  in  the  European  Union  and  10  EU 
Neighbouring Countries in 2022. 

-European skills and jobs survey, 2021-2024. A joint survey: Cedefop in cooperation with the ETF. 

(iii) Multilateral cooperation 

The  Agencies  also  engaged  in  several  multilateral  (involving  more  than  two  agencies) 
initiatives, in particular on digitalisation, the COVID-19 pandemic, Russia’s war of aggression 
against Ukraine, and readiness for future challenges / foresight.  

Led by Cedefop and the ETF and in cooperation with other international organisations (the ILO, 
the OECD and UNESCO), a flash joint international survey launched in March 2020 provided 
a snapshot of career guidance in the face of the COVID-19 pandemic. This led to a joint report 
published  by  Cedefop  entitled  Career  Guidance  policy  and  practice  during  the  COVID-19 
pandemic and a joint background note focusing on social dialogue and partnerships in response 
to the post-COVID environment. 

In 2020, Eurofound and EU-OSHA held a discussion with the European Parliament Committee 
on Employment and Social Affairs on the impacts of the COVID-19 pandemic. In the following 

346 EU-OSHA (2018), Foresight on new and emerging occupational safety and health risks associated with 
digitalisation by 2025, available at: https://osha.europa.eu/en/publications/foresight-new-and-emerging-
occupational-safety-and-health-risks-associated.  

347 See: https://www.eurofound.europa.eu/nl/events/european-parliament-discussion-with-eurofound-and-eu-

osha-impacts-of-the-COVID-19-pandemic.  

168 

 
 
 
 
year, Eurofound participated in the scoping phase of EU-OSHA’s psychosocial risk assessment, 
which covered the consequences of COVID-19 on workers’ mental well-being. 

Five EU agencies (EU-OSHA, Cedefop, Eurofound, the European Labour Authority and the 
ETF) and the European Parliament Committee on Employment and Social Affairs organised 
the ‘Youth First! Employment, skills and social policies that work for young Europeans in times 
of uncertainty’348 event in September 2022 to discuss policies on young people and challenges 
related  to  employment,  working  conditions  and  mental  health.  Follow-up  cooperation  is 
planned. 

Four agencies (EU-OSHA Eurofound, Cedefop and the European Institute for Gender Equality) 
provided input to Towards age-friendly work in Europe: a life-course perspective on work and 
ageing from EU Agencies in 2017. This publication was the outcome of a collaborative project 
with topics covered by different agencies according to their mandate and expertise.  

(iv) Thematic contributions to DG EMPL policies, events, working groups, and advisory 
committees 

Each agency interacted with DG EMPL according to its specificities. Most often, the agencies 
provide research, expertise and inputs into the policy documents or proposals, and participate 
in  a variety  of formal committees and other meetings. Examples of the ways in  which each 
agency provided input to DG EMPL policies and programmes are set out below. 

Eurofound provided input to EU policies in areas such as employment, social affairs, skills 
and  labour  mobility.  In  2019,  Eurofound  analysed  upward  convergence  in  the  EU, 
developments  in  the  state  of  middle  classes,  working  conditions,  platform  work,  work-life 
balance, representativeness studies of European social partner organisations, and involvement 
of social partners in national policymaking349. In 2021, Eurofound and EU-OSHA analysed the 
digitalisation  of  the  workplace,  telework  and  the  right  to  disconnect.  In  2021,  Eurofound 
material  contributed  to  the  2022  joint  employment  report  and  the  employment  and  social 
development in Europe report350.  

Cedefop contributed to DG EMPL work in the areas of VET and skills. The agency played an 
important  role  in  the  development  of  a  new  Europass  platform,  took  part  in  preparing  the 
blueprint for sectoral cooperation on skills351, and published a report on progress in greening 

348 European Labour Authority, https://www.ela.europa.eu/en/events/youth-first-employment-skills-and-social-

policies-work-young-europeans-times-uncertainty.  

349 DG EMPL (2020), Annual Activity Report 2019, p. 22, available at: 

https://commission.europa.eu/system/files/2020-06/empl_aar_2019_en.pdf. 

350 DG EMPL (2022), Annual Activity Report 2021, p. 21, available at: 

https://commission.europa.eu/system/files/2022-06/annual-activity-report-2021-employment-social-affairs-
and-inclusion_en.pdf.  

351 Blueprint for sectoral cooperation on skills website: 

https://ec.europa.eu/social/main.jsp?catId=1415&langId=en. 

169 

 
 
 
 
the economy and skills352. Cedefop cooperates regularly with DG EMPL on the Skills-OVATE 
project353. 

The  ETF  provided  expert  input  to  DG  EMPL  on  VET,  national  qualification  systems, 
work-based learning and vocational excellence. In 2017, together with Cedefop, it participated 
in the European Qualifications Framework (EQF) Advisory group. The ETF provided updates 
on the implementation of national qualifications systems in south-eastern Europe and Türkiye 
and provided analysis on future links of non-EU countries to the EQF by identifying policy 
options and criteria for prioritising countries354. In 2017-2018, the ETF provided expert input to 
DG EMPL for ministerial meetings as part of the Berlin Process355 on work-based learning and 
digital  skills  in  the  Western  Balkan  countries356.  In  2019,  the  ETF  contributed  to  the 
Employment and Social Affairs Platform involving representatives of DG EMPL, Ministries of 
Labour/Employment  and  public  employment  services357.  The  ETF  contributed  to  a  meeting 
organised by DG EMPL in October 2020 as part of the consultation ‘A strong social Europe for 
just transitions’ to prepare the action plan for the European Pillar of Social Rights358.  

EU-OSHA contributed to DG EMPL’s strategic priorities that focus on safe working conditions 
for all. It primarily developed tools for employers and workers and promoted the application of 
EU  occupational  safety  and  health  legislation  in  the  Member  States  with  support  from  DG 
EMPL. Together with Eurofound, EU-OSHA delivered research on workplace digitalisation, 
telework, and the right to disconnect, which contributed to DG EMPL’s understanding of the 
current challenges and opportunities faced by workers and employers359.  

The  agencies  also  cooperate  with  DG  EMPL  in  contributing  to  each  other’s  events  and 
supporting  each  other’s  communication  and  dissemination  activities,  for  instance  on  the 
European Vocational Skills Week (Cedefop)360.   

(v) Agencies’ contribution to the European Union Agencies Network on Scientific Advice 

Eurofound chaired the network between July 2021 and mid-2022 and hosted the 17th and 18th 
meetings of the European Union Agencies Network on Scientific Advice (EU-ANSA) meetings 
in  November  2021  and  May  2022.  EU-OSHA  hosted  an  inter-agencies  crisis  preparedness 

352 Cedefop (2020), Consolidated Annual Activity Report 2019, p.44, available at: 

https://www.cedefop.europa.eu/en/content/consolidated-annual-activity-report-2019-including-governing-
boards-analysis-and-assessment.  

353 Intellera consulting (2022), Cedefop Ex-Post Evaluation 2021, p. 9. 
354 ETF (2019), Consolidated Annual Activity Report 2018. 
355 The Berlin Process is an initiative of several EU Member States, under German leadership, to engage with the 
six Western Balkan partners and promote regional cooperation and the European perspective of the region. It 
consists of annual summits and a series of ministerial meetings. 

356 Ibid. 
357 ETF (2020), Consolidated Annual Activity Report 2019. 
358 ETF (2021), Consolidated Annual Activity Report 2020. 
359  European  Commission,  Directorate-General  Employment  Social  Affairs  and  Inclusion,  available  at: 
https://commission.europa.eu/system/files/2022-06/annual-activity-report-2021-employment-social-affairs-
and-inclusion_en.pdf. 

360  ICON-INSTITUTE  GmbH &  Co.  KG  Consulting  Gruppe  (2021),  Cedefop  Ex-post  evaluations 2019-2020, 

p. 25. 

170 

 
 
 
exercise in May 2023 to draw lessons from the COVID-19 crisis and increase preparedness for 
future crises. EU-OSHA and Eurofound (alongside the European Centre for Disease Prevention 
and Control, DG EMPL, DG SANTE, the ILO and the WHO) also jointly worked on the project 
‘Healthy Workplaces Stop the Pandemic’. Cedefop, Eurofound and EU-OSHA also conducted 
the EU-ANSA survey on the socio-economic effects of sustainable development and published 
a related report in May 2020. A follow-up in the form of a tripartite knowledge summit in 2024 
is under discussion. In 2022, Cedefop and Eurofound organised a knowledge-sharing seminar 
for EU-ANSA on quality jobs and the green transition. 

(vi) Agencies’ cooperation with the ILO 

The  Eurofound  European  working  conditions  survey  of  2015  led  to  a  joint  report  entitled 
Working anytime, anywhere: the effects on the world of work in 2017. A further joint report 
was published in 2019 on working conditions in a global perspective. Cedefop, the ETF and the 
ILO have jointly worked on skills anticipation and matching methods, including a compendium 
of six guides on ways to anticipate skills needs, the last volume dating from 2017. Cooperation 
between the ETF and the ILO on joint surveys saw increased use of online surveys partly due 
to the COVID-19 pandemic. A joint ETF-ILO global survey on the impact of COVID-19 on 
enterprises  fed  into  a  joint  report  issued  together  with  several  other  organisations,  called 
Skilling, upskilling and reskilling of employees, apprentices & interns during the COVID-19 
pandemic. The longstanding cooperation between EU-OSHA and the ILO on OSH in SMEs 
has  led  to  the  ILO  and  countries  outside  the  EU  (Canada,  US,  Singapore)  referencing 
EU-OSHA research outputs in publications, but also to these countries updating their national 
tools and policies addressing OSH risks in SMEs. 

5.  Agencies’ relevance to EU policy needs 
The four Agencies’ mandates, objectives and activities were highly relevant and the Agencies 
highly contributed to EU policy priorities. The Agencies developed their work programmes in 
response to EU policies and priorities. There was a clear link between the EU priorities and 
the Agencies’ key programming and reporting documents – the Agencies’ work was often 
referred  to  in  EU  policy  documents.  The  expertise  of  the  Agencies’  staff  was  leveraged  to 
inform  policy  and  programme  development  (see  Table  8  in  Annex  VII,  which  shows  how 
Agencies’  outputs  correspond  to  the  European  Commission’s  2019-2024  strategic 
priorities).  

Eurofound  closely  aligned  its  activities  with  three  main  areas  of  action  of  DG  EMPL: 
supporting  the European Semester, fostering stronger social dialogue, and promoting decent 
and safe working conditions for all361. For instance: 

-  Eurofound’s work on digitalisation and the platform economy was particularly relevant to 
promoting decent and safe working conditions for all (e.g. EU Platform Work Directive);  

361 DG EMPL (2020), Annual Activity Report 2019, p. 22, available at: 

https://commission.europa.eu/system/files/2020-06/empl_aar_2019_en.pdf.  

171 

 
 
 
-  Eurofound’s work also directly contributed to reporting on the principles of the European 

Pillar of Social Rights within the framework of the European Semester362;  

-  Eurofound’s report on cross-border labour inspections supported the discussion leading to 

the set-up of the European Labour Authority363;  

-  Eurofound also hosted webinars and #AskTheExpert sessions in cooperation with partners 

such as the European Institute for Gender Equality and the EU Presidency.  

EU-OSHA successfully contributed to EU OSH legislation and strategies. For instance: 

-  EU-OSHA contributed significantly to the development and implementation of successive 

- 

- 

strategic frameworks on health and safety at work (2014-2020 and 2021-2027);  
in  particular,  EU-OSHA’s  work  on  costs  to  society  of  work-related  injury  and  illness, 
psychological  risks,  and  the  ageing  workforce  was  quoted  in  the  2021-2027  strategic 
framework on health and safety at work;  
the strategic framework also outlined further areas of research that the Commission plans to 
carry out in cooperation with EU-OSHA (digitalisation, psychosocial risks, health and care 
sectors)  through  activities  on  OSH,  the  platform  economy,  work-related  musculoskeletal 
disorders, and the highly relevant healthy workplace campaigns;  

-  EU-OSHA  aligned  its  activities  with  EU  action  and  initiatives  and  contributed  to  policy 
planning through its healthy workplace campaigns (for instance on dangerous substances); 
in this respect, EU-OSHA’s ESENER survey was highly relevant as it provided comparable 
and otherwise hard to retrieve data on how businesses manage OSH (case study); 

-  EU-OSHA’s tools for OSH management, especially OiRA, have helped address EU policy 

needs by promoting a culture of prevention, particularly in SMEs. 

CEDEFOP closely aligned its work with EU upskilling and reskilling policies, in particular 
the  European  Skills  Agenda,  the  Council  Recommendations  on  VET,  the  Osnabrück 
Declaration, and the digital education action plan. For instance: 

-  Cedefop’s expertise on VET policies and its work on monitoring the implementation of VET 

across Member States informed relevant EU policy planning364;  

-  Cedefop’s skills forecasts were cited in the policy documents of several EU institutions; for 
example,  the  European  Parliament  noted  the  Agency’s  contribution  to  Europass,  the 
European skills index, the Copenhagen Process and the Skills Panorama365;  
the  agency  supported  the  latest  policy  developments  in  the  area  of  adult  upskilling  and 
reskilling;  its  outputs  were  relevant  to  both  EU-  and  national-level  policymakers  (case 
study);  

- 

362 Eurofound (2018), Consolidated annual activity report of the Authorising Officer for the year 2018, p. 4, 
available at: https://www.eurofound.europa.eu/publications/annual-report/2019/consolidated-annual-
activity-report-of-the-authorising-officer-for-the-year-2018. 
363 Eurofound (2019), Consolidated Annual Activity Report 2019, p. 8. 
364 Cedefop (2021), Minutes of the 99th management board virtual meeting, available at: 

https://www.cedefop.europa.eu/en/about-cedefop/public-documents/minutes-conclusions.  

365 European Parliament (2022), 2020 Discharge: European Centre for the Development of Vocational Training 

(Cedefop), available at: https://www.cedefop.europa.eu/en/content/2020-discharge-european-centre-
development-vocational-training-cedefop. 

172 

 
 
 
-  Cedefop  organised,  in  cooperation  with  the  European  Commission  and  the  European 
Economic  and  Social  Committee,  a  number  of  policy  learning  events  on  upskilling 
pathways, bringing together governments, social partners and civil society to help draft and 
implement the Council Recommendation on upskilling pathways for adults.  

The  ETF  closely  aligned  its  work  with  EU  policy  priorities.  The  agency  also  provided 
significant contributions to the EU’s policies and programmes on external relations.  

-  The  ETF  provided  analyses  of  partner  countries’  labour  market  developments  and 

socio-economic contexts that were not otherwise available (stakeholders). 

-  The  ETF’s  research  on  skills  demand  anticipation  (e.g.  the  creation  of  the  Skills  Lab 
Network of Experts) helped achieve the EU objective of helping people thrive in the green 
and digital transitions (Action 2 – European Skills Agenda). It did so by providing valuable 
insights on how developments in skills needs vary outside the EU (case study). 

-  The ETF supported DG INTPA, in particular through contributions to the Africa Continental 
Qualifications Framework and training of EU  delegations. This support has increased the 
visibility and impact of the ETF and has enabled the agency to access additional funding. 
However,  this  support  has  also  extended  the  ETF’s  geographical  scope  beyond  partner 
countries and has led to questions as to whether this support leads to the agency’s resources 
being spread too thinly366. 

While the ETF’s work is highly relevant, the last point illustrates that its close relations with 
several different Commission services led to  competing expectations about the geographical 
scope and focus of its activities. Consequently, there is room to clarify the longer-term vision 
for the agency and which of its activities should be prioritised. 

366 IAS (2022), Final audit report on cooperation and coordination mechanisms between the European Training 

Foundation (ETF) and the European Commission (EC) services. 

173 

 
 
 
 
 
ANNEX X: LIST OF INDICATORS PER AGENCY 

Table 1: Eurofound indicators 

Indicator 

Targets 

Comments 

KPIs 

Budget implementation % 
Activity-based-budget (ABB) implementation % 

Positions filled % 

Number of staff per category 
Gender balance (%) 
Programme delivery % (outputs delivered) 
No. of key EU policy documents quoting Eurofound 
Recognition of the scientific quality of Eurofound's 
research  (number  of  references  in  peer  reviewed 
publications) 
Number of completed publications published 
No. of EU policy documents quoting Eurofound 
Number of projects/initiatives (including E-surveys) 
to explore the impact of COVID-19 
Uptake of Eurofound knowledge in the media 

Core activities, 
outputs & 
results 

Uptake  of  and  engagement  with  Eurofound’s 
knowledge  through  its  website  and  other  corporate 
platforms 

Outreach 
communication 

User 
satisfaction 

Contributions to policy development through events/ 
Eurofound's  engagement  with  stakeholders 
in 
meetings and events 
Contributions to EU-level events  

Registrations to participate in webinars % 

used 

publications 

Number of downloads in PDF format 
Number of COVID-19 items published  
Number of HTML page views  
Satisfaction  with 
satisfied/satisfied) % 
Stakeholders  agreeing  that  Eurofound  publications 
provide high quality content % 
Stakeholders  who 
recommended  a  Eurofound 
publication to a colleague or someone else in the past 
year % 
Stakeholders  agreeing  that  Eurofound  provides 
reliable data in its outputs % 

(very 

174 

98% 
- 
97% (93% in 
2021) 
- 
- 
80% 
- 
- 

- 
- 
- 

- 
- 

- 

- 
- 

- 
- 
- 
- 

- 

- 

- 

Available for 2020 
only. 
Introduced in 2020 
In 2020, the data 
breakdown by activity 
area was introduced. 
Previously 10 most 
downloaded 
publications were 
reported. 

Data available since 
2020. 

 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Satisfaction  with  the  quality  of  content,  reliability, 
policy relevance and format of Eurofound's outputs 
(%) 

- 

% budget on continuation of previous WP  

Rate of outturn (payment appropriations) (EUR) 
Total budget (EUR) 
Budget per title (EUR) 
Engagement of staff % 
Number  of  interinstitutional  tenders  Eurofound 
participated in 

9% in 2017 
1% in 2018 
- 
- 
- 
- 
- 

Other 

Table 2: Cedefop indicators 

Indicator 

Budget implementation rate (%) 

Timeliness  of  recruitment  procedures  (number 
of days) 

Target 

- 

60 days 

Comments 

Days from deadline for 
applications to 
finalisation of selection 
board report to AIPN. 

Establishment plan filled (%) 

95% 

Number of staff per category 

KPIs 

% of total staff per category 

Gender balance (%) 

Resource allocation per activity (#) 
Policy  documents  to  the  preparation  of  which 
Cedefop has contributed (of EU institutions) (#) 
Policy  documents  to  the  preparation  of  which 
Cedefop  has  contributed  (of 
international 
institutions) (#) 
EU policy documents citing Cedefop's work (#) 
References to Cedefop work in EU-level policy 
documents (#) 
Citations of Cedefop in the literature (#) 
Number of publications (#) 
Policy documents of international organisations 
citing Cedefop's work (#) 
References  to  Cedefop  work  in  documents 
issued by international organisations (#) 
Active participation in conferences and events 

175 

Core activities, 
outputs & results 

See comment  Depends on the 

category. The numbers 
are set in the 
establishment plan. 

See comment  Depends on the 

category as above. 
Target was introduced 
in 2021. 

60% female 
/40% male 
- 
- 

- 

- 
- 

- 
- 
- 

- 

- 

 
 
 
 
 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
  
 
 
 
 
 
 
 
 
Outreach 
communication 

User satisfaction 

Other  

Cedefop's contribution to meetings that inform 
policies and their implementation (#) 
Number  of  meeting/events  organised  by 
Cedefop (#) 
Type of audience (%) 
Number of participants at Cedefop events (#) 
Number of events organised virtually 
Downloads of Cedefop publications (#) 
Downloads of Cedefop briefing notes (#)  

Number of videos (#) 

Number of views on social media (#) 
Visits (#) 
Unique visitors (#) 
Page views (#) 
Skills Panorama portal users 
Skills Panorama portal sessions 
Skills Panorama traffic increase (%) 
Facebook followers (#) 
Twitter followers (#) 
Facebook followers increase (%) 
Twitter followers increase (%) 
LinkedIn followers (#) 
Media  coverage,  take-up  of  articles  and  press 
releases (#) 
Quality  and  expected 
organised (%) 

impact  of  events 

Stakeholders’  assessment  of  the  quality  and 
usefulness of agency’s services 

- 

- 

- 
- 
- 
- 
- 
- 

- 
- 
- 
- 
- 
- 
- 
- 
- 
- 
- 
- 
- 

- 

- 

Rate of outturn (payment appropriations) (%) 

Rate  of 
appropriations (budget execution) (%) 

implementation  of  commitment 

- 
98% 

Introduced in 2020. 

Not available for the 
full evaluation period. 
Available since 2018. 

Introduced in 2019 and 
available every two 
years. 

The target was 97% in 
2017 but increased to 
98% since 2018. 

the 

Rate  of  payments  completed  within 
legal/contractual deadlines (%) 
Total budget (EUR) 
Budget per title (EUR) 
Budget execution per title (%) 
Financial management statistics (EUR) 
Cedefop’s  consultation  with  key  stakeholders 
(%) 
Engagement of Cedefop staff members (%) 
Number and type of procurement procedures (# 
and %) 
Rate  of  accepted  audit 
implemented within agreed deadlines  

recommendations 

- 

- 
- 
- 
- 
- 

- 
- 

- 

176 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Environmental indicators 

See comment  Indicators were 

introduced in 2020, the 
targets were set in 2021. 

Visits to Europass website (#) 
Europass CVs generated online (#) 

- 
- 

Table 3: ETF indicators 

Indicator 

Target 

Comments 

and 

support 

Commitment  appropriation  implementation 
(%) 
Payment appropriations cancellation rate (%) 
Rate (%) of outturn 
Payments Executed out of year's budget (PAY 
C1+ PAY C8T3) / Budget 
Administrative 
coordination/operational staff ratio (%) 
Staff engagement in satisfaction survey (%) 
Average vacancy rate (%) 
Activities completion rate (%) 
Timely  achievement  of  activities  of  the  WP 
(%) 
Timely submission of WP (%) 
Rate  of  countries  successfully  progressed 
within  stage  or  next  stage  of  policy 
development (%) 
Requests for assistance from EU Services and 
Delegations (#) 
Geographic  and  thematic  coverage  of  EU 
Requests 
to  projects  and 
for  support 
programmes design (%) 
Share of ETF budget invested in EU requests 
(%) 

Web news items 

Knowledge  products  finalised  and  shared 
externally vs initial plan (%) 
Knowledge  products  finalised  in  time  vs 
initial deadline (%) 
ETF  knowledge  products  contributing  to 
Policy Advice (#) 
Share 
knowledge 
contributing to Policy Advice (%) 
Uptake  of  the  developed  ETF  knowledge 
products 

of  ETF 

products 

177 

98% 

<2% 
96% 
- 

<30% 

2% increase 
<5% 
90% 
80% 

100% 
70% 

- 

- 

- 

- 

- 

- 

- 

- 

- 

Not monitored since 
2021. 

Introduced in 2021. 

Introduced in 2021. 

Available for 2017 
and 2018. 
Introduced in 2021. 

Introduced in 2021. 

Introduced in 2021. 

Introduced in 2021. 

Introduced in 2021. 

KPIs 

Core activities, 
outputs & results 

 
 
 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
  
  
Mechanisms  for  national  and  sub-national 
coordination  in  VET  introduced  in  partner 
countries 

Participation in QA policy learning forum 

Requested  to  provide  support  in  eligible 
partner countries 
Invitations/contributions  from  third  parties 
for  ETF  participation  in  external  events  and 
knowledge products (articles, blogs, etc.) (#) 
Number  of  participants  in  international  and 
regional events 

Number of downloads of publications (#) 

Number of unique visitors to the website 

Facebook (# of total followers) 
Twitter (# of total followers) 
LinkedIn (# of total followers) 
YouTube views (#) 
Instagram (# of total followers) 
OpenSpace followers (#) 
Weighted  sum  of  No.  of  people  reached 
through digital media, downloads, and events 
Growth  in  number  of  total  subscribers  to 
Facebook, LinkedIn, Twitter (%) 
Post  link  clinks  on  Facebook,  LinkedIn, 
Twitter (% increase from previous period) 
Rate  of  positive  feedback  from  EC  project 
request implemented in year (%) 
Audit recommendation implementation (%) 
Total budget (EUR) 
Budget per title (EUR) 
Commitments (#) 

Timely payments (%) 

Average  time  to  assess  and  pay  mission 
expenses reimbursement (# of days) 
Budget transfers (#) 
Invoices (#) 
Payments (#) 
Staff (% of budget) 
Infrastructure (% of budget) 

178 

85% of 
targeted 
countries 
50% of partner 
countries:2017 
14 partner 
countries: 
2018 
16 partner 
countries: 
2019 and 2020 
50% 

- 

- 

- 

- 

- 
- 
- 
- 
- 
- 
- 

- 

- 

80% 

90% 
- 
- 
- 
90% 

15 days 

<10 
- 
- 
- 
- 

Available until 2020. 

Target is not set since 
2021. 

Available until 2020. 

Available for 2017 
and 2018. 
Available for 2017 
and 2018. 
Available for 2017 
and 2018. 

Introduced in 2021. 
Introduced in 2018. 
Introduced in 2021. 
Available until 2020. 

Introduced in 2021. 

Introduced in 2021. 

Available until 2020. 

In 2017 the target was 
lower at 80%. 
Target set in 2021. 

Target set in 2021. 

Outreach 
communication  

Other  

 
 
  
  
  
  
  
  
  
  
  
  
  
 
 
 
 
 
 
 
 
 
 
 
 
Operational Activities (% of budget) 

- 

See comment  The target set for each 

Budget utilisation by activity area 

Engagement  rate  of  active  members  in  ETF 
networks (%) 
Number  and 
stakeholders 
typology  of 
engaged in ETF activities (# of stakeholders) 
Number  and 
stakeholders 
typology  of 
in  ETF  activities  (%  primary 
engaged 
stakeholders) 
% of key country stakeholders participated in 
ETF activities / No of stakeholders (%) 
Level of stakeholders’ engagement in Policy 
Advice (%) 
People  engaged  with  Open  Space  (#  of 
members) 
Participants in green skills award (# countries 
covered in applications) 
Number of people voting in green skills award 
Annual governing board decisions (#) 

Approved vs Proposed GB decisions (%) 

Completion  rate  of  Level  1  monitoring  by 
partner country (%) 
Calls for tender (#) 

Average  time  consumed  by  procurement 
procedures -launch to award (# of days) 

Contracts prepared (#) 
Contract amendments (#) 
L&D training sessions (#) 
Recruitment procedures launched (#) 
Interinstitutional tenders (#) 
agreements 
partnership 
Share 
operationalised  through  annual  action  plans 
and  implemented  through  joint  activities, 
events (%) 
Environmental indicators 

of 

activity area and is 
reported in Annex 3 
of the evaluation 
report (p. 91-92). 
Introduced in 2021. 

Introduced in 2021. 

Introduced in 2021. 

- 

- 

- 

90% 

Available until 2020. 

- 

- 

- 

- 
- 
100% 

- 

- 
Open tenders: 
<5 months 
Negotiated 
procedures: <2 
months 
- 
- 
- 
- 
- 
- 

Introduced in 2021. 

Introduced in 2021. 

Introduced in 2021. 

Introduced in 2021. 
Introduced in 2018. 
Introduced in 2018, 
target set in 2021. 
Introduced in 2021. 

Introduced in 2018. 
Introduced in 2018, 
target set in 2021. 

Introduced in 2018. 
Introduced in 2018. 

Introduced in 2018. 
Introduced in 2021. 
Introduced in 2021. 

- 

Introduced in 2021. 

179 

 
 
 
 
 
 
Table 4: EU-OSHA indicators 

KPIs 

Core activities, 
outputs & results 

Outreach 
communication 

User satisfaction 

Indicators 

Target 

Comments 

Budget implementation (%) 

Budget per title (%) 
Share of outputs delivered in the planning year vs 
planned  outputs,  calculated  on  the  basis  of 
completion status/Annually (%) 

Work Programme implementation (output status) 

Number  of  publications  (reports,  evaluations, 
studies, etc.)  
Covid-19: Resources for the workplace (absolute 
number) 
Covid-19 Related Actions  
Uptake  of  electronic  information  for  Strategic 
objectives 1, 2, 4 (%) 
Number of events organised by the Agency (under 
the activity 'Awareness raising actions' and under 
'FAST') (absolute value) 
Number  of  stakeholders  reached  through  events 
(absolute  value) 
through 
networking 
Outreach  capacity  of  intermediaries  through 
networking  -  Events  count  across  all  activities 
across all priority areas where work of the Agency 
has been actively presented (policy and workplace 
practice oriented), either organised by the Agency 
or organised by others/Annually (absolute value) 

/  Reach  of  users 

Uptake of publications etc. (downloads) (%) 

Visitors to EU-OSHA's websites (absolute value) 
/ Reach of online users  
Reach via websites (absolute value) 
Social media statistics 
Stakeholders’  assessment  of  the  quality  and 
usefulness of agency’s services: EU Added Value 
(%) 
Stakeholders’  assessment  of  the  quality  and 
usefulness  of  agency’s  services:  Relevance  to 
needs (%) 
Stakeholders’  assessment  of  the  quality  and 
usefulness of agency’s services: Usefulness (%) 
Stakeholders’  assessment  of  the  quality  and 
usefulness  of  agency’s  services:  Impact  on 
workplace practice (%) 
Stakeholders’  assessment  of  the  quality  and 
usefulness of agency’s services: Impact on policy 
(%) 

180 

95% 

- 
90% 

- 

- 

- 

- 
5% increase per 
year 
200-250 

10 000 

Reduced to 90% since 
2020. 

Qualitative updates are 
available. 

Available for 2021. 

Available for 2021. 

Introduced in 2019, the 
target was reduced to 80 
in 2021. 
Available for 2017 and 
2018. 

350 

Introduced in 2019. 

5% increase per 
year 
10% increase 
per year 
2 500 000 
- 
80% 

Available for 2017 and 
2018. 
Available for 2017 and 
2018. 
Introduced in 2019. 
Available for 2017. 
Introduced in 2019. 

80% 

Introduced in 2019. 

80% 

Introduced in 2019. 

80% (except for 
2020 when it 
was 70%) 
80% (except for 
2020 when it 
was 70%) 

Introduced in 2019. 

Introduced in 2019. 

 
 
 
 
 
 
 
 
 
  
  
  
  
  
  
Relevance, usefulness, EU Added Value, Impact 
breakdown  per  activity  (data  available  for  each 
activity implemented) (%) 

Quality of EU-OSHA's work (%) 

Budgetary  and  Financial  Management  (€)  - 
Revenue  breakdown,  Expenditure  breakdown, 
Specific projects breakdown 
Total budget (EUR) 
Activity based costing 2016 by activity 
Fulfilment of payment appropriations (%) 
Implementation  of  commitment  appropriations 
(%) 
Staff breakdown by contract type, function group 
and gender balance  
Distribution  by  gender  of  TAs  posts  in  grades 
AD10 to AD14 
Staff distribution by nationality  
Gender  balance  at  the  senior  management  level 
(absolute value) 
Gender  balance  on 
(absolute value) 

the  management  board 

OSHmail newsletter subscribers (absolute value) 

Other 

Available for 2017 and 
2018. 

- 

80% 

- 

- 
- 
- 
95% 

- 

- 

- 
- 

- 

5% increase 

Available for 2017 and 
2018. 

181 

 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ANNEX XI: COST-EFFECTIVENESS OF THE AGENCIES’ KEY ACTIVITIES 

The  table  below  summarises  the  results  of  the  activity-level  cost-effectiveness  analysis  in 
Annex 6 of the supporting study. Based on this analysis, a score is assigned to each of the key 
activities  with  respect  to:  a)  reducing  costs;  and  b)  improving  effectiveness.  The  following 
scoring framework is used: 

•  Costs:  3  =  no  scope  to  reduce  costs;  2  =  some  limited  scope  to  reduce  costs;  1  = 

significant scope to reduce costs 

•  Effectiveness: 3 = no scope to improve effectiveness; 2 = some limited scope to improve 

effectiveness; 1 = significant scope to improve effectiveness   

The scores for cost and effectiveness are then summed for each individual activity to give an 
overall  cost-effectiveness  score  (maximum  6  =  no  scope  to  reduce  costs  or  improve 
effectivness, minimum 2 = significant scope to reduce costs and improve effectiveness). The 
table lists the agencies’ key activities in descending order, from the highest to the lowest score, 
and includes a summary of the findings that contributed to this score.  The assessment is based 
on expenditure data, external evaluations and stakeholder feedback via surveys and interviews. 

Table 1 - Cost-effectiveness of the Agencies’s key activities 

Agency and 
activity 
heading 

Score 
(out 
of 6) 

Summary 

Eurofound: 
Employment 
and labour 
markets 

Eurofound: 
Quality of 
life/living 
conditions 

Cedefop: VET 
systems and 
institutions 

6 

6 

6 

Effectiveness = 3, cost = 3 

Stakeholders have stated that the ‘living and working in the EU monitor’ is one of 
Eurofound’s main achievements and that the availability of databases (such as the 
European Jobs Monitor) has allowed stakeholders to better understand recent 
labour market and employment trends. Providing such information means that 
stakeholders are better equipped to improve labour market functioning in their 
respective countries. Interviewees also noted costs had been reduced over the 
evaluation period by reducing the amount of work that Eurofound contracts out 
under this activity heading. This has helped reduce the amount of money spent on 
completing projects. There is no scope to reduce costs without compromising 
quality. 

Effectiveness = 3, cost = 3 

Stakeholder feedback shows that stakeholders have been very impressed with 
Eurofound’s performance on ‘quality of life’. For example, the 2020 Eurofound 
user survey reported 64% of respondents found this activity to be relevant to their 
work. There is no scope to reduce costs without compromising quality. 

Effectiveness = 3, cost = 3 

Cedefop’s monitoring reports that use data from VET policy monitoring are highly 
rated by stakeholders: 72% of stakeholders rated them as being of very or rather 
high quality. These outputs are also highly rated by staff and viewed as cost-
effective. In a difficult economic climate of rising costs and inflation, ReferNet and 
its related outputs can be viewed as maintaining a good level of cost-effectiveness 

182 

 
 
 
 
Cedefop: 
Skills and 
labour market 

ETF: 
Employment, 
skills and 
employability 
(including 
skills and 
migration) 

ETF: 
Qualifications 
and 
qualification 
systems 

EU-OSHA: 
Anticipating 
change 

Eurofound: 
Reacting to 
ad-hoc 
information 
requests 

6 

6 

6 

6 

6 

while operating within a fixed budget. No opportunities have been identified for 
cutting costs that would not result in compromising quality. 

Effectiveness = 3, cost = 3 

Under this activity heading, Cedefop developed a tool for online vacancy analysis 
(Skills-OVATE) over the evaluation period. An ex-post evaluation of Skills-
OVATE revealed that stakeholders greatly appreciate quick access to the data, the 
ability to compare countries, and the effective visualisations. There is no scope to 
reduce costs without compromising quality. 

Effectiveness = 3, cost = 3 

Before launching the Skills Lab Network of Experts in 2021, preparatory work was 
carried out by the ETF to understand what the added value of such a network would 
be. This included a survey, expert consultations and a feasibility analysis that 
confirmed that a thoughtfully structured ETF network of labour market experts 
could fill existing information gaps. There is no scope to reduce costs without 
compromising quality. 

Effectiveness = 3, cost = 3 

Stakeholder feedback on the Torino Process (TRP) demonstrates a high regard for 
the ETF’s monitoring role through this initiative. Interviewees highlighted that TRP 
provides valuable insights, contributes to policy dialogue, and informs the agency 
on partner country needs. The process was also praised by partner country 
governments, EU institutions and a partner country civil society organisation. From 
2019 to 2021, the ETF set targets in its single programming documents for 50% of 
partner countries to use the TRP for national and/or EU policy dialogue; by 2019, 
78% of partner countries were using the TRP for policy dialogue. There is no scope 
to reduce costs without compromising quality. 

Effectiveness = 3, cost = 3 

The foresight activity under this heading was used to inform national OSH 
strategies and national policy debates. The evaluation of the EU-OSHA’s Large 
scale Foresight Activity dedicated to digitalisation and OSH concluded that this 
activity was managed and implemented in a cost-effective way. The budget was 
used to produce a large number of outputs, including studies, scenario-building and 
dissemination workshops, and event materials. Additionally, EU-OSHA actively 
looked for ways to increase cost-effectiveness by identifying potential synergies, 
such as those with the Joint Research Centre that allowed the agencies to cooperate 
on foresight topics and share production costs. This, combined with the fact that 
actual expenditure on this activity heading was lower than planned over the 
evaluation period, indicates that the budget allocated to ‘Anticipating change’ at the 
planning stage could be reduced. However, no scope has been identified to reduce 
expenditure below the agency’s current expenditure level without reducing quality. 

Effectiveness = 3, cost = 3 

The fact that actual expenditure on this activity heading was lower than planned 
over the evaluation period indicates that the budget allocated at the planning stage 
could be reduced – however, no scope has been identified to reduce expenditure 
below the agency’s actual expenditure level without reducing quality. 

183 

 
 
Eurofound: 
Social 
dialogue and 
industrial 
relations 

EU-OSHA: 
Awareness 
raising and 
communicatio
n 

EU-OSHA: 
Facts and 
figures 

5 

5 

5 

EU-OSHA: 
Tools for OSH 
management 

5 

EU-OSHA: 
Networking 
knowledge 

5 

Effectiveness = 2, cost = 3 

No scope to reduce costs without compromising quality has been identified, and 
stakeholder feedback on this activity is positive. However, there is scope for 
Eurofound to improve its work in the area of industrial relations by having more 
regular contact with social partners at EU and national levels. 

Effectiveness = 2, cost = 3 

The agency has been creating more synergies between its awareness-raising 
activities and other activities to create cost efficiencies. Interviews with EU-
OSHA’s focal points (FOPs) indicate that national stakeholders have been 
increasingly expressing their interest in participating in the healthy workplaces 
campaigns. This suggests that this activity is meeting its objective of making 
multiple beneficiaries and intermediaries aware of workplace risks and how to 
prevent them. However, translation challenges were reported as part of the 
evaluation of the healthy workplaces campaign (2018-2019). While it was 
acknowledged that EU-OSHA did make an effort to translate resources into other 
languages, it was noted that more and better-quality translations are required.   

Effectiveness = 2, cost = 3 

EU-OSHA’s FOPs found ESENER to be a useful tool to rethink and redesign their 
OSH policies and encourage businesses and workers to take action in response to 
specific risks. EU-OSHA’s bi-annual stakeholder surveys support this view: most 
respondents are satisfied with ESENER (95% in 2022). External evaluations found 
that OSH overview activities were of good value with the available resources. The 
2020 ex-post evaluation stated that objectives were achieved and resulted in a 
greater understanding among stakeholders. However, the same evaluation noted 
that more resources would have been beneficial, especially on dissemination 
activities and data quality checks. Dissemination and communication were also 
noted as limiting the effective use of good practices on work on micro and small 
enterprises. 

Effectiveness = 2, cost = 3 

The 2020 mid-term evaluation of the Online Interactive Risk Assessment (OiRA)[1] 
(a key tool for OSH management) found that OiRA was highly efficient in terms of 
achieving its outputs and outcomes. It had saved costs at national level by 
providing readily available online risk assessment tools, thereby eliminating the 
need to develop such tools from scratch. EU-OSHA stakeholders’ assessments 
confirm the growing satisfaction and popularity of OiRA, with satisfaction rates 
reaching 93% in 2022. However, the mid-term evaluation noted that the uptake of 
tools could be improved by promoting them more.  

Effectiveness = 2, cost = 3 

Respondents to the stakeholder survey indicated that the main achievements of the 
agency between 2017 and 2022 include underpinning the EU strategic framework 
on health and safety at work and carrying out good quality work to update 
information on health and safety topics. The activity also exceeded its set targets 
from 2019 to 2022 in terms of usefulness, relevance, EU-added value and impact, 
based on the assessment of stakeholders. However, compared to other activities of 
the agency, this activity had a low share of planned outputs delivered on time 
between 2017 and 2022 (79%). There is no scope to reduce costs without 
compromising quality. 

184 

 
 
Eurofound: 
Corporate 
communicatio
n and 
infrastructure 

Eurofound: 
Survey 
management 
and 
development 

Cedefop: 
Transversal 
activities 

ETF: VET 
provision and 
quality 

EU-OSHA: 
Strategic and 
operational 
activities/ 
networking 

4 

4 

4 

4 

4 

Effectiveness = 2, cost = 2 

While staff generally consider activities under this heading to be cost-effective, 
difficulties in finding the right balance of in-person, hybrid or online events have 
been identified. Fully understanding which model works best for different types of 
meetings and events would improve cost-effectiveness by promoting networking 
and engagement while reducing the overall cost of meetings and events. Changes 
could also be made to make it clearer to stakeholders which output is most relevant 
to them (e.g. publishing just one monitoring report on a topic rather than several 
reports). 

Effectiveness = 2, cost = 2 

Surveys are a large and crucial investment for Eurofound; the data collected are 
highly valued by stakeholders. However, there is the potential for Eurofound to 
reduce costs by involving other agencies in the development of the surveys. This 
could make the surveys more relevant to these agencies and their stakeholders and 
open up the possibility to share production and design costs with these other 
agencies. Similarly, work to find the most cost-effective mix of face-to-face 
interviews, CATI and online methods is needed to ensure data quality is not 
reduced as a result of moving away from face-to-face interviewing. 

Effectiveness = 2, cost = 2 

Challenges to find the right balance of in-person, hybrid or online events have been 
identified. Fully understanding which model works best for different types of 
meetings and events would improve cost-effectiveness by promoting networking 
and engagement while also potentially reducing the total cost of meetings and 
events. 

Effectiveness = 1, cost = 3 

A key area of the agencies’ work that needs to be improved is making support for 
capacity building of teachers and trainers relevant and improving work-based 
learning – 53% of EU stakeholders found these outputs to be only slightly relevant. 
This, coupled with the fact that the agency has regularly spent more on this 
headline activity than planned over the evaluation period, indicates that there is 
room to improve the cost-effectiveness of this activity by making its outputs more 
relevant to stakeholders.  

Effectiveness = 2, cost = 2 

Linked to the move to online/hybrid meetings and associated translation/ language 
barriers and reduced networking opportunities, engagement in meetings fell over 
the evaluation period. However, online/hybrid meetings have contributed to cost 
savings (e.g. a reduction in travelling costs and time) and promoted more 
exchanges with experts located outside the EU. Fully understanding which model 
work best for different types of meetings and events and what translation services 
are optimal would improve cost-effectiveness by promoting networking and 
engagement while potentially reducing the total cost of meetings and events.  

185 

 
 
ANNEX XII: RESULTS ACHIEVEMENT 

The expected results (result indicators or RI) for each agency are presented below and 
correspond to the results of the individual intervention logic (see Annex VI). 

Table 1 – Expected results, as presented in the Agencies’ intervention logics 

Eurofound 

Cedefop 

ETF 

EU-OSHA 

1  Key stakeholders 
are provided with 
information that 
supports the 
improvement of 
working conditions 
and the promotion 
of sustainability of 
work over the life 
course 

2  Management and 

labour are provided 
with information 
that supports social 
dialogue on 
industrial relations 

Policymakers and 
stakeholders are 
provided with high-
quality relevant 
evidence on VET, 
skills and 
qualifications 
produced by Cedefop 
in strategic areas of 
operation 

EU-level and 
national actors 
benefit from 
exchanges supported 
by Cedefop in terms 
of policy learning  

3  Key stakeholders 

are better equipped 
to support the 
improvement of 
labour-market 
functioning and 
inclusiveness 

4  Key stakeholders 
have increased 
understanding of 
ways in which to 
improve the quality 
of life, public 
services and society 

5  Key stakeholders 
are provided with 
tools and 
information to help 

Partner countries 
receive relevant and 
timely support to 
improve the 
governance of their 
human capital 
development systems  

Policymakers and 
researchers are 
provided with good 
quality data, enabling 
them to take timely 
and effective action on 
new and emerging 
risks 

Partner countries 
receive relevant and 
timely support on 
labour market polices 
and skills anticipation 

Partner countries 
receive relevant and 
timely support on 
improving lifelong 
learning provision 

Policymakers and 
researchers better 
understand current 
OSH risks and their 
health effects thanks to 
information provided 
by EU-OSHA 

More good quality 
tools are available to 
SMEs 

Partner countries 
receive relevant and 
timely support to 
modernise national 
qualification 
frameworks and 
systems 

Partner countries 
receive relevant and 
timely information on 
the performance of 

Multiple beneficiaries 
and intermediaries are 
aware of workplace 
risks and how to 
prevent them due to 
EU-OSHA campaigns  

The OSH community 
benefits from an 
increased body of 

186 

 
 
 
 
 
 
 
them anticipate and 
manage the impact 
of change 

6  Key stakeholders 
are provided with 
information to help 
them improve social 
cohesion and 
promote 
convergence in the 
EU on better living 
and working 
conditions 

7 

lifelong learning 
polices and systems 

high-quality 
knowledge  

Partner countries 
receive relevant and 
timely support to 
enable them to offer 
more opportunities for 
entrepreneurial 
learning and enterprise 
skills development 

EU-OSHA 
stakeholders and 
networks take part in 
EU-OSHA activities 
and their needs are met 
by these activities 

EU external relations 
policymakers receive 
relevant advice 

Source: Supporting study, based on Agencies’ planning and reporting documents 

Before reviewing the results indicators specific to each agency, Table 2 provides a snapshot 
comparison of the Agencies’ outputs and key thematic areas using two criteria: frequency of 
use and output quality. Table 2 also links each output/area with the agency result to which it 
contributed (see right column). 

Table 2 - Most used and highest quality outputs/themes per agency 

Top ranked outputs / themes 

Eurofound 

Living, working and COVID-19 

Results indicator 

RI1, RI3, RI4 

The three regularly repeated pan-European surveys: European 
working conditions survey, European quality of life survey, 
European company survey 

Cross-cutting, underpins 
research across all the 
results indicators  

Minimum wages 

Cedefop 

Skills anticipation and matching 

Monitoring and analysing VET policy developments 

Promoting access to and attractiveness of VET 

ETF 

Skills demand anticipation 

RI2 

RI1 

RI1 

RI1 

RI2 

Monitoring and diagnostics reports (Torino process reports, rapid 
education diagnostics reports) 

RI5, RI6 

Innovative teaching and learning (e.g. New Learning network) 

RI3 

Vocational excellence provision (e.g. the ETF Network for 
Excellence) 

187 

 
 
 
 
 
 
 
 
 
EU-
OSHA367 

OiRA 

Healthy workplaces campaigns 

Source: Supporting study 

RI3 

RI3 

RI4 

(i) 

EUROFOUND 

Eurofound’s achievement of results indicators 1, 3, 4, 5 and 6 in Table 1 above can be inferred 
from the uptake of Eurofound expertise in key European-level policy documents (Table 2 of 
Annex VIII) and the recognition of the scientific quality of Eurofound’s research (measured by 
way  of  articles  mentioning  Eurofound  in  academic  journals  (Table  3,  Annex  VIII)).  Both 
indicators show improvements in Eurofound’s performance during the evaluation period. While 
the 2016 baseline for the number of key EU documents referencing Eurofound was 24.8% (out 
of the total number of references to Eurofound in registered European-level policy documents), 
it remained fairly stable at around 30% between 2017 and 2020. This increased to a 74% uptake 
of Eurofound’s expertise in key European-level policy documents in 2021 and 82% in 2022 
(163 key documents out of 198 in total). Similarly, the number of articles mentioning Eurofound 
in academic journals increased year on year between 2017 and 2021 (and compared to the 2016 
baseline). Both point to the fact that Eurofound has indeed had an effect through its work. 

Results indicator 1: Key stakeholders are provided with information that supports the 
improvement of working conditions and the promotion of sustainability of work over the 
life course  

Eurofound has provided evidence for a wide range of existing and planned legal instruments. 
This includes input on working time, equal treatment and anti-discrimination. Research on the 
impact of COVID-19 on working conditions has fed into renewed discussions on telework, the 
right  to  disconnect368,  the  design  of  workplaces  and  the  outcome  on  performance  and  well-
being369.  Feedback from the Eurofound stakeholder survey also highlighted the knowledge it 
provided on working and living conditions as one of its main achievements (e.g. achievements 
highlighted by stakeholders in the open questions include: ‘the knowledge-based approach to 
work and labour’, ‘timely and objective knowledge information  about  the world  of work in 
Europe’)370.  Moreover,  by  providing  knowledge  about  policy  interventions  that  have  been 
successful,  Eurofound  has  supported  mutual  learning  and  peer-review  processes  among 
Member States. For example, knowledge shared on the institutional frameworks and policies 
that  support  making  work  more  sustainable  inform  the  European  Semester  process  and  the 

368 Eurofound  (2021),  Right  to  disconnect:  Exploring  company  practices,  Publications  Office  of  the  European 

Union, Luxembourg. 

369 Eurofound and Cedefop (2020), European Company Survey 2019: Workplace practices unlocking employee 
potential, European Company Survey 2019 series, Publications Office of the European Union, Luxembourg. 

370 Evaluation stakeholder survey, conducted in 2023 (N=65). 

188 

 
 
 
 
 
drafting  of  country-specific  recommendations371.  This  direct  impact  on  sustainable  work 
processes  in  Member  States  exemplifies  the  effectiveness  of  information  provided  on  this 
thematic area by Eurofound given its use in policymaking.  

Results indicator 2: Management and labour are provided with information that supports 
social dialogue on industrial relations 

89% of respondents to the Eurofound stakeholder survey (N=65) viewed Eurofound’s work in 
supporting social dialogue as ‘very good’ or ‘rather good’. This shows that Eurofound’s work 
in this field is highly rated but at a lower level than some other areas. These areas include: (i) 
promoting the involvement of key stakeholders in EU policy processes, showing flexibility and 
responsiveness to changing needs, contributing to the debate on the digital transition (all three 
areas rated by 96% of stakeholders as very good or rather good); (ii) raising awareness about 
European policy priorities, introducing new concepts of ideas in public policy (both rated as 
very good or rather good by 92%); and (iii) contributing to the debate on the green transition 
and policy development on ageing society (90%). Eurofound’s work on minimum wages stands 
out with such outputs. As identified in Eurofound Case Study 3, the agency’s work on minimum 
wages  contributed  to  the  2022  Directive  on  adequate  minimum  wages  in  the  EU  (further 
information is included in the impact section)372. 

One  of  the  key  outputs  through  which  Eurofound  has  achieved  this  result  are  the 
representativeness studies it produces on an annual basis. These reports are of key importance 
for  EU  lawmaking  and  social  dialogue  more  generally.  Article  155  of  the  Treaty  on  the 
Functioning  of  the  European  Union  (TFEU)  provides  the  possibility  to  put  social  partner 
agreements into practice in EU law. However, this can only be done if the social partners who 
negotiated  the  agreement  are  considered  as  representative.  This  assessment  is  based  on  the 
representativeness studies. Moreover, under Article 154, the Treaty obliges the Commission to 
consult social partners in certain instances. The representativeness studies are used to determine 
which social partners should be consulted. Finally, representativeness is an eligibility criterion 
for setting up a European Sectoral Social Dialogue Committee or participating in one.  

Moreover, the validation workshop for this study highlighted that the representativeness studies 
are crucial for ensuring proper social dialogue at EU level to facilitate social dialogue at national 
level  (thus  having  an  indirect  national  impact).  Eurofound  has  developed  a  reliable 
methodology for conducting representativeness studies, which ensures transparency and data 
validity. Eurofound’s extensive experience and expertise in this regard is an important factor in 
the credibility of these studies. The representativeness studies are also useful to social partners, 
to whom they provide visibility and ensure that social dialogue is on the agenda of national 
governments.  

Results indicator 3: Key stakeholders are better equipped to support the improvement of 
labour-market functioning and inclusiveness 

371 Eurofound (2020), Programming document 2021–2024: Towards recovery and resilience, p. 12. 
372 Directive (EU) 2022/2041 on adequate minimum wages in the European Union. 

189 

 
 
 
Eurofound’s analysis of structural labour market change and shifts in supply and demand were 
intended to enable stakeholders to go beyond standard statistical data to better understand recent 
trends and developments and identify related opportunities and challenges. Databases such as 
the European Jobs  Monitor, which tracks structural  change in  European labour markets,  the 
European Restructuring Monitor (ERM) and EU PolicyWatch let stakeholders to do just that. 
Reports  such  as  ‘COVID-19:  Implications  for  employment  and  working  life’  provided  an 
overview  of  policy  approaches  adopted  to  mitigate  the  impact  of  the  crisis  on  businesses, 
workers and citizens and examined the development, content and impact of support schemes373. 
The tools above help stakeholders to be more effectively equipped to understand how the labour 
market works in their respective countries. Eurofound also uses these tools to create research 
outputs.  

Results indicator 4: Key stakeholders have increased understanding of ways in which to 
improve the quality of life, public services and society 

The ‘Living, working and COVID-19’ e-survey has been one of the key outputs related to this 
indicator.  Five  rounds  of  this  unique  survey  were  conducted  between  9 April 2020  and 
2 May 2022. The fifth round included collaboration with the ETF, Albania, Georgia, Jordan, 
Kosovo, Lebanon, Moldova, Morocco, North Macedonia, Tunisia and Palestine. The timeliness 
and  relevance  of  this  research  made  this  a  key  output  of  Eurofound  and  was  crucial  to 
understanding  the  impact  of  the  COVID-19  pandemic  in  EU  society  and  further  afield;  for 
example,  Eurofound’s  work  showed  that  women  and  young  people  were  disproportionately 
affected by the pandemic374. Eurofound more than doubled its media impact, as measured by 
KPIs, thanks to the dissemination of this research.  

Results  indicator  5:  Key  stakeholders  are  provided  with  tools  and  information  to  help 
them anticipate and manage the impact of change, including just transitions that promote 
employment, good working conditions, social protection and workers’ rights.  

Respondents to the stakeholder survey rated Eurofound highest of the four Agencies in these 
three categories: contribution to the debates on ageing societies, the green transition and the 
digital transition. Eurofound’s contribution was rated as very good in 38% of cases relating to 
ageing society, 42% concerning the green transition, and 49% concerning the digital transition 
(N=65).  

Results  indicator  6:  Key  stakeholders  are  provided  with  information  to  help  them 
improve social cohesion and promote convergence in the EU on better living and working 
standards 

To  address  this  objective,  Eurofound  has  worked  on  monitoring  convergence,  including  a 
‘convergeEU’ tool in 2018. The tool analyses upward convergence for the full set of indicators 

373  Eurofound  (2021),  COVID-19:  Implications  for  employment  and  working  life,  Publications  Office  of  the 

European Union, Luxembourg. 

374 Case Study 1: Eurofound’s contribution to discussions and policy decisions related to the COVID-19 pandemic 

and Russia’s war of aggression against Ukraine. 

190 

 
 
 
of the Social Scoreboard375 accompanying the European Pillar of Social Rights, as well as for a 
selected set of Eurofound indicators drawn from the European quality of life survey (EQLS) 
and the European working conditions survey (EWCS)376. The 2022 external evaluation of the 
new cross-cutting activities for 2017-2020 highlighted the success of monitoring convergence 
activity, which produced 20 outputs and adapted to policy needs as previously mentioned. This 
was a very new topic in 2017377 and, given that the concept of convergence was only emerging, 
there were some issues with language being too specific and thus only accessible to specialised 
audiences. This made it difficult to align this activity with Eurofound’s other publications and 
the policy debate378. Overall, however, the activity was effective in contributing to timely and 
policy-relevant knowledge.  

(ii) 

Cedefop 

Results  indicator  1:  Policymakers  and  stakeholders  are  provided  with  high-quality 
relevant evidence on VET, skills and qualifications produced by Cedefop in strategic areas 
of operation. 

The number of downloads of Cedefop’s outputs indicates this result was achieved. Downloads 
generally increased throughout the evaluation period (313 850 in 2017 compared to 444 000 in 
2022, with a peak of 465 000 in 2020)379. A full table comparing downloads for each agency 
between 2016 and 2022 can be found in Table . This trend can be explained partially by the 
continuing popularity of publications published in previous years (i.e. older publications are 
still  being  referenced  and  downloaded  as  reflected  in  Cedefop’s  performance  measurement 
system data) and the relevance of the evidence Cedefop produces.  

According  to  the  user  satisfaction  survey  in  2019  (N=453)  and  2022  (N=486),  99%  of 
respondents found information provided by Cedefop to be reliable, and 95% of respondents in 
2019 (N=463) and 96% in 2022 (N=507) felt Cedefop’s activities met their needs380. The level 
of user satisfaction has significantly increased compared to results from the stakeholder survey 
in the previous evaluation (2011-2016), where 70-75% (N=213) of respondents felt their needs 
were met by the agency’s outputs381. 

Of  the  four  strategic  areas  of  operation  (shaping  VET,  informing  VET,  valuing  VET  and 
communication  and  dissemination),  shaping  VET  is  the  area  with  both  the  highest  resource 
allocation  (32%  to  33%  of  the  total  budget  throughout  the  evaluation  period)  and  most 

375 See: https://ec.europa.eu/eurostat/web/european-pillar-of-social-rights/indicators/social-scoreboard-indicators. 
376 See: https://www.eurofound.europa.eu/data/convergence-hub/convergence-methodology.  
377 Ibid, executive summary, p. i. 
378 ICF (2022), Evaluation of two new crosscutting activities delivered as part of the 2017-2020 programme. The 
digital age: opportunities and challenges for work and Employment and Monitoring convergence in the 
European Union, executive summary, p. ii. 

379 Cedefop Annual Reports 2017-2022. 
380 PPMI (2020) Cedefop user satisfaction survey 2019; PPMI (2022) Cedefop user satisfaction survey 2022. 
381 PPMI, Ecorys (2018) Evaluation of the EU Agencies under the remit of DG Employment: EUROFOUND, 

CEDEFOP, ETF and EU-OSHA. 

191 

 
 
 
 
downloads (with a peak of 94 000 downloads in 2020)382. High percentages of both staff (89%, 
N=45)383  and  stakeholders  (79%,  N=174)  rated  Cedefop’s  performance  in  introducing  new 
concepts or ideas in public policy as either ‘very good’ or ‘rather good’. 48% of stakeholders 
responded with ‘very good’ – the highest percentage across all the agency’s performance areas 
and the highest of the four Agencies in this category. This is an important indicator of Cedefop’s 
ability  to  provide  its  stakeholders  with  highly  relevant  evidence  based  on  empirical  data, 
research evidence and analytical knowledge. Moreover, Figure 1 shows large percentages of 
stakeholders rating Cedefop’s performance highly in providing evidence and policy analysis.  

Cedefop’s work on monitoring and analysing VET policy developments, skills anticipation and 
matching and promoting access to and attractiveness of VET were among the most referenced 
and downloaded areas throughout the evaluation  period. Cedefop’s work on monitoring and 
analysing VET policy developments contributed directly to shaping the common priorities in 
the Council Recommendation on VET and the Osnabrück Declaration. Cedefop’s  output on 
skills  anticipation  and  matching  was  also  among  the  most  widely  used  compared  to  other 
thematic areas. The skills forecast tool was referred to at the Commission-OECD roundtable 
and in the EU migration network. The tool's added value was explored beyond the thematic 
scope of Cedefop and DG EMPL when it was used to test different scenarios in the work of DG 
GROW, DG ENER and DG ENV. Compared with the two other strategic areas of Cedefop’s 
work,  the  results  of  valuing  VET  (and  skills)  seem  to  have  been  used  less  by  stakeholders 
overall.  However,  certain  themes  in  this  area  (promoting  access  and  attractiveness  of  VET) 
were also among the most cited (especially by international organisations) and downloaded (see 
above  and  state  of  play).  With  an  increasing  share  of  the  budget  allocated  to  this  area,  the 
references and downloads were on an upward trend. 

Figure 1: Perceptions on the achievement of Cedefop’s objectives: Contributing to policy 
decisions and providing analysis with evidence, % of respondents 

382 Ibid. 
383 Cedefop staff survey, N=45. 

192 

 
 
 
d
n
a

e
c
n
e
d
v
e

i

i

i

g
n
d
v
o
r
P

b
i
r
t
n
o
C

o
t
g
n
i
t
u

i

s
s
u
c
s
d

i

d
n
a

s
n
o

y
c

i
l

o
p

i

i

o
s
c
e
d

i

s
s
y
a
n
a

l

y
c

i
l

o
p

s
n

Staff survey

Stakeholder survey

OPC

Staff survey

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

To a large extent

To a moderate extent

To a small extent

Not at all

Do not know / cannot answer

Source: Cedefop staff survey (N=45), stakeholder survey (N=174), and public consultation (N=44), 
question: In your view, to what extent, if at all, was Cedefop successful in achieving the following 
objectives in 2017-2022?.  The results of the assessment of contributions to discussions and policy 
decisions are only available from the staff survey. 

Results indicator 2: EU-level  and national  actors benefit from exchanges  supported by 
Cedefop in terms of policy learning. 

learning  exchanges  supported  by  Cedefop.  Despite 

Looking  at  the  second  results  indicator,  EU  and  national  actors  have  indeed  benefited  from 
the  overall  number  of 
policy 
meetings/events organised by Cedefop falling throughout the evaluation period, the number of 
participants in these events has increased year on year since 2017 (1 159 in 2017 compared to 
1 961 in 2021)384. This has led to a rating of 98% for ‘quality and expected impact of events 
organised  by  Cedefop’  in  2021,  the  highest  of  the  evaluation  period.  The  positive  trend  in 
stakeholder participation from 2017 to 2021 is rather a recovery from a gradual decline in the 
previous evaluation period (from above 2 500 participants between 2011 and 2013 to 1 795 by 
the  end  of  evaluation  period  in  2016  and  consequently  1 159  in  2017)385.  According  to  this 
evaluation’s stakeholder survey, 68% (N=174) of respondents consider the quality of Cedefop’s 
events to be high or rather high. This shows that Cedefop’s written contributions are valued 
more  than  in-person  activities  (81%  and  87%  valued  national  and  EU-level  reports).  A 
significant proportion of stakeholders (28%) indicated they did not have enough information to 
evaluate the events.  

(iii)  ETF 

Results  indicator  1:  Partner  countries  receive  relevant  and  timely  support  to  improve 
governance of their human capital development systems 

384 Cedefop Consolidated Annual Activity Reports 2016-2021. 
385 Cedefop Consolidated Annual Activity Reports 2012-2017. 

193 

 
 
 
 
 
 
 
 
 
 
The ETF has partially achieved this result. The agency has scored highly in the Policy Delphi 
consultation  on  the  activities  aimed  at  strengthening  the  capacity  of  partner  country 
stakeholders and policymakers: 80% of respondents reported that the ETF contributed to this 
objective to a large or some extent (the highest rated activity in the consultation). This finding 
was also confirmed by some stakeholders consulted during the evaluation who valued the ETF’s 
expertise  in  governance.  An  example  of  the  achievements  in  this  field  is  ETF’s  work  on 
Moldova,  which  resulted  in  sectoral  committees  being  established  to  address  labour  market 
needs386. 

However, as a non-governmental organisation with limited resources, the ETF can only support 
policymaking in partner countries to a limited extent. Assessing a result in this area is hindered 
by the lack of indicators to  measure progress  against objectives. The ETF  set  a quantitative 
target  of  introducing  coordination  mechanisms,  legislation  and  multilevel  governance 
methodologies  in  85%  of  targeted  countries.  However,  progress  on  the  indicator  was  not 
measured quantitatively. This makes it difficult to assess to what extent this target was achieved. 
Qualitative  information  indicates  that  in  2017  progress  in  system-level  coordination  was 
achieved in Egypt, Kosovo, Serbia and Ukraine. However, information on the targeted is not 
available. Between 2017 and 2020, when the data for this indicator is available, support was 
also provided to Georgia, Kazakhstan and Tunisia, in addition to the countries mentioned above. 
However, it is not clear to what extent it contributed to improving the governance of human 
capital development systems in those countries.  

Results  indicator  2:  Partner  countries  receive  relevant  and  timely  support  on  active 
labour market policies and skills anticipation 

The  focus  of  activities  in  this  area  of  work  related  to  the  development  of  ALMPs,  skills 
anticipation  and  transition  to  work.  The  evaluation  found  that  the  ETF’s  work  on  skills 
anticipation  was  particularly  effective.  The  ETF  has  designed  new  and  upgraded  existing 
methodological  tools  to  analyse  skills  needs,  leading  to  better  intelligence  on  the  topic. 
Previously, tools used in the EU were adapted for ETF partner countries. However, as a result 
of the ETF’s work, regional knowledge is now captured through expert networks, such as the 
Skills  Lab  and  conducting  in-depth  country-specific  reports.  Some  respondents  to  the 
stakeholder  survey  mentioned  the  ETF’s  work  on  skills  anticipation  as  one  of  its  main 
achievements between 2017 and 2022, particularly in countries and situations where data are 
scarce (see case study on Skills Labs for further details). The other notable examples of outputs 
that contributed to achieving this objective included the ETF’s work on the Youth Guarantee, 
with SEET countries committing to implement the EU Youth Guarantee and Future of Work 
methodology.  This  generated  interest  and  uptake  in  partner  countries  (including  Tunisia, 
Albania and Armenia), and the initiatives were adapted to Latin American countries through 
cooperation with UNDP387. 

386 ETF website (2024), Moldova.   
387 ETF (2023), Consolidated Annual Activity Report: 2022. 

194 

 
 
 
The ETF did not report quantitatively on: (i) the extent to which 2017-2020 targets to develop 
and implement mechanisms to facilitate transitions to work in 50% of partner countries were 
met; or (ii) whether labour market and skills intelligence informed VET and skills development 
policies in 50% of partner countries388. However, the ETF’s targets often aimed to influence 
complex policy reforms in the partner countries. Such policy reforms are affected by a range of 
factors outside the ETF’s control and take a long time to implement. The willingness of  the 
partner country to follow the ETF’s advice and accept support also plays a role in this. It is a 
continuing challenge for the ETF, which was already identified in the previous evaluation. 

Result indicator 3: Partner countries receive relevant and timely support on improving 
lifelong learning provision 

Overall, the ETF achieved this specific objective, with evidence from quantitative indicators, 
qualitative information and stakeholder feedback pointing to this conclusion. One of the main 
contributions of the ETF in this area was the creation of networks to promote excellence and 
increase collaboration. This included the Creating New Learning network and the ETF Network 
for Excellence, which by 2021 had 1 300 members and 258 members respectively389. The ETF’s 
networks  were  highly  valued  by  most  of  their  users  and  facilitated  more  collaboration  both 
within  and  across  partner  countries.  In  the  Policy  Delphi  conducted  in  2018  during  the 
development  of  the  2027  ETF  strategy,  the  agency’s  contribution  to  modernising  partner 
countries’ VET systems was the most positively assessed activity. 41% of respondents indicated 
that the agency helped partner countries to a large extent390. 

Results indicator 4: Partner countries receive relevant and timely support to modernise 
national qualifications frameworks and systems 

The result on qualifications systems and frameworks was achieved by consistently monitoring 
and  supporting  partner  countries  in  developing  their  national  qualifications  frameworks 
(NQFs). Based on the work in this field at the national level, the ETF, together with Cedefop 
and  UNESCO,  produced  the  Global  Inventory  of  National  and  Regional  Qualifications 
Frameworks. The ETF built on its expertise in supporting partner countries and played a major 
role in supporting the development of the African Continental Qualifications Framework.  

The 2020 evaluation of ETF action to reform qualifications systems in partner countries found 
that  the  agency’s  work  had  an  impact  on  partner  countries’  mechanisms,  frameworks  and 
processes for implementing reforms of qualifications systems. The evaluation emphasised that 
combined  ETF  action  was  a  major  factor  contributing  to  the  success  of  interventions  on 
qualifications.  For  example,  ETF  work  on  strengthening  stakeholder  capacity,  providing 
knowledge, insights and advice based on international comparative analysis and tools helped 
achieve  results.  The  evaluation  highlighted  that  ETF  expert  support  informed  the  NQF 
implementation  action  plan,  national  qualifications  agencies’  work  and  the  qualifications 
register  in  Ukraine.  There  were  also  positive  contributions  in  this  field  identified  in  North 

388 ETF, Consolidated Annual Activity Reports, 2017-2020. 
389 ETF (2023), Consolidated Annual Activity Report: 2022. 
390 PPMI (2018), Evaluation of the ETF Functions: Policy Delphi Results 1st round: June/July 2018, p. 14. 

195 

 
 
 
Macedonia and Kosovo. To further increase effectiveness in this field, there is a need to better 
integrate end-user perspectives in the ETF’s work and to develop links with other related areas 
of work, e.g. social dialogue, work-based learning and lifelong learning391. The ETF’s work in 
these areas was assessed favourably during the previous evaluation period, indicating that the 
agency continued to build on its strengths. 

Results  indicator  5:  Partner  countries  receive  relevant  and  timely  information  on  the 
performance of lifelong learning policies and systems 

The ETF achieved this result based on the stakeholder feedback and data analysed during the 
evaluation. For example, in 2019, 78% of partner countries were using the Torino Process for 
policy dialogue. This  exceeds  the 50% target  substantially.  Partner  countries representatives 
considered  the  Torino  Process  to  be  one  of  the  most  valuable  activities  provided  by  the 
agency392. For example, when confirming their participation in the sixth round of the Torino 
Process, the Ministry of Education and Research in Moldova expressed their appreciation for 
the process as ‘one of the most efficient tools for the policy dialogue, monitoring and evaluation 
of  VET  reforms’.  The  representatives  praised  ETF’s  role  in  facilitating  policy  dialogue, 
informing  strategies  on  education  and  training,  and  evaluating  ongoing  reforms  and 
programmes. 

Results indicator 6: Partner countries receive relevant and timely support to enable them 
learning  and  enterprise  skills 
to  offer  more  opportunities  for  entrepreneurial 
development 

The  increasing  importance  of  ETF’s  support  related  to  enterprise  skills  development  was 
highlighted  by  the  European  Parliament’s  discharge  report.  The  report  highlighted  that  the 
initiative on Skills for Enterprise Development helped businesses respond to challenges related 
to COVID-19 and the digital and green transitions393.  This is a significant development as ETF’s 
work in this field was assessed less favourably in the previous evaluation. Despite progress on 
the ETF’s work on enterprise skills, targets related to entrepreneurial learning were only met in 
2017. This shows that performance in this area is less favourably perceived than in the previous 
evaluation period. 

Table 6 Quantitative data for results indicator 6 

Indicator 

2017 

2018 

2019 

2020 

2021 

2022 

Improvement  in  human 
capital  dimension  of 
Small  Business  Act 
performance  in  partner 
countries 

3 
countries 

7 
countries 

6 
countries 

Target:  3 
countries 

Target: 10 
countries 

Target: 10 
countries 

391 Ockham IPS (2020), Evaluation of the ETF actions on the reform of qualifications systems in Partner 

Countries, pp. 40-45. 

392 Interviews conducted for the purposes of this evaluation, as well as the stakeholder survey. 
393 European Parliament (2022), Discharge 2020: European Training Foundation. 

196 

 
 
 
 
 
 
 
 
 
Source: ETF, consolidated annual activity reports 2017-2022 

Results indicator 7: EU external relations policymakers receive relevant advice 

The two ETF targets for  providing support for EU external relations policy were met in the 
years  during  which  the  indicators  were  monitored  (2017-2020).  The  ETF  was  consistently 
asked to provide support in more than half the agency’s partner countries, ranging between 58-
62% of countries (against a target of 50%). Furthermore, 100% of the support provided by the 
agency was assessed positively each year. The interviews with the Commission show that rapid 
education  diagnostics  (RED)  reports  introduced  during  the  current  evaluation  period  for 
Lebanon and Kosovo improved dialogue with those countries and informed decisions on EU 
external assistance. 

(iv)  EU-OSHA 

Results indicator 1: Policymakers and researchers are provided with good quality data 
enabling them to take timely and effective action on new and emerging risks 

Overall,  the  evidence394  suggests  that  outputs  were  successful  in  raising  awareness  among 
policymakers on new and emerging  risks, thus helping them  make decisions. This holds for 
both  foresight reports on  the circular economy  and digitalisation. In particular,  EU-OSHA’s 
foresight reports on digitalisation brought added value even in countries that were familiar with 
the issue, including by exploring aspects that were under-researched or otherwise not present 
in  national  research. Management  board members and  focal  points  also  found that foresight 
exercises  were  very  relevant  for  EU  and  national  business  associations  and  employers’ 
organisations. This is a considerable change compared to the baseline when the foresight studies 
were used less than other outputs. 

Results indicator 2: Better understanding of current OSH risks and their health effects 
thanks to information provided by EU-OSHA. 

The most significant way in which EU-OSHA has achieved this result has been through the 
European  Survey  of  Enterprises  on  New  and  Emerging  Risks  (ESENER).  EU-OSHA’s  bi-
annual stakeholder surveys between 2016 and 2022 showed a growing majority of respondents 
were satisfied with ESENER (85% in 2016, 90% in 2018, 94% in 2020 and 95% in 2022). In 
terms  of  more  cross-cutting  recent  policy  priorities,  there  are  insights  from  EU-OSHA’s 
contribution to the debates on ageing societies, the green transition and the digital transition. 
While staff rated the agency’s performance as ‘very good’ in 65% of cases related to ageing 
society,  56%  related  to  the  green  transition,  and  81%  related  to  the  digital  transition, 
respondents to the stakeholder survey were of the same opinion in only 20%, 17% and 39% of 
cases respectively. This large gap indicates that EU-OSHA stakeholder needs are perhaps not 
met by agency activities in these areas. 

Results  indicator  3:  SMEs  know  and  use  the  tools  provided  by  EU-OSHA  to  manage 
health and safety 

394 Evidence gathered from interviews and annual activity reports. 

197 

 
 
 
For  this  result  (that  there  are  more  tools  of  good  quality  available  to  SMEs),  during  the 
evaluation period, there were more than 300 OiRA tools available in 17 EU languages, roughly 
200 more than in 2016395. These tools covered over 30 sectors, from manufacturing, hairdressing 
and beauty salons to education and live performance. The evidence suggests that these tools 
have  helped  companies,  in  particular  SMEs,  to  take  their  first  steps  in  risk  assessment.  For 
example, Cyprus has seen a high number of risk assessments with OiRA tools, with more than 
95% of its companies being SMEs. More broadly, OiRA tools have been used extensively in 
Slovenia (3 681 users), Cyprus (5 605 users), Lithuania (6 176 users), Latvia (7 093 users) and 
France. France, with more than 42 000 users, made up roughly half of all OiRA users396.  During 
the  COVID-19  pandemic,  an  OiRA  tool  was  also  reported  to  have  helped  employers  and 
workers prevent the spread of the virus at the workplace.  

EU-OSHA stakeholders’ assessments confirm the growing satisfaction and popularity of OiRA 
tools. This supports the conclusion that it has been used by SMEs to improve or manage health 
and safety.  

Figure 3:  OiRA key performance indicators and bi-annual stakeholder surveys397 

Familiarity with OiRA over time

Satisfaction with OiRA over time

44%

46%

48%

33%

92%

93%

83%

2016 (N = 1,549)

2018 (N = 1,728)

2020 (N = 1,751)

2022 (N = 907)

2018
(N = 1,728)

2020
(N = 1,751)

2022
(N = 907)

395 EU-OSHA (2016). EU-OSHA Annual Activity Report 2016.  
396 EU-OSHA (2023). Tender Specifications. OSHA/2023/OP/0004. Qualitative Research Insights into the 

process of risk assessment with OiRA at European Workplaces. 

397 Satisfaction with OiRA over time excludes ‘do not know’ and ‘cannot answer’ responses. 

198 

 
 
 
 
 
OiRA KPIs

93%

82%

67%

90%

92%

88%

76%

71%

81%

80%

70%

100%

87%

80%

60%

40%

20%

0%

2019

2020

2021

2022

Relevance

Usefulness

EU added value

Sources: bi-annual stakeholder survey 2016-2022, key performance indicators 2019-2022 (AAR) 

Results  indicator  4:  Multiple  beneficiaries  and  intermediaries  are  aware  of  workplace 
risks and how to prevent them 

For this result, interviews with  EU-OSHA’s  focal  points  revealed that national  stakeholders 
expressed  increasing  interest  in  participating  in  the  healthy  workplaces  campaigns  (HWCs), 
while some of them398 requested access to some of the campaigns’ material. In addition, based 
on  EU-OSHA’s  data,  the  HWCs  grew  in  terms  of  partnerships  and  event  activities.  For 
example, in 2022, EU-OSHA had more than 80 official campaign partners, including 29 media 
partners.  It  also  held  1 322  promotional  activities.  Evidence  from  EU-OSHA’s  annual 
stakeholder survey corroborates the usefulness, EU-added value and impact of the campaigns, 
as shown in Figure 4.  

Figure 4:  HWCs: EU-OSHA stakeholders’ assessment on key performance indicators 2019-2022  

Usefulness, EU-added value and Impact of HWC 2019-2022

93%

96%

84%

82%

95%

92%

97%

98%

90%

84%

81%

67%

120%

100%

80%

60%

40%

20%

0%

Dangerous substances (2019)

MSDs (2021)

MSDs (2022)

Usefulness

Usefulness of FAST events

EU added value

Impact

Source: EU-OSHA AARs, Key performance indicators, 2019-2022 

398 Examples include Cyprus and Finland.  

199 

 
 
 
 
 
Results  indicator  5:  The  OSH  community  benefits  from  a  larger  body  of  quality 
knowledge 

For this result, EU-OSHA’s A5 area of activities, ‘Networking knowledge’, aims to support the 
OSH community with a larger body of quality knowledge. There were 107 outputs under this 
activity between 2017 and 2022. These outputs were related to an OSH wiki, an exchange of 
knowledge on e-tools, expert support to the Commission, and an online database of legislation, 
non-binding instruments and social dialogue initiatives. While there is no direct measurement 
of this work, a stakeholder survey conducted by EU-OSHA suggests that this activity exceeded 
its set targets in terms of usefulness (94%, target 80%)399, relevance (100%, target 80%), EU-
added  value  (100%,  target  70%)  and  impact  (76%,  target  70%).  Furthermore,  the  staff  and 
stakeholder  survey  results  for  EU-OSHA  also  show  that  the  agency  was  successful  in 
introducing new concepts and ideas to public policy (confirmed by 90% of staff and 91% of 
stakeholders). In the public consultation, the category ‘Quality of Information’ emerged as the 
most  valued  characteristic  of  EU-OSHA's  work  from  2017  to  2022,  accounting  for  63%  of 
responses  (N=41)400.  These  findings  lead  to  the  conclusion  that  EU-OSHA's  networking 
knowledge outputs achieved their expected results. 

Results  indicator  6:  EU-OSHA  stakeholders  and  networks  take  part  in  EU-OSHA 
activities and their needs are met 

EU-OSHA achieved this result, although to a lesser degree than other results. Engagement in 
planning, monitoring and implementing the agency’s work programme decreased throughout 
the  second  half  of  the  evaluation  period401,  falling  from  95%  in  2019  to  88%  in  2022. 
Nevertheless,  engagement  remained  high402.  There  is  also  evidence403  to  suggest  that  some 
members of the management board struggled to quickly become involved in board meetings 
and make meaningful contributions to them due to engagement and language barriers. 

399 This metric applies to OSHwiki.  
400 Public consultation, N=41. 
401 There are no indicators on this before 2019. 
402 EU-OSHA Annual Activity Reports 2019-2022, Key Performance Indicators. Engagement indicator. Source: 

Board and FOP annual survey.  

403 Interviews conducted for the purpose of this study. 

200