COMMISSION STAFF WORKING DOCUMENT EVALUATION of the 2014-2018 ESF support to employment and labour mobility, social inclusion and education and training
Date: 2021-01-27
Institutions: Directorate-General for Employment, Social Affairs and Inclusion, European Commission
Subjects: Cohesion Fund, EU employment policy, European Regional Development Fund, European Social Fund, distribution of EU funding, education, employment aid, operational programme, social integration, vocational training

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 27.1.2021  SWD(2021) 10 final  COMMISSION STAFF WORKING DOCUMENT  EVALUATION  of the 2014-2018 ESF support to employment and labour mobility, social inclusion and education and training {SWD(2021) 11 final}  
 
 
Contents 

1 

2 

INTRODUCTION ................................................................................................................................ 6 

BACKGROUND TO THE INTERVENTION ..................................................................................... 6 

2.1 

Policy context .................................................................................................................. 6 

2.1.1 

ESF objectives and functioning ................................................................................ 8 

2.1.2 

ESF support to employment and labour mobility (thematic objective 8) ............... 9 

2.1.3 

ESF support to social inclusion (thematic objective 9) .......................................... 10 

2.1.4 

ESF support to education and training (thematic objective 10) ........................... 10 

2.2 

Baseline and points of comparison ............................................................................... 10 

2.2.1 

Employment .......................................................................................................... 11 

2.2.2 

Labour mobility ..................................................................................................... 13 

2.2.3 

Social inclusion ...................................................................................................... 14 

2.2.4 

Education and training .......................................................................................... 16 

3 

IMPLEMENTATION / STATE OF PLAY ........................................................................................ 17 

3.1 

3.2 

3.3 

3.4 

3.5 

Programming and budget ............................................................................................. 17 

Employment and labour mobility .................................................................................. 18 

Social inclusion .............................................................................................................. 19 

Education and training .................................................................................................. 20 

Typology of Operations funded ..................................................................................... 22 

3.5.1 

ESF support to employment and labour mobility ................................................. 22 

3.5.2 

Social inclusion ...................................................................................................... 23 

3.5.3 

Education and training .......................................................................................... 24 

3.6 

Financial and operational implementation ................................................................... 26 

4  METHOD............................................................................................................................................ 27 

4.1 

4.2 

Short description of methodology ................................................................................ 27 

Limitations and robustness of findings ......................................................................... 27 

5 

ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS .......................................... 28 

5.1 

Effectiveness .................................................................................................................. 28 

5.1.1 

Financial and output achievements ...................................................................... 28 

5.1.2 

Findings on results and contribution to overarching objectives ........................... 32 

5.1.3 

Effects on participants ........................................................................................... 36 

5.1.4 

Factors hindering or promoting effectiveness ...................................................... 38 

5.2 

Efficiency ....................................................................................................................... 40 

1 

 
 
5.2.1 

Assessment of efficiency ....................................................................................... 40 

5.2.2 

Factors hindering or promoting efficiency ............................................................ 41 

5.2.3 

Visibility of ESF interventions ................................................................................ 42 

5.3 

Relevance ...................................................................................................................... 43 

5.3.1 

Alignment with needs ........................................................................................... 43 

5.3.2 

Observed factors that improve relevance ............................................................. 47 

5.3.3 

Flexibility in a changing context ............................................................................ 49 

5.4 

Coherence ..................................................................................................................... 50 

5.4.1 

ESF Internal coherence .......................................................................................... 50 

5.4.2 

ESF external coherence ......................................................................................... 53 

5.5 

EU added value .............................................................................................................. 58 

5.5.1 

Volume effects ...................................................................................................... 58 

5.5.2 

Scope effects ......................................................................................................... 60 

5.5.3 

Role effects ............................................................................................................ 62 

5.5.4 

Process effects including sustainability ................................................................. 63 

6 

CONCLUSIONS ................................................................................................................................. 65 

6.1 

Lessons learned ............................................................................................................. 71 

ANNEX 1: PROCEDURAL INFORMATION ............................................................................................ 73 

ANNEX 2: STAKEHOLDER CONSULTATION ....................................................................................... 76 

ANNEX 3: METHODS AND ANALYTICAL MODELS ........................................................................ 105 

ANNEX 4: ESF INTERVENTION LOGIC ............................................................................................... 114 

List of figures 

Figure 1. Allocations to thematic objectives 8, 9, and 10 (ESI Funds plus national co-financing) - 
EUR millions ..................................................................................................................................... 7 
Figure 2. Europe 2020 - Distance to targets for headline indicators ............................................ 11 
Figure 3. Employment rate as function of age in 2014 and in 2018 ............................................. 12 
Figure 4. Proportion (in %) of the EU population at risk of poverty or social exclusion ............... 15 
Figure 5. Total ESF and YEI budget per thematic objective - EUR millions ................................... 18 
Figure 6. TO8 - Indicative allocations per investment priority - in EUR millions ........................... 18 
Figure 7. TO9 - Indicative allocation by investment priority - in EUR millions .............................. 20 
Figure 8. TO10 - Indicative allocations by investment priority - in EUR millions .......................... 21 
Figure 9. Distribution of thematic objective 10 eligible costs by area of operation ..................... 25 
Figure 10. Number of participations by selected characteristic and thematic objective ............. 31 
Figure 11. How would you qualify the following administrative arrangements for the 
implementation of operations? .................................................................................................... 39 

2 

 
 
 
 
LIST OF TABLES 

Table 1.. Types of operations per investment priority – TO8 ....................................................... 23 
Table 2. Types of operations per investment priority - TO9 ......................................................... 24 
Table 3. Project selection rates ..................................................................................................... 29 
Table 4. Absorption rates .............................................................................................................. 30 
Table 5. Participant profiles .......................................................................................................... 30 
Table 6. Achievement rates for specific output indicators ........................................................... 32 
Table 7. Reported results (common result indicators only; upon leaving and after six months) . 33 
Table 8. Cost per participation ...................................................................................................... 41 

LIST OF BOXES 

Box 1. Identifying the target group needs for social inclusion operations ................................... 44 
Box 2. Ensuring the relevance of early leavers from education and training operations through 
risk assessment ............................................................................................................................. 45 
Box 3. Integrated approaches for ensuring relevance of support to employment ...................... 48 
Box 4. Examples of flexibility ......................................................................................................... 49 
Box 5. Preliminary reflections on the ESF’s possible role in addressing COVID 19 crisis .............. 50 
Box 6. Types of ESF added value ................................................................................................... 58 
Box 7. Significant volume effects of ESF support to education and training ................................ 60 
Box 8. Social inclusion scope effects: broadening actions to more groups .................................. 61 
Box 9. Examples of TO8 role effects .............................................................................................. 62 

3 

 
 
 
Glossary 

Term or acronym 

Meaning or definition 

AIR 

Beneficiary 

CPR 

CSR 

DG EMPL 

DG REGIO 

EAFRD 

ELET 

EMFF 

ERDF 

ESF 

FEAD 

GDP 

ISCED 

JRC 

MS 

IP 

NEET 

NGO 

OP 

Operation 

Participant 

Annual Implementation Report 

Public or private body responsible for initiating or both initiating 
and implementing operations 

Common Provisions Regulation  

Country-specific recommendations 

Directorate-General Employment, Social Affairs & Inclusion 

Directorate-General Regional and Urban Policy 

European Agricultural Fund for Rural Development 

Early leavers from education and training 

European Maritime and Fisheries Fund 

European Regional Development Fund 

European Social Fund  

Fund for European Aid to the Most Deprived 

Gross domestic product 

International Standard Classification of Education 

Joint Research Centre 

Member State 

Investment priority 

Person not in employment, education or training 

Non-Governmental Organisation 

Person not in employment, education or training 

Operation means a project, contract, action or group of projects 
selected by the managing authorities of the programmes 
concerned, or under their responsibility, that contributes to the 
objectives of a priority or priorities 

A person benefiting directly from an ESF intervention who can be 
identified and asked for their characteristics, and for whom 
specific expenditure is earmarked 

4 

 
 
 
Term or acronym 

Meaning or definition 

System for Fund Management in the European Union 

Staff working document 

Thematic objective 

Vocational education and training 

Youth Employment Initiative 

SFC 

SWD 

TO 

VET 

YEI 

5 

 
1 

INTRODUCTION 

This  staff  working  document  (SWD)  presents  results  of  the  evaluation  of  the  European  Social 
Fund  (ESF)  support  to  policies  in  the  areas  of  employment  and  labour  mobility  (thematic 
objective  8),  social  inclusion  (thematic  objective  9),  and  education  and  training  (thematic 
objective 10).  

The conclusions of the evaluation are relevant for the implementation of the final stages of the 
2014-2020  ESF  investments  and  the  preparation  of  the  2021-2027  programming  period  of 
European  Social  Fund  Plus  (ESF+)  by  providing  lessons,  notably  as  regards  cost-effectiveness, 
outreach  and  target  groups  most  in  need.  The  evaluation  can  support  the  negotiation  of  the 
ESF+ programmes and the design of operations by the Member States in the respective areas. In 
addition, it paves the way for the ex-post evaluation of the ESF. 

The evaluation assesses the effectiveness, efficiency, relevance, coherence and the EU added-
value including the sustainability of activities funded by the ESF in the period 2014-2018. It also 
addresses the complementarity and coherence with other relevant initiatives during the period 
concerned. It covers all Member States during that period (i.e. including the United Kingdom). It 
builds  on  the  results  of  three  separate  studies,  one  for  each  thematic  objective.  The  ESF  and 
Youth Employment Initiative (YEI) support to youth employment, (part of thematic objective 8), 
was the subject of another thematic evaluation1.  ESF support to Institutional capacity building 
(thematic objective 11) has been covered by a study2. 

The onset of the COVID-19 pandemic happened when the field work for the supporting studies 
was  completed  or  about  to  be  completed.  Also  the time  frame  covered, 2014-2018,  excludes 
the  pandemic  and  the  subsequent  crisis.  Nevertheless,  a  number  of  aspects  of  the  lessons 
learned take on a particular relevance (see Box 5). 

2  BACKGROUND TO THE INTERVENTION 

2.1  Policy context 

The ESF is one of the five European Structural and Investment (ESI) Funds. These are EU funding 
instruments providing structural and investment financing across the EU3.  Their total budget in 
2014-2020,  to  be  invested  in  11  areas  called  ‘thematic  objectives’  (TO)  as  defined  in  the 

1  The 

staff 

working 

document 

is 

available 

at: 

https://ec.europa.eu/social/main.jsp?langId=en&catId=89&newsId=9793&furtherNews=yes 

2  The  report  is  available  at:    https://op.europa.eu/en/publication-detail/-/publication/c7c56421-9fc4-

11ea-9d2d-01aa75ed71a1  

3  The other funds are the European Regional Development Fund (ERDF), the Cohesion Fund (CF), the 
European Agricultural Rural Development Fund (EARDF) and the European Maritime and Fisheries 
Fund (EMFF). 

6 

 
 
                                                           
Common  Provisions  Regulation  (CPR)4,  is  EUR  461  billion.  The  funds  allocated  for  thematic 
objectives 8, 9 and 10 amount to EUR 173 billion (Figure 1).  

Figure 1. Allocations to thematic objectives 8, 9, and 10 (ESI Funds plus national co-financing) - 
EUR millions 

Source: https://cohesiondata.ec.europa.eu          

While  the  five  funds  address  different  stakeholders  and  sectors,  they  share  two  important 
features: 

• 

• 

They  support  growth  and  job  creation,  are  aligned  with  the  Europe  2020  strategy 
specific 
implementation  of 
contribute 
and 
recommendations (CSR) 5; 
They are delivered under shared management6.  

relevant 

country 

the 

to 

The  Europe  2020  strategy7  emphasises  smart,  sustainable  and  inclusive  growth  as  a  way  to 
overcome  the  structural  weaknesses  in  Europe’s  economy,  improve  its  competitiveness  and 
productivity and underpin  a sustainable social market  economy. The ESF should contribute to 

4  Regulation (EU) 1303/2013 of the European Parliament and Council laying down common provisions 
on  the  ERDF,  ESF,  the  Cohesion  Fund,  the  EAFRD  and  the  EMFF  or  European  Structural  and 
https://eur-lex.europa.eu/legal-
Funds 
Investment 
content/EN/TXT/?uri=celex%3A32013R1303  

Funds). 

(ESIF 

5  More  information  on  the  European  Semester  is  available  at:  https://ec.europa.eu/info/business-

economy-euro/economic-and-fiscal-policy-coordination/eu-economic-governance-monitoring-
prevention-correction/european-semester/framework/eus-economic-governance-explained_en  

6 

7 

For an explanation of shared management, see the section ESF objectives and functioning. 

https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Glossary:EU_2020_Strategy  

7 

 
                                                           
achieving  the  headline  targets  of  the  Europe  2020  strategy  on  employment,  education  and 
poverty reduction8.   

The  Communication  ‘Investing  in  jobs  and  growth  –  maximising  the  contribution  of  European 
Structural and Investment Funds’9 provides an overview of what the ESI Funds are expected to 
achieve in 2014-2020. 

In November 2017, the Commission, the European Parliament and the Council proclaimed the 
European  Pillar  of  Social Rights10  which  is  about  delivering  new  and more  effective  rights  for 
citizens. It builds upon 20 key principles, structured around three chapters: 

• 
• 
• 

Equal opportunities and access to the labour market; 
Fair working conditions; 
Social protection and inclusion. 

The  Pillar  relies  on  previous  policy  guidance  available  at  EU  level,  which  oriented  the  use  of 
resources provided by ESI Funds (ESIF).  

2.1.1  ESF objectives and functioning 
The  ESF  contributes  to  four  of  the  11  thematic  objectives  defined  by  the  Common  Provision 
Regulation11:   

• 

• 

• 

• 

Thematic  objective  8:  Promoting  sustainable  and  quality  employment  and 
supporting labour mobility;  
Thematic  objective  9:  Promoting  social  inclusion,  combating  poverty  and  any 
discrimination; 
Thematic  objective  10:  Investing  in  education  and  training  and  vocational  training 
for skills and lifelong learning; 
Thematic  objective  11:  Enhancing  institutional  capacity  of  public  authorities  and 
stakeholders and efficient public administration.12  

In each Member State, the partnership agreement (PA)13 describes the investment framework 
for the ESI Funds in that Member  State. The  partnership agreements  are aligned with Europe 

8  Article 4.1. of the ESF Regulation: (Regulation (EU) No 1304/2013 of the European Parliament and of 
the  Council  of  17 December  2013  on  the  European  Social  Fund  and  repealing  Council  Regulation 
(EC) 
https://eur-lex.europa.eu/legal-
content/en/TXT/?uri=CELEX%3A32013R1304   

No 1081/2006, 

20.12.2013, 

347, 

OJ 

L 

9   https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52015DC0639 

10  https://ec.europa.eu/commission/priorities/deeper-and-fairer-economic-and-monetary-union/european-

pillar-social-rights/european-pillar-social-rights-20-principles_en  

11  The scope of ESF assistance for the 2014-2020 programming period is defined by Article 3 of the ESF 

Regulation No 1304/2013 

12  Not  covered  in  this  evaluation,  as  it  was  recently  the  subject  of  a  separate  study.  See: 

https://op.europa.eu/en/publication-detail/-/publication/c7c56421-9fc4-11ea-9d2d-
01aa75ed71a1/language-en  

13  http://ec.europa.eu/contracts_grants/agreements/index_en.htm  

8 

 
                                                           
 
2020,  the  EU  strategy  for  smart,  sustainable  and  inclusive  growth,  taking  into  account  where 
appropriate the relevant country specific recommendations14.   

The actual implementation is framed by programmes agreed between the Commission and the 
Member States, known as operational programmes for cohesion policy funds15.  Besides the EU 
funding,  operational  programmes  also  receive  national  funding,  public  or  private.    The  total 
budget for ESF operations for 2014-2020 is EUR 121 billion, with an EU contribution of EUR 84 
billion16.    National  co-funding  ranges  from  about  20%  to  over  50%,  and  depends  on  the 
development  of  regions,  with  less  developed  regions  expected  to  contribute  less  than  more 
developed ones.  

Operational  programmes  are  structured  by  priority  axes,  each  composed  of  one  or  more 
investment  priorities  setting  at  least  one  specific  objective. Priority  axes may combine  one  or 
more complementary investment priorities from the ESF, the European Regional Development 
Fund (ERDF) and the Cohesion Fund. They may have a national or regional dimension. In total, 
187  operational  programmes  received  ESF  funding  in  the  2014-2020  programming  period. 
Ninety-two  operational  programmes  are  multi-fund  meaning  that  they  also  receive  ERDF  or 
Cohesion Fund funding. 

Under  shared  management  the  actual  day-to-day  management  of  operational  programmes  is 
entrusted to national bodies, working in accordance with rules established at Union level. The 
Commission supervises implementation and retains ultimate responsibility for the execution of 
the  EU  budget.  The  operational  programmes  fund  projects  or  operations  run  by  a  range  of 
public and private organisations called beneficiaries. These projects benefit their participants – 
for the ESF, these are usually individuals, but they can also be companies or organisations. The 
selection of projects, their monitoring, their control and the reporting are done by the national 
bodies.  

The overall intervention logic of the ESF is given in Annex 4. 

2.1.2  ESF support to employment and labour mobility (thematic objective 8) 
Thematic  objective  8  promotes  sustainable,  quality  employment  by  boosting  job  creation, 
helping  people  into  employment,  and  supporting  labour  mobility.  The  investment  priorities 
supported within TO8 can be separated into three broad categories: 

• 

investment priorities directly supporting access to employment  

14  https://ec.europa.eu/info/publications/partnership-agreements-european-structural-and-investment-

funds_en  

15  Besides  the  ESF,  the  cohesion  policy  funds  include  the  European  Regional  Development  Fund 
(ERDF)  and  the  Cohesion  Fund  (CF).  The  2014  -  2020  7-year  European  budget  set  aside  EUR352 
billion  for  cohesion  policy.    More  than  half  of  this  –  EUR182.2  billion  –  was  set  aside  for  less 
developed  regions.  These  are  regions  with  a  per  capita  GDP  less  than  75  %  of  the  EU-27  average. 
EUR35 billion was allocated to transition regions, which are those with a per capita GDP between 75 
% and 90 % of the EU average, and EUR54 billion to more developed regions which are those with a 
per capita GDP of more than 90 % of the EU average. 

16  https://ec.europa.eu/regional_policy/en/policy/how/is-my-region-covered/  

9 

 
                                                           
o  8i  Access  to  employment  for  job-seekers  and  inactive  people,  including  the 
long-term  unemployed  and  people  far  from  the  labour  market,  also  through 
local employment initiatives and support for labour mobility; 

o  8iv  Equality  between  men  and  women  in  all  areas  including  in  access  to 
employment,  career  progression,  reconciliation  of  work  and  private  life  and 
promotion of equal pay for equal work;  

o  8vi Active and healthy ageing  

It also includes 8ii (sustainable integration into the labour market of young people) 
which has been covered by another evaluation exercise17.  

• 

• 

investment priorities supporting businesses  
o  8iii Self-employment, entrepreneurship and business creation; 
o  8v Adaptation of workers, enterprises and entrepreneurs to change. 
investment priority supporting institutions 
o  8vii Modernisation of labour market institutions.  

2.1.3  ESF support to social inclusion (thematic objective 9) 
Thematic  objective  9  promotes  social  inclusion  and  combats  poverty  and  any  discrimination. 
The  six  investment  priorities  supported  under  TO9  tend  to  be  characterised  by  a  focus  on  a 
specific  target  group  (marginalised  communities  -  9ii)  or  an  issue  (discrimination  and  equal 
opportunities - 9iii, access to services - 9iv and the social economy - 9v). Investment priority 9vi 
is about the instrument chosen to deliver policy (community-led local development strategies). 
Investment priority 9i (active inclusion) is very broadly defined. In a Member State, a minimum 
of  20%  of  the  ESF  allocation  has  to  be  allocated  to  TO9  (social  inclusion).  This  is  the  only 
thematic objective with such earmarking.  

2.1.4  ESF support to education and training (thematic objective 10)  
Thematic  objective  10  focuses  on  ‘investing  in  education,  training  and  vocational  training  for 
skills and lifelong learning’, delivered through four interlinked investment priorities, which cover 
all stages of education and target a wide range of potential groups:  

• 

• 

• 

• 

10i  Reducing  and  preventing  early  school  leaving  and  promoting  equal  access  to 
education,  including  formal,  non-formal  and  informal  learning  pathways  for 
reintegrating into education and training;  
10ii Improving the  quality and efficiency of, and access  to, tertiary and equivalent 
education and training;  
10iii  Enhancing  equal  access  to  lifelong  learning  for all  age  groups  in  formal,  non-
formal and informal settings; and   
10iv Improving the labour market relevance of education and training systems.  

2.2  Baseline and points of comparison  

Four  of  the  headline  targets  of  the  Europe  2020  strategy  for  jobs  and  smart,  sustainable  and 
inclusive  growth18, are directly relevant  to the ESF scope  of action. They represent  the  labour 

17  https://ec.europa.eu/social/main.jsp?langId=en&catId=89&newsId=9793&furtherNews=yes  

18  For 

more 

information 

on 

the 

Europe 

2020 

strategy 

please 

see: 

https://ec.europa.eu/eurostat/web/europe-2020-indicators  

10 

 
                                                           
market, social inclusion and education and training dimensions of the strategy. Figure 2 below 
shows the distance to the EU target for these four headline indicators. By the end of 2018, the 
EU seemed to be on track for the targets on the employment, early leavers from education and 
training and tertiary educational attainment. However, the target of lifting 20 million people out 
of  poverty  or  social  exclusion  seemed  out  of  reach.  There  are  however,  large  differences 
between and even within Member States. The COVID 19 pandemic is already having an impact 
on these indicators. 

Figure 2. Europe 2020 - Distance to targets for headline indicators 

Source: Eurostat: https://ec.europa.eu/eurostat/web/europe-2020-indicators/europe-2020-
strategy/main-tables  

Abbreviations: lhs: left hand scale; rhs: right hand scale. 

2.2.1  Employment   
One  of  the  headline  targets  of  the  Europe  2020  strategy  adopted  in  2010  by  the  European 
Council is to  increase the employment rate of the population aged 20-64 to at least 75% by 
2020. In addition, Member States set national targets for the employment rate in 2020.  

Between  2014  and  2018,  the  EU  employment  rate  grew  by  4.0  percentage  points  to  73.2%, 
surpassing the pre-crisis 2008 rate of 70.2% in 2015. However, large differences in employment 
rate  remained  between  Member  States  and  also  between  gender  and  age  groups. 
Employment  rates  (in  2018)  are  below  70%  in  Belgium,  Greece,  Spain,  Croatia,  Italy  and 
Romania but close to 80% or above in Czechia, Germany, Estonia, the Netherlands and Sweden. 
Employment rates, in Greece (in particular), Cyprus and Spain (marginally) were still below pre-
crisis levels.  

The employment rate gap between men and women is almost 12 percentage points. The main 
reason behind the employment gap between women and men is the high share of women in 

11 

 
 
inactivity (in 2018 27.6% vs. 15.7% of men). In 2014, nearly a third of inactive women aged 20-
64 (29.4%) were not seeking employment due to family and caring responsibilities.  

The employment rate is correlated with age. The highest rate is noted for the 40-44 age group 
(83.1%  in  2018).  The  employment  rate  drops  to  71.8%  for  the  55-59  age  group  and  to  only 
44.4%  for the 60-64 age  group. Nevertheless,  the employment rate of the older workers has 
been increasing more rapidly than the overall employment rate by 7.8% for the 60-64 age group 
compared  to  a  4.0%  overall  increase  from  2014  to  2018  (see  Figure  3).  Active  ageing,  i.e 
increasing  the  proportion  of  older  workers  still  in  employment  remains  key  to  achieving  the 
overall employment target. 

Figure 3. Employment rate as function of age in 2014 and in 2018 

Source: Eurostat: lfsa_ergan 

In  2013  when  the  unemployment  rate  in  the  EU  reached  its  peak  (10.8%),  about  20.5  million 
people  in  the  25-64  age  group  were  unemployed.  In  2018,  though  the  unemployment  rate 
decreased  to  6.8%  (below  the  pre-crisis  level  of  7.0%),  there  were  still  16.9  million  people 
unemployed.  At  the  same  time,  the  share  of  long-term  unemployed  among  unemployed, 
though  considerably  down  from  its  peak  in  2014  (38.5%)  was  still  above  the  pre-crisis  level 
(35.0%; 33.4% in 2008).19  

Between 2014 and 2018, as the economic situation improved, youth unemployment rates fell in 
all  Member  States  and  by  an  average  of  7  percentage  points  at  EU  level.  However,  not  all 
Member  States  saw  the  same  degree  of  improvement.  Over  that  period,  the  rate  of  young 
people  not  in  employment,  education  or  training  fell  from  12.5%  in  2014  to  10.5%  in  2018. 
More  importantly,  it  fell  in  all  countries  but  Germany  and  Denmark,  where  it  only  marginally 

19  Eurostat: lfsa_upgan 

12 

 
 
 
                                                           
increased. The improvement derives entirely from the fall in the number of unemployed young 
people (down 3 percentage  points) as the number of inactive  young people slightly increased 
(up  1.2  percentage  points).  The  composition  of  the  target  population  is  thus  shifting,  with  a 
higher  share  of  inactive  young  people  (58%  in  2018).  For  young  migrants  (15-24  years),  the 
unemployment rate fell from 29.2% in 2014 to 20.9% in 2018.  

In  2014,  29.9  million  people  aged  25-64  were  self-employed  which  corresponds  to  15.3%  of 
those employed. By 2018 the share of self-employed dropped by 1 percentage point to 14.4% 
which is 29.6 million people aged 25-64 years. Thus, at EU level the number of self-employed 
people  has  hardly  changed  since  2014,  but  since  the  total  number  of  employed  has  risen 
significantly the share of self-employment has gone down.     

It is broadly acknowledged that the success of EU businesses and industry, coupled with a fair 
and balanced socio-economic legal framework,  is vital to increasing employment  and creating 
new  and  better  jobs.  In  2014,  there  were  23.3  million  SMEs  in  the  EU-28’s  non-financial 
business economy accounting for more than 99% of all enterprises. The overwhelming majority 
of these (92.5% of all enterprises) were enterprises employing fewer than 10 people. By 2016 
(the  most  recent  year  for  which  there  are  complete  available  data  for  the  EU  as  whole),  the 
number of SMEs increased to 24.7 million (still accounting for more than 99% of all enterprises 
in the EU’s non-financial business economy) employing 94.9 million people.  

2.2.2  Labour mobility 
Labour mobility has both a geographical (place of employment) and an occupational dimension 
(type  of  work  performed).  Free  movement  of  workers  (geographical  mobility)  is  one  of  the 
central tenets of EU policies. Labour mobility in the European Union is increasing. One measure 
of geographical mobility is the share of non-nationals in total employment20.  

Between 2014 and 2018, the share of non-nationals in total employment increased from 7.1% 
to  8.3%21.    There  are  very  big  differences  between  Member  States.  In  Luxembourg,  over  half  
the  employed  are  non-nationals  while  in  Bulgaria,  Croatia,  Lithuania,  Poland,  Slovakia  and 
Romania,  less  than  1%  of  the  employed  persons  are  non-nationals.  Other  aspects  of 
geographical mobility include cross-border work and commuting between regions. Occupational 
mobility  is  more  difficult  to  define  and  measure  but  also  seems  to  be  increasing.  Job  tenure 
tends to become shorter while the age at which people leave the labour market is increasing. In 
the EU28, the share of long job tenures (60 months or more) declined from 62.9% in 2014 to 
60.8% in 201822.  The EU28 average duration of the working life has increased by nearly one full 
year between 2014 (35.3 years) and 2018 (36.2 years)23.  

20  For 

more 

statistics 

on 

mobility 

see: 

https://ec.europa.eu/eurostat/cache/digpub/eumove/index.html?lang=en  

21  Eurostat: lfsa_egan 

22  Eurostat: lfsa_egad 

23  Eurostat: lfsi_dwl_a 

13 

 
                                                           
2.2.3  Social inclusion 
The EU generally saw an improvement in living standards in 2014-2018. This improvement is 
attributed  in  part  to  increases  in  real  median  income  and  household  incomes,  as  well  as 
improvements in economic activity and the labour market. Gross disposable household income 
has increased in real terms since 2012-2013 in nearly all Member States, although in some this 
has not yet returned to pre-crisis levels (notably some southern Member States)24.     

Despite  these  positive  developments,  as  well  as  the  Europe  2020  target  of  lifting  20  million 
people out of poverty, over one fifth of the EU population remains at risk of poverty or social 
exclusion. The Europe 2020 target was set before the 2008 economic and financial crisis, which 
had a detrimental impact on the EU’s ability to reach this target.  The highest rates are found in 
the  southern  and  eastern  regions  of  the  EU.  The  proportion  of  people  suffering  from  severe 
material and social deprivation declined between 2014 and 2018 from 8.9% to 5.9% in the EU-
2825.    The  proportion  of  people  in  the  EU-28  experiencing  severe  housing  deprivation  slightly 
declined from 5% to 4% in the same period. Those who were disproportionately affected by this 
condition  included  persons  at  risk  of  poverty,  tenants,  households  with  dependent  children, 
people in rural areas and Roma.  

24  Employment  and  Social  Developments 

in  Europe  –  Annual  Review  2018.  Available  at: 

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8110  

25  Eurostat: ilc_mdsd07 

14 

 
                                                           
Figure 4. Proportion (in %) of the EU population at risk of poverty or social exclusion 

Source: Eurostat, People at risk of poverty or social exclusion by age and sex [ilc_peps01.  

Notes:  The  middle  line  is  the  EU28  average;  the  outer  lines  represent  the  maximum  and  minimum 
observed at the level of the Member States.  

On access to services, the proportion of people in the EU-28 self-reporting unmet needs for 
medical  examination  declined  from  6.7%  to  3.6%  between  2014  and  2018.  However,  strong 
inequalities persist across certain groups of the population, with people in lower income groups, 
the  unemployed,  people  with  low  education  and  people  living  in  rural  areas  more  likely  to 
report unmet needs for medical examination. Across the EU-28, the reason most often cited is 
that it is too expensive, followed by long waiting lists. Although in 2016, 86.3% of EU children 
from 3  up  to  the  minimum  compulsory  school  age  received  formal  early  childhood  education 
and care, this varied widely between countries, from 51.3% in Croatia to 98.6% in Belgium26.  

Although employment is generally seen as a route out of poverty, in 2018 the EU average in-
work poverty rate was 9.5%, unchanged from 2014. Groups more at risk of in-work poverty are 
people in households with low work intensity, single parents with dependent children, people 
with low education, migrants, people with a disability and young people. Regional disparities in 
unemployment  and  the  prevalence  of  in-work  poverty  persisted.  Although  unemployment 
rates declined between 2014 and 2018, strong regional disparities persisted, notably in Belgium, 
Bulgaria, France and Italy.  

26  Eurostat: ilc_caindformal 

15 

 
 
 
                                                           
2.2.4  Education and training 
The rate of early leavers from education and training (ELET), one of the Europe 2020 headline 
targets,  fell  from  11.9%  in  2013  to 10.6%  in 201827    although  it  has still  not  reached  the  EU 
2020 target (below 10% of 18-24 year olds) and rates of ELET vary significantly across Member 
States.  A total of 17 Member  States  were  below  the overall EU target of 10%  in 2018.  16 of 
them  had  already  achieved  it  in  2013,  but  reductions  can  be  observed  in  almost  all  Member 
States.  However,  despite  some  improvement  in  the  Member  States  that  had  a  particularly 
unsatisfactory performance at the start of the period (Spain, Italy, Malta, Portugal and United 
Kingdom),  there  are  still  Member  States  that  are  not  reaching  the  EU  target.  These  include 
those mentioned above as well as Bulgaria, Denmark, Germany, Estonia, Hungary and Romania.  

Tertiary  education  attainment,  another  Europe  2020  headline  target,  experienced  steady 
growth during the programming period, from 37.1% in 2013 to 40.7% in 2018. Approximately 
two-thirds of the Member States (18 of 28) had already achieved the EU target by 2018 (40% of 
30-34 year olds have completed tertiary or equivalent  education),  up from 16 Member States 
that had achieved it in 2013. All countries made significant progress, most notably some Eastern 
European Member States. There are still several countries where further efforts are needed to 
achieve their national targets, however, including Bulgaria, Germany, Croatia, Hungary, Malta, 
Portugal  and  Romania.  Moreover,  significant  gender  imbalances  remain  in  tertiary  education 
enrolment  in  specific  subjects;  women  are  underrepresented  in  STEM  (science,  technology, 
engineering and mathematics) disciplines (e.g. in ICT, female enrolment is 18% v. 82% for men, 
and in engineering, manufacturing and construction it is 26.1% vs 73.9%)28.  

Regional  disparities  in  ELET  and  tertiary  education  attainment  remain  significant  across  the 
EU.  Overall, 60 EU  regions of 281 considered in the analysis and for which data are available, 
have improved their relative performance in ELET during the current programming period. 170 
EU regions show  relative  stability while ELET  performance  deteriorated over the  period in 51. 
The  challenge  of  reducing  and  preventing  early  leavers  from  education  and  training  is  still 
particularly  substantial  in  several  Mediterranean  and  Eastern  European  regions.  As  regards 
tertiary education attainment, 40 regions improved their relative performance in 2013 – 2018, 
184 regions remained relatively stable and the performance of 52 regions decreased. There are 
significant  remaining  challenges  to  improving  the  quality  and  efficiency  of,  and  access  to, 
tertiary education in several central and eastern European regions, in Italy’s south and in several 
Eastern German regions. 

The participation of adults in education and training increased over time, reaching 10.1% for 
men  and  12.1%  for  women  in  2018.  The  participation  of  employed,  unemployed  and  inactive 
persons in education and training recovered after the economic crisis and increased during the 
current  programming  period  but,  in  this  case  too,  there  are  significant  disparities  between 
countries and more than half of the Member States have a participation rate which is below the 
EU average. 

27  The share of early school leavers is systematically higher among males and this gender disparity is not 

showing any signs of reduction over time. 

28  The  main  driver  being  possibly  the  persistence  of  man/women  stereotypes  in  education  and  socio-

cultural environment. 

16 

 
                                                           
Overall, the number of Vocational Education and Training (VET) students has been declining in 
the  EU  over  the  decade,  with  some  exceptions  such  as  an  increase  of  upper  secondary  level 
students  in  Estonia,  Cyprus29,  Luxembourg,  Malta,  Slovenia  and  the  United  Kingdom;  post-
secondary  level  students  in  Czechia  and  France  and  short-cycle  tertiary  level  students  in 
Sweden. 

The share of students enrolled in vocational education are highest in post-secondary (91.5% in 
2017)  and  short-cycle  tertiary  levels  (86.4%).  In  lower  secondary  education,  the  share  of  VET 
students  is  low  (7.4%)  while  in  upper  secondary  education  there  is  a  substantial  equilibrium 
between students in general and vocational education.  

Educational  outcomes  on  basic  skills  (reading,  mathematics  and  science),  according  to  the 
OECD Programme for International Student Assessment (PISA),  also vary significantly across 
countries  and  regions,  and  these  disparities  persist,  with  some  Member  States  recording  a 
performance  that  is  consistently  below  the  average.  Furthermore,  no  improvement  can  be 
observed  in  PISA  2018,  in  the  sense  that  the  EU  has  not  made  any  progress  towards  the  ET 
202030  benchmark on the share of low-achievers. 

Finally,  the  employment  rate  of  recent  graduates  increased  during  the  2014-2018  evaluation 
period,  although  has  not  fully  recovered  from  the  economic  crisis.  The  challenges  include  a 
persistent  gender  gap  in  the  employment  rate  of  recent  graduates,  which  has  not  changed 
substantially  during  the  current  programming  period,  driven  by  the  advantage  that  men 
continue  to  have  in  entering  the  labour  market.    In  2018,  for  upper  secondary  and  post-
secondary non-tertiary education graduates, the employment rate for men was 76.7% vs 71.4% 
for women. The gender gap was therefore 5.3 percentage points. For recent tertiary education 
graduates,  the  employment  rate  was  83.9%  for  women  and  87.4%  for  men  (gap  of  3.5 
percentage points in 2018). 

3 

IMPLEMENTATION / STATE OF PLAY 

3.1  Programming and budget 

About EUR 130 billion are available for ESF and YEI operations in the 2014 – 2020 programming 
period. The EU contribution corresponds to 71% of total planned expenditure. National public 
and private contributions make up the balance31.  The YEI is implemented through investment 
priority  8ii  under  thematic  objective  8.  Figure  5,  below,  shows  the  budgets  per  thematic 
objective. 

29  Although from a very low baseline rate. 

30  The  strategic  framework  for  European  cooperation  in  education  and  training  (ET  2020). 

https://ec.europa.eu/education/policies/european-policy-cooperation/et2020-framework_en  

31  Source: Open data portal: https://cohesiondata.ec.europa.eu/  

17 

 
                                                           
Figure 5. Total ESF and YEI budget per thematic objective - EUR millions 

Source: https://cohesiondata.ec.europa.eu  

3.2 

Employment and labour mobility 

ESF  support  to  employment  and  labour  mobility  is  supported  specifically  (excluding  youth 
employment  support)  through  151  operational  programmes,  in  all  Member  States.  The  total 
allocation  of  these  programmes,  excluding  8ii32,  is  EUR 32.1 billion,  which  is  the  equivalent  of 
26%  of  the  total  ESF  budget  for  2014-2020.  This  underlines  that  Member  States  consider 
support  for  employment  and  labour  mobility  as  a  central  objective  within  the  ESF.  Figure  6 
shows the indicative allocations per investment priority33.  

Figure 6. TO8 - Indicative allocations per investment priority - in EUR millions 

32 

Investment  priority  8ii  corresponds  to  youth  employment.  The  ESF  support  to  Youth  Employment 
at 
staff 
https://ec.europa.eu/social/main.jsp?langId=en&catId=89&newsId=9793&furtherNews=yes  

document 

working 

found 

can 

be 

33  While  mobility  is  mentioned  explicitly  in  the  title  of  TO8,  there  is  no  specific  investment  priority 
dedicated  to  labour  mobility.  In  theory,  Member  States  can  programme  operations  towards  mobility 
under any investment priority and it can mean different things as explained in the text above. 

18 

 
 
 
                                                           
Source: https://cohesiondata.ec.europa.eu   

Access to employment (8i) is the dominant investment priority, absorbing more than half of the 
entire budget allocated to TO8 (53%) and programmed by all Member States except Denmark34.  
Adaptability  to  change  (8v)  is  another  substantial  priority,  corresponding  to  EUR 7.3  billion 
(23%).  It  is  particularly  important  in  Germany  (EUR 2.2  billion)  and  France  (EUR 1.3  billion), 
where it represents over half of the total TO8 investments.  

The ESF TO8 actions in transition regions is comparatively less focused on access to employment 
(44%; EU average 53%). These regions spend more on entrepreneurship (17%) and adaptability 
(33%).  Programmes  in  less  developed  regions  concentrate  comparatively  less  on  adaptability 
(16%; 23%), to favour access to employment.  

3.3 

Social inclusion 

In the current  programming period TO9 operations are  financed by 145 operational programs 
financing covering all Member States and regions. They account for a total planned expenditure 
(including  EU  and  national  co-financing)  of  approximately  EUR 31.3  billion  (of  which  EUR 21.4 
billion EU funding). This corresponds to about one quarter of total ESF funding which is more 
than the required earmarking. The absolute amounts planned vary considerably across Member 
States,  from  EUR  8  million  in  Luxembourg  to  EUR  4.0  billion  in  Germany.  The  total  amounts 
planned for TO9 as percentage of total ESF does range from 20% in Finland to over 70% in the 
Netherlands. These budgets have remained relatively stable in 2014-2018, with a slight overall 
increase of less than 1%. The biggest increases in absolute terms were in Greece. Figure 7Figure 

34  Note that youth employment (formally also addressed by thematic objective 8) is excluded from these 

figures, since this is the subject of a separate evaluation. 

19 

 
 
                                                           
7.  TO9  -  Indicative  allocation  by  investment  priority  -  in  EUR  millions  shows  the  indicative 
allocation per investment priority.  

Figure 7. TO9 - Indicative allocation by investment priority - in EUR millions 

Source: https://cohesiondata.ec.europa.eu  

Similar amounts are allocated to less developed regions (43%) and to more developed regions 
(42%). The balance (15%) goes to transition regions. Looking at investment priorities (Figure 7), 
around  73%  (EUR  22.1  billion)  is  allocated  to  9i  (active  inclusion).  The  next  most  important 
investment priority in terms of indicative allocation is 9iv (access to services) at 17% (EUR 4.8 
billion).    The  remaining  investment  priorities  represent  between  1%  and  6%  of  total  funding 
allocated to TO9. A similar picture emerges at MS level. Generally, the bulk of the allocations is 
to  9i  (active 
inclusion).  In  eleven  MS  (Belgium,  Denmark,  Germany,  Ireland,  France, 
Luxembourg, Austria, Finland, Sweden and United Kingdom), 9i even amounts to 90% or more 
of the total TO9. Only eight MS allocate the biggest share of the total to a priority other than 9i  
(9ii – Hungary; 9iii – Cyprus; 9iv – Bulgaria, Estonia, Croatia, Latvia and Romania).  

3.4 

Education and training 

Total  planned  expenditure  on  TO10  in  the  current  programming  period  is  approximately  EUR 
39.2  billion  (including  EU  and  national  co-financing).  This  corresponds  to  nearly  one  third  of 
total  ESF  (32%).  The  amounts  planned  vary  considerably  across  Member  States,  from  EUR  5 
472.8  million  in  Italy  to  EUR  8  million  in  Luxembourg  and  zero  in  the  Netherlands.  The  total 
amounts  planned  on  TO10  as  a  percentage  of  total  ESF  range  from  a  maximum  of  53%  in 
Portugal, to a minimum of 15% in Cyprus, ignoring the Netherlands.  

Planned expenditure on TO10 relating to public spending on education is significant in several 
EU  countries.  For  example,  total  TO10  is  7.1%  of  general  government  expenditure  on 

20 

 
 
 
education35    in  Portugal,  4.2%  in  Romania  and  4.1%  in  Poland.  TO10  is  also  important  in  the 
Baltic countries, Bulgaria, Greece, Croatia, Hungary and, Slovakia.  

More than half of the total TO10 funds, 51.4%, goes to less developed regions, 34.8% to more 
developed regions and 13.7% to transition regions. Looking at investment priorities (see Figure 
8), around 29% (EUR 711.24billion) is planned on 10i (early leavers from education and training), 
15% (6 billion) on 10ii (tertiary education), 30% (11.8 billion) on 10iii (lifelong learning) and 26% 
(10.19 billion) on 10iv (labour market relevance). 

Figure 8. TO10 - Indicative allocations by investment priority - in EUR millions 

Source: https://cohesiondata.ec.europa.eu  

Several Member States have selected all investment priorities, but only a few (such as Portugal) 
have a fairly equal distribution of funds across all. Many Member States decided to concentrate 
on a limited number. For instance, France focuses on 10iii and 10i, the United Kingdom on 10iii 
and 10iv, Belgium, Slovenia and Sweden on 10iii, and Czechia on 10i and 10ii. 

Between 2016 and 2018 there were changes in ESF resources planned under TO10. Overall, in 
the  EU,  there  was  a  0.1%  increase  in  total  planned  resources  (EU  and  national  co-financing). 
Looking  at  the  ESF  component  only,  the  positive  change  between  2016  and  2018  was 
EUR +262.6 million (+1%). However, considering that the total net increase (EU and national co-
financing) was much smaller and amounted approximately to EUR 55 million between 2016 and 
2018,  the  observed  positive  change  in  planned  ESF  (i.e.  EUR 262.2  million)  was  clearly 
compensated by a reduction in national resources.  

35  Annual average for 2014-2018. 

21 

 
 
 
                                                           
In absolute values,  the  biggest increases (considering both EU and national co-financing) took 
place in Spain (EUR +300.7 million) and Greece (EUR +182.3 million) while the largest reductions 
were recorded in the United Kingdom (EUR -322 million) and Italy (EUR -116.4 million) where, 
however, there was a 32.6 million increase in EU resources planned under TO1036.  

3.5 

Typology of Operations funded 

3.5.1  ESF support to employment and labour mobility 
The  supporting  study  conducted  a  mapping  of  the  types  of  operations.  ESF  TO8  operations 
combining types of operations are the most common and represent 18% of the costs. Support 
for  entrepreneurs  is  also  around  18%,  closely  followed  by  guidance  and  support  (12%), 
adaptability (14%), and financial incentives (10%).  

Of investments in the area of entrepreneurship (8iii), 99% consist of operations that provide 
some  sort  of  support  for  entrepreneurs.  Likewise,  adaptability  operations  are  basically  only 
found within the adaptability investment priority (8v), where these operations represent half of 
the reported eligible costs. Operations in support of active ageing dominate the active ageing 
investment  priority  (66%  of  that  investment  priority’s  costs),  while  operations  that  seek  to 
strengthen  institutional  capacity  represent  84%  of  the  costs  reported  for  the  investment 
priority  on  labour  market  institutions  (8vii).  For  access  to  employment,  a  variety  of  types  of 
operation  can  be  found,  such  as  guidance  and  support  (20%),  financial  incentives  (17%), 
education  and  training  (8%),  and  a  considerable  share  of  operations  that  combine  multiple 
types (30%).  

36 

In percentage terms, the most significant changes which can be observed between 2016 and 2018 are 
those concerning Cyprus, where there was a 34.6% reduction, and to a lesser extent Denmark (-8.9%) 
and the UK (-8.6%). Conversely, in Greece there was a 14.9% increase and in Spain a 10.5% increase. 
Overall, the distribution of funds across IPs did not vary considerably. 

22 

 
                                                           
 Table 1. Types of operations per investment priority – TO8 

Programmed under 

Type of operation 

Total TO8 
(excl. 8ii) 

Work-based learning 
4% 
Education & training 
7% 
Guidance & support 
12% 
Financial incentives 
10% 
Support Entrepreneur 
18% 
4% 
Institutional capacity 
Women in employment  5% 
Adaptability 
Active ageing 
Integrated pathways 
Other 
Combined 

14% 
2% 
6% 

18% 

Total 

100% 

Access to 
employ-
ment (8i) 
5% 
8% 
20% 
17% 
3% 
4% 
1% 
1% 

10% 

30% 

100% 

Entre-
preneurship 
(8iii) 

Gender 
equality 
(8iv) 

Adaptability 
(8v) 

Active 
ageing 
(8vi) 

LM insti-
tutions (8vii) 

10% 

3% 
3% 

80% 

2% 

2% 

100% 

7% 
9% 
7% 
2% 
7% 
1% 
1% 
55% 
2% 
3% 

5% 

100% 

99% 

37

100%

8% 

84% 

3% 

5% 

100% 

1% 
66% 

33% 

100% 

Source: mapping by authors, based on qualitative screening of the annual implementation reports 2015-
2018. 

Mapping of all operations under TO8 shows that in practice almost all mobility operations can 
be  found  under  the  ‘Access  to  employment’  (8i),  and  to  a  lesser  extent  in  support  to  labour 
market institutions (8vii). Investments targeting labour market institutions are relatively small, 
with  an  overall  investment  of  EUR  1.3  billion.  These  investments  include  support  for  labour 
market institutions to improve their provision of labour mobility services (e.g. through EURES).  

3.5.2  Social inclusion 
Social Inclusion operations are complex, highly diverse and delivered to a range of vulnerable 
groups  as  well  as  entities.  There  is  significant  overlap  between  investment  priorities  under 
TO9.  The  definition  of  9i  is  broad  and  actions  may  more  easily  fit  within  it  than  under  other 
more  narrowly  defined  investment  priorities  such  as  9v.  As  a  result,  most  operational 
programmes covered 9i. 

The supporting study on social inclusion38  identifies six types of operations on the basis of the 
objective actually pursued and the target groups aimed at. Four types of operations encompass 
actions directed to people, while the two remaining types comprise actions directed to people 
and to organisations or society at large (e.g. capacity building of social services, support to social 
enterprises,  information  campaigns).  Table  2  below  presents  the  six  types  of  operations  and 
their prevalence across the operational programmes, including TO9 actions. 

37  All other types of operations that support entrepreneurship amount cumulatively to 1%. 

38  https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes  

23 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
                                                           
Table 2. Types of operations per investment priority - TO9 

Main target 
groups 

  people with a 
disability 
39

  LTU
  LTU 
  people with a 
disability 
  Low skilled 
  people with a 
disability 

  LTU 
  Recipients of 
minimum 
income 

  people with a 
disability 

  LTU 
  people with a 
disability 
  SMEs, social 
enterprises 

Type of operation 

1  

Employment-
focused actions 

2   Enhance basic skills 

3  

Basic school 
education 

4   Access to services 

5  

6  

Social 
entrepreneurship 

Actions influencing 
attitudes and 
systems 

Number of 
operational 
programmes 

programmed under 

9i 

I9ii 

9iii 

9iv 

9v 

9vi 

136 

95% 

12% 

6% 

1% 

9% 

29 

69% 

17% 

14% 

25 

32% 

48% 

20% 

97 

75 

54% 

22% 

21% 

68% 

1% 

2% 

25% 

67% 

12% 

79 

47% 

19% 

16% 

41% 

25% 

9% 

Source: mapping by authors, based on qualitative screening of the annual implementation reports 2015-
2018. 

This  pattern  of  prevalence  of  investment  priorities  and  types  of  operations  across  the 
operational programmes is also reflected in terms of finances. Investment priority 9i accounts 
for 71% of indicative TO9 allocations, 73% of the planned allocation and 80% of the declared 
expenditure.  This concentration on one investment priority is less marked at the level of the 
types of operations. The operations aiming to move participants in employment (type 1) are the 
most  important.  They  represent  45%  of  the  planned  allocations  and  48%  of  the  declared 
expenditure. 

Investment priority 9vi (community-led local development, CLLD) is a very specific case. CLLD is 
an  instrument  to  involve  local  private  and  public  actors  and  to  propose  integrated,  local 
solutions. In total, 14 Member States plan the use of CLLD, combining ESF and other ESI Funds 
resources,  mostly  EARDF40.  Investment  priority  9vi  accounts  for  2.4%  of  the  indicative  TO9 
allocations, 1% of the planned allocations and only 0.2% of the declared expenditure.  

3.5.3  Education and training  
The analysis of the operations financed by ESF under TO10 highlighted that secondary and post-
secondary education and training is the most important area of intervention in financial terms. 
Overall, 28.3% of total eligible costs are related to operations supporting vocational secondary 
education  and  training,  and  a  similar  share,  27.4%  of  the  total  supports  general  secondary 
education and training.   

39  Long-term unemployed 

40  See 

conference. 
https://leaderconference2019.minhaterra.pt/rwst/files/I119-CLLDX231219XSERVILLOXKAH.PDF  

presented 

Leader 

paper 

2019 

the 

at 

24 

 
 
 
 
 
 
 
 
 
 
 
 
  
                                                           
As  regards  the  other  areas  of  intervention,  21.2%  is  allocated  to  non-formal  job-related 
education and training and 4.3% to non-formal non job-related education and training; 15.4% 
of  the  total  went  to  tertiary  education  and  3.5%  is  allocated,  altogether,  to  pre-primary  and 
primary education (see Figure 9 below). 

The distribution of funds by area of operation varies considerably between Member States. For 
example, more than 50% of the funds are allocated to vocational secondary and post-secondary 
education and training in Cyprus, Austria and Portugal while the focus is on general (lower and 
upper)  secondary education and training in Bulgaria,  Estonia, Spain, Italy, Latvia, Slovenia and 
Romania. Non-formal job-related education and training prevail in Belgium, Denmark, Ireland, 
Slovenia, Finland and the United Kingdom, while non-formal non job-related education has an 
important role in Ireland. Tertiary education is central in Croatia, Lithuania, Malta and Poland, 
while pre-primary and primary education are significant in Czechia.  

Figure 9. Distribution of thematic objective 10 eligible costs by area of operation 

Source: mapping by authors, based on qualitative screening of the annual implementation reports 2015-
2018. 

The  distribution  of  funds  by  area  of  operation  varies  also  across  investment  priorities.  Under 
10i,  68.2%  of  total  resources  are  allocated  to  support  general  secondary  and  post-secondary 
education  and  training.  20.1%  of  total  allocations  go  to  vocational  secondary  and  post-
secondary education and training and 8.6%  to primary  education. Under 10ii, most  resources, 
86.7%,  go to tertiary  education operations, followed by non-formal job-related education and 
training (9.1% of the total).  

Under 10iii, funds are concentrated on non-formal job-related education and training (53.8%), 
while  31%  go  altogether  to  vocational,  general  secondary,  post-secondary  education  and 
training, and 13% to non-formal non job-related education and training. As regards 10iv, funds 
are mostly used for vocational secondary and post-secondary education and training (72.1% of 
total).  14.6%  is  used  for  non-formal  job-related  education  and  training,  and  4.9%  for  tertiary 
education. 

25 

 
 
 
3.6 

Financial and operational implementation 

The  performance  review  foreseen  by  the  Common  Provisions  Regulation  was  carried  out  in 
2019.  It  examined  the  achievement  of  the  milestones  for  financial  and  output  indicators  as 
reported  in  the  2018  annual  implementation  reports  submitted  by  Member  States  in  201941. 
Across  all  thematic objectives,  ESF  performing  priorities  represented  83%  of  the  performance 
reserve for the ESF (i.e. EUR 4.2 billion) that is definitively allocated on the basis of Commission 
decisions  on  the  performance  review.  However,  there  are  important  differences  between 
Member  States  and  programmes.  Nine  Member  States  managed  to  achieve  all  their  ESF 
milestones. On the contrary, in some Member States, most priority axes have not achieved their 
milestones. This is the case in Romania, for example.  

There were also large differences between thematic objectives. In absolute terms, the highest 
amount to reallocate concerned TO9 social inclusion, TO10 education and training and multi-TO 
priorities42.  TO8  employment  performed  well  both  in  absolute  and  relative  terms  with  an 
achievement rate of around 90%. 

At the end of 2018, the number of non-performing priority axes was high, about one fifth, and 
far  beyond  what  was  expected  in  light  of  the  programme  amendments  adopted  in  previous 
years.  It  demonstrates,  inter-alia,  that  the  setting  of  milestones  and  the  selection  of  the 
indicators  are  very  challenging.  The  European  Court  of  Auditors  found  that  for  around  one 
quarter of the indicators included in the performance framework of the examined operational 
programmes  (28  %)  it  was  not  possible  to  draw  conclusions  about  the  accuracy  of  the  target 
value  for  the  milestone,  either  because  the  calculation  methodology  could  not  be  verified  or 
because  there  was  no  detailed  explanation  of  the  unit  costs  or  the  assumptions  made.  It 
considered that around 18 % of the remaining milestones and target values are not realistic and 
achievable  or  insufficiently  ambitious.  Taken  together  this  means  that  in  around  46 %  doubts 
remain as to the milestones specified for releasing the performance reserve43.  

Financial implementation as measured by the ratio of interim payments to the Member States 
in relation to the total allocation is running slower than at a comparable time  in the previous 
programming period. In May 2013, this ratio was approaching 50%. It is just over one third in 
May 2020. The main reasons for this delay in implementation include the late adoption of the 
Regulations, and a more relaxed automatic de-commitment rule44 (a move from a combination 
of N+2 and N+3 in 2007-2013, to a general N+3 for all Member States in 2014-2020). 

41  All figures in this section cover the whole ESF and the YEI. 

42  However, in relative terms, compared to the total allocation by TO, the most significant short-coming 
concerns  TO11  institutional  capacity.  This  thematic  objective  is  not  covered  by  this  staff  working 
document. 

43  ECA special report No 15. Available at: 

https://www.eca.europa.eu/Lists/ECADocuments/SR17_15/SR_PARTNERSHIP_EN.pdf  

44  The  funds allocated have to be spent  within 2 or 3 years of their allocation, failing that they are not 

available anymore, 

26 

 
                                                           
4  METHOD 

4.1  Short description of methodology 

Three  separate  supporting  studies  were  contracted  out  (one  corresponding  to  each  thematic 
objective)45. The evaluation questions specific to each supporting study, derived from the better 
regulation  evaluation  criteria,  were  broken  down  into  a  number  of  sub-questions  for  each 
thematic objective and agreed with the corresponding inter-service groups. The studies applied 
an  appropriate  mix  of  evaluation  methods.  This  includes  desk  research,  public  consultations, 
case  studies,  modelling,  interviews  with  stakeholders  and  Member  States’  officials  and  focus 
groups.  They  are  explained  in  further  detail  in  Annex  C  to  this  staff  working  document.  The 
results of the three twelve-week public consultations can be found in annex 2. 

4.2  Limitations and robustness of findings 

The  findings  and  conclusions  are  reasonably  robust,  since  they  are  based  on  sufficient 
triangulation  of  sources  and  tools,  such  as  the  data  reported  by  the  monitoring  systems, 
stakeholder consultation, case studies…. However, the evaluation also faced several limitations, 
notably: 

• 

• 

• 

• 

Data quality and timeliness. The figures in this SWD are based on the situation as 
reported by the Member States in the annual implementation reports for 2018. The 
cut-off  date  for  all  data  is  thus  31 December 2018.  During  the  process  of 
examination and acceptance of the annual implementation reports, corrections to 
the data may be made. This explains small differences in figures in the supporting 
studies  as the  data  were  extracted from the electronic data exchange  platform at 
different times.  
Detailed  programme  information  is  usually  not  available.  The  ESF  is  funding 
thousands of projects or operations, sometimes even very small ones, each having 
their own logic, implementation partners and procedures.  
Demarcation  of  operations  focused  on  employment  and  labour  mobility  from 
those  focused  on  youth  employment  (covered  by  a  separate  SWD46).  Although 
youth  employment  operations  are  mainly  funded  under  8ii,  they  can  be  found 
across other investment priorities.   
Public  consultations  which  by  nature  cannot  be  considered  as  representative  by 
definition,  although  some  mitigation  measures  have  been  applied,  notably 
triangulation with other data sources. 

45  The final reports of the three studies are available at: 

  TO8 

– 

Employment 

and 

labour 

mobility:  

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8330&furtherPubs=yes  

  TO9 

– 

Social 

Inclusion: 

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes  

  TO10 

–Education 

and 

training: 

46  See 

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8346&furtherPubs=yes  

Youth 

Employment 

staff 

working 

document: 

https://ec.europa.eu/social/main.jsp?langId=en&catId=89&newsId=9793&furtherNews=yes  

27 

 
                                                           
 
• 

• 

• 

• 

• 

• 

Towards the end of the field work, holding focus groups in the Member States and 
at  EU  level  became  impossible  due  to  the  COVID-19  pandemic.  Instead  EU  level 
Delphi surveys47 were held instead. 
Data  for  comparative  analysis  of  the  socio-economic  context,  notably  SILC  data, 
were sometimes only available till 2017. 
Cost effectiveness  analyses,  including  the  attempts  to  conduct  cost  benefit 
analyses, are affected by timeliness and completeness of data, due to delayed and 
non-simultaneous reporting on outputs, results and costs.  
Counterfactual  impact  evaluations48  were  used  only  for  a  small  number  of 
operations49.  The main limitation is that the results cannot be extrapolated to the 
whole  of  the  ESF.    They  also  inform  about  the  effect  of  treatment  single 
intervention, not about the ESF as such.  
Rhomolo  analysis,  like  all  general  equilibrium  models,  need  to  be  based  on  a 
number of simplifying assumptions, although it is a high-quality and mature model 
very well-suited for estimating macroeconomic impacts. 
COVID-19 pandemic. The fieldwork for the studies supporting this SWD was carried 
out  before  the  COVID-19  pandemic  arrived  in  Europe.  This  is  not  a  limitation  to 
assess  the  2014-2018  period  as  such.  It  does  present  challenges  to  assess  the 
relevance of ESF support in the current context.  

Annex 3 to this SWD explains in more detail how these limitations were dealt with.   

5  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS 

5.1  Effectiveness 

5.1.1  Financial and output achievements 
At the end of 2018, financial execution50   was 7%  lower than at a comparable moment in the 
previous programming period (20% versus 27% at the end of 2011). However, this was due to 
expenditure  declarations  (from  final  beneficiaries  to  Member  States  authorities  and  from 
Member  States  to  the  Commission)  rather  than  commitments  or  project  selection  on  the 
ground,  where  levels  were  comparable  to  those  in  2007-201351.  The  delay  in  terms  of  claims 
submitted by MS is only 4.24% (accumulated between 2014 and 2017). In fact, proportionally 

47  The  Delphi  survey  is  a  technique  that  relies  on  successive  rounds  of  interactions  with  a  panel  of 
experts. The experts answer  questionnaires in two or  more rounds.  After each round, a facilitator or 
change agent provides an anonymised summary of the experts' reactions  from the previous round as 
well as the reasons they provided for their judgments. The objective is to build consensus on a number 
of issues. The method was originally developed to produce forecasts. 

48  These methods estimate the net impact of a treatment by comparing the situation for the participants of 

an intervention with that of non-participants. 

49  Pilot and feasibility study on the sustainability and effectiveness of results for European Social Fund 
participants using counterfactual impact evaluations. Available at: https://op.europa.eu/en/publication-
detail/-/publication/84cc9eb9-b33d-11e9-9d01-01aa75ed71a1  

50  Financial execution is the ratio of interim payments made by the Commission to Member States to the 

total amount allocated. 

51  By  July  2020  (latest  data  available)  the  overall  ESF/YEI  project  selection  rate  was  92%  of  total 

programmes allocation. 

28 

 
                                                           
more claims were received in 2018 (13.2%) for the 2014-20 period than in 2011 (12.92%) for the 
2007-13 period. 

Financial  implementation  was,  on  average,  comparable  across  the  three  thematic  objectives. 
There  was  however,  considerable  variation  between  Member  States,  regions,  operational 
investment  priorities  and  actions  types.  Other  things  being  equal,  ESF 
programmes, 
implementation tended to be slower in less developed regions, a notable exception being the 
TO10 (education and training) operations in Portugal. 

Project selection rates52  up to 2018 were similar, 72% for TO8 (employment and mobility), 71% 
for TO9 (social inclusion) and 69% for TO10 (education and training) (see Table 3).  

Table 3. Project selection rates 

TO8 (employment and 
labour mobility) 
72 
38-77 

Project selection rate  
(in %) 
EU average 
Range per investment 
priority 
Range per Member 
State 
Source: https://cohesiondata.ec.europa.eu  

58-111 

TO9 (social inclusion) 

71 
26-77 

48-114 

TO10 (education and 
training) 
69 
64-70 

26-144 

These similar average rates hide considerable differences across Member States and investment 
priorities. Globally,  the  differences  between  thematic  objectives  (TOs)  within  Members  States 
were  limited.  The  major  exceptions  were  Bulgaria,  Slovakia  and  Romania.  Six  Member  States 
(Ireland,  Cyprus,  Luxembourg,  Hungary,  Malta  and  the  Netherlands)  had  basically  already 
engaged  the  full  allocated  amounts  by  the  end  of  2018  across  all  thematic  objectives.  In  the 
other Member States, there was a clear difference between thematic objectives. For instance, 
Bulgaria and Slovakia had fully allocated the planned amounts  for TO8 but  only a third of the 
amount for TO10, with TO9 being in between. The project selection rates were generally higher 
for TO8. In Italy, Poland, Portugal and Finland, the highest selection rates were for TO10. The 
highest projection selection rates in Germany, Austria and Romania were for TO9.   

The  absorption  rates,  defined  as  the  proportion  of  expenditure  claimed  by  beneficiaries  to 
allocated  resources,  were  still  relatively  low  (see  Table  4).  In  three  Member  States,  the 
absorption was very low, less than 10 % for TO9 (Ireland) and TO8 (Romania), a TO10 (Slovakia, 
Romania). 

52  This is the ratio of funds allocated by the managing authority to specific projects or operations to the 

total amount available (by TO) or indicatively planned (by IP). 

29 

 
                                                           
Table 4. Absorption rates 

TO8 (employment and 
mobility) 

Declared expenditure 
rate  
(in %) 
EU average 
Range per investment 
priority 
Range per Member 
State 
Source: https://cohesiondata.ec.europa.eu  

33 
16-41 

7-53 

TO9 (social inclusion) 

TO10 (education and 
training) 

28 
3-32 

5-68 

27 
21-32 

6-45 

The beginning of the programming period coincided with the aftermaths of the economic and 
financial crisis. Across all thematic objectives, the focus of implementation has been on tackling 
urgent  challenges  related  to  the  employment  crisis,  partly  at  the  expense  of  more  structural 
longer-term policies. The econometric analysis carried out for TO8 (employment and mobility) 
indicates that the reduction of unemployment rates in certain countries/regions was associated 
with  lower  financial  progress,  further  stressing  the  fact  that  in  several  Member  States  the 
improvement  of  labour  market  conditions  resulted  in  a  reduction  of  the  pace  of  financial 
implementation.  

A person can participate more in than one in ESF project. Therefore the number reported is of 
participations,  not  participants.  In  total,  there  were  22.2  million  participations over  the 2014-
2018 period (see Table 5). The  differences  in participant  profiles  across thematic objectives  is 
largely in line with what can be expected. TO10 (education and training) had a higher proportion 
of  inactive  young  persons.  TO8  (employment  and  mobility)  focused  on  the  unemployed.  The 
highest proportions of people with a disability, people with a migrant background or minorities 
and people with other disadvantages were recorded for TO9 (social inclusion). 

Table 5. Participant profiles 

Participations (in millions) 
of which 
Employed 
Unemployed 
Long term unemployed 
Inactive 

Women 

Young people (16-24 years) 
Older workers (55-64 years) 
Low skilled (ISCED 1-2) 

TO8 
(IP8ii 
excluded) 
6,9 

25% 
68% 
23% 
7% 

54% 

15% 
11% 
33% 

TO9 

6,2 

15% 
53% 
24% 
31% 

52% 

22% 
11% 
54% 

TO10 

9,1 

25% 
10% 
3% 
66% 

51% 

69% 
4% 
61% 

Total 
ESF/YEI53 

26,3 

20% 
44% 
16% 
36% 

52% 

43% 
7% 
48% 

53  The  differences  can  be  attributed  to  IP8ii  (youth  employment)  and  TO  11  (institutional  capacity) 

which are not covered by this staff working document. 

30 

 
                                                           
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Migrants, people with a foreign background, 
minorities** 
People with disabilities** 
Other disadvantages** 

Homeless or affected by housing exclusion 
From rural areas 

13% 

7% 
11% 

1% 
23% 

28% 

16% 
37% 

4% 
16% 

9% 

3% 
11% 

0% 
23% 

15% 

7% 
17% 

1% 
21% 

Source: https://cohesiondata.ec.europa.eu  

The collection of some types of data has proven problematic. The data marked ** in Table 5 are 
in  special  categories  defined  under  Article  8  of  Directive  95/46/EC.  Data  on  sensitive 
characteristics  were  not  collected  consistently  in  some  Member  States  (for  instance  Portugal 
collected  data  on  disability  only  in  operations  targeted  to  disabled  people).  The  data  on 
homelessness and rural areas were estimated based on sampling and reported only once.  

Figure 10. Number of participations by selected characteristic and thematic objective 

 Source: https://cohesiondata.ec.europa.eu  

Access  to  employment  operations  (investment  priority  8i)  significantly  over  performed  other 
investment priorities under thematic objective 8 (employment and mobility). The employment-
focused  operations  under  thematic  objective  9  (social  inclusion)  received  the  bulk  of  the 
available  financing  and  had  the  highest  success  rates  (calculated  as  the  proportion  of 
participants  for  whom  positive  results  were  recorded).  Based  on  progress  measured  against 
common  output  indicators,  activities  under  investment  priority  10i  (early  leavers  from 
education  and  training  and  access  to  learning  pathways)  and  10iii  (lifelong  learning)  have 

31 

 
 
 
 
 
 
 
 
 
 
 
 
generally performed better than activities under the other thematic objective 10 (education and 
training) investment priorities. 

Regarding TO8, progress towards result targets set to be achieved by 2023 was relatively in line 
with  the  progress  towards  output  targets.  Operations  related  to  access to employment made 
more progress towards their result targets than other investment priorities. By the end of 2018, 
an average of 55% of all result targets have been achieved at EU level (42% if weighted by actual 
size  of  the  targets).  Progress  in  transition  regions  (38%)  was  considerably  lower  than  in  less 
developed  (56%)  and  more  developed  regions  (61%).  Progress  in  Cyprus  and  Sweden  was 
somewhat  of  an outlier,  due  to  possibly  incorrect  target  settings.  Progress  towards  the  result 
targets  was  relatively  modest  in  Bulgaria,  Ireland,  Croatia  and  Malta,  and  has  not  yet  been 
reported for Romania. Overall, adaptability investments showed a similar progress towards the 
targets,  though  it  was  also  influenced  by  a  number  of  outliers  (Bulgaria,  Czechia,  Hungary). 
Progress towards the result targets for investment in entrepreneurship, gender equality, active 
ageing and labour market institutions was slightly lower, varying between 30% and 40%. While 
slightly lower than for other investment priorities, these figures are within the acceptable limits. 

For  TO9  (social  inclusion),  most  targets  were  set  against  programme  specific  indicators.  The 
average  achievement  rate  for  specific  output  indicators54  was  99%,  meaning  that  on  average, 
the targets set for the end of the programming period (2023) were already achieved by the end 
of 2018 (see Table 6). The achievement rates for specific output indicators under TO10 were a 
bit  lower.  There  was  also  less  variation  between  investment  priorities  and  Member  States 
indicating more even implementation.  

Table 6. Achievement rates for specific output indicators 

SOI achievement rates 
(in % of 2023 targets) 

EU average 
Range per investment 
priority 
Range per Member 
State 
Source: https://cohesiondata.ec.europa.eu  

2%-279 

TO8 (employment and 
labour mobility) 
excluding IP8ii 
59 
24%-90 

TO9 (social inclusion) 

TO10 (education and 
training) 

99 
19 -118 

1 -540 

79 
44-87 

3-108 

However, progress towards set targets, including performance framework milestones, may not 
be a very reliable indication of implementation performance. Targets were often set far too low, 
making  it  easy  to  overshoot  them  sometimes  even  by  multiples.  On  the  other  hand  some 
achievement  rates  were  very  low.  The  most  likely  explanations  are  that:  (i)  targets  were  not 
realistic, (ii) implementation had barely started when progress was measured or (iii) reporting 
was not reliable.  

5.1.2  Findings on results and contribution to overarching objectives  
There  has  been  good  progress  on  results  achieved  from  participation  in  ESF funded 
operations,  with  a  total  of  9.5  million  positive  results  by  December  2018  (this  includes 

54  This is the ratio of achievements to targets for specific output indicators with targets. 

32 

 
                                                           
participants who engaged in job searching or found a job, were in employment including self-
employment; entered education/training, or gained a qualification (see Table 7).  

Table 7. Reported results (common result indicators only; upon leaving and after six months) 

(Figures in thousands) 

Results upon leaving 
Inactive participants engaged in job search 
Participants in education or training 
Participants gaining a qualification  
Participants in employment  
Disabled participants with positive result 
Total immediate common result indicators 
Results at six months 
Participants in employment 
Participants with improved labour market 
situation 
Older workers in employment 
Disadvantaged in employment, in education or 
training or gaining a qualification 
Total results at six months 
Source: https://cohesiondata.ec.europa.eu  

TO8 
(IP8ii 
excluded) 

TO9 

TO10 

Total 
(ESF & YEI) 

31 
185 
697 
1 389 
611 
2 303 
- 
1 595 

269 

106 

316 

309 
289 
365 
697 
1 202 
1 659 
- 
1 118 

167 

62 

722 

155 
792 
2 222 
210 
630 
3 380 
- 
338 

208 

10 

84 

1 864 

1 284 

546 

862 
1 695 
3 949 
3 153 
2 950 
9 660 
- 
4 105 

734 

213 

1 489 

4 839 

It is impossible to disentangle the impact of ESF funding from other factors. However, evidence 
shows  that  the  actual  contribution  of  ESF  operations  towards  the  EU  2020  targets  is  clearly 
positive,  both  from  a  micro  and  a  macro  perspective.  This  assessment  is  based  on  results 
achievements, qualitative evidence from stakeholder views and micro and macro econometric 
modelling. 

As regards employment and labour mobility support (TO8) the most outstanding result was the 
1.6  million  participants  in  employment  after  6  months,  contributing  to  the  labour  market 
integration  of  the  unemployed.  In  terms  of  investment  priorities,  access  to  employment  (8i) 
operations  made  more  progress  towards  their  result  targets  than  others,  also  according  to 
stakeholders,  followed  by  support  to  self-employment/entrepreneurship  (8iii).  This  confirms 
that  in  the  first  years  of  ESF  implementation,  most  of  the  focus  was  on  fighting  high 
unemployment  levels  and  less  attention  was  paid  to  more  structural  long-term  policies. 
Evidence  of  the  effects  of  ESF  support  shows  that  these  have  been  mostly  positive,  though 
typically  stronger  for  people  furthest  from  the  labour  market  and  for  women.  These 
quantitative  findings  are  underpinned  qualitatively  by  stakeholders’  views,  including  on  soft 
outcomes  (e.g. increased self-confidence,  motivation, active  engagement  and cooperation). In 
addition, effects are generally more positive and stronger in regions that: (i) are net receivers of 
EU support, (ii) have a larger labour supply and (iii) have a stronger export orientation. Many of 
these are  in Belgium, Spain, Italy, Poland, Portugal, Slovenia and Slovakia, as well as, to some 
extent, the United Kingdom. 

33 

 
 
 
 
 
ESF support to social inclusion has contributed to progress made towards achieving of Europe 
2020  target  of  lifting  20  million  people  out  of  poverty55.  Although  the  extent  to  which  TO9 
operations contributed to this progress cannot be directly assessed, evidence on the scale and 
type of results generated by ESF support to TO9 suggests that the contribution was positive. In 
total, more than 3 million positive results were reported in terms of engagement in job search, 
participation in education and training and accessing employment including self-employment. In 
terms  of  engagement  in  employment,  the  most  successful  operations  were  Type  1 
(employment-focused actions).  In addition to reflecting a high degree of effectiveness, these 
achievements  may  also  indicate  some  degree  of  'creaming'  effects  whereby  participants  in 
these operations were very close to the labour market from the start.  

On the other hand, although the proportion of recorded results for enhanced basic skills and 
basic school (Types 2 and 3 operations) was low, their perceived effectiveness is high. A high 
proportion of respondents to the public consultation noted that basic skills training (90%) and 
training  and  education  (89%)  were  mostly  useful  or  very  useful  in  the  promotion  of  social 
inclusion  and  in  combating  poverty  and  discrimination.  Respondents  also  perceived  the 
effectiveness of support to overcoming barriers to job search actions as high (85%)56.   

ESF support to education and training resulted in notable contributions to progress towards 
achieving  the  Europe  2020  education  and  training  targets.  Based on  the  available  data,  over 
4.34  million  participants  (46%  of  total)  achieved  positive  results  from  participation  in  ESF 
support  to  education  and  training  (50%  of  participants  with  such  results  are  women).  The 
majority  of  positive  results  related  to  gaining  a  qualification  upon  leaving  (3.3  million)  or 
engaging  in  education  and  training.  Stronger  results  success  is  seen  for  early  leavers  from 
education and training (10i) and lifelong learning (10iii), in line with the finding that Member 
States have been effective in their use of ESF funds to support national and regional strategic 
priorities  in  these  areas.  However,  many  Member  States  have  faced  challenges  related  to 
achieving positive results under ESF tertiary education (10ii) operations. The country reviews 
suggested  that  only  Bulgaria,  Hungary,  Ireland  and  Portugal  were  on  course  to  achieve  their 
output  and  result  targets  under  this  investment  priority.57  Particular  achievements  are 
highlighted in the country reviews and case studies (see them in the TO10 supporting study).  

Further  evidence  is  provided  by  stakeholder  views,  which  consider  that  ESF  operations  have 
played  a  key  role  in  helping  the  EU  reach  its  headline  targets  in  education,  contributing 
particularly to the drop in the proportion of early leavers from education and training and the 
rise  in  participation  in  adult  learning.  For  instance,  managing  authorities  intermediate  bodies 
and  public  respondents  to  the  public  consultation  considered  that  the  ESF’s  contribution  was 

55  The number of people at risk of poverty or social exclusion (AROPE) in the EU decreased from about 

122 million in 2014 to 110 million in 2018 – a decrease of about 12 million. 

56  The  total  number  of  responses  to  these  questions  was  486.  There  was  little  difference  between 

stakeholder types. 

57  Since  2014  there  has  been  a  general  improvement  in  the  proportion  of  the  population  completing 
tertiary  education  (especially  in  some  eastern  European  countries  such  as  Czechia  and  Slovakia). 
However  there  is  limited  evidence  from  the  evaluation  that  links  general  improvements  in  tertiary 
education participation with the role of ESF. 

34 

 
                                                           
either  ‘very  effective’  or  ‘mostly  effective’  in  supporting  the  development  of  work-based 
learning  systems  and  improving  access  to  lifelong  learning  opportunities  (69.2%;  N=104).  The 
responses  to  the  open-ended  public  consultation  questions  also  suggested  that  the  focus  on 
prevention and early intervention was beneficial in reducing school drop-out rates.58 

Moreover,  there  are  also  some  examples of  positive  impacts  on  early  leavers  from education 
and training and lifelong learning based on counterfactual evidence to show what would have 
happened  without  intervention  (for  example  in  Spain  and  the  United  Kingdom)59.  However 
limited impact studies have been completed to date.  

At this stage of the programming period, the ability of quantitative analysis to show the overall 
effects of ESF operations on reaching Europe 2020 targets is limited. However, it is already clear 
that  ESF  support  for  the  development  of  system  level  changes  and  sustainable  innovative 
measures  can  have  a  significant  long-term  impact.  Qualitative  evidence  (country-based 
analyses including case  studies)  suggests that ESF support to social inclusion also has  broader 
level  impacts  including  improved  access  to  public  services,  deinstitutionalisation,  and  cross-
sectoral  collaboration  to  promote  innovative  approaches.  The  ESF’s  TO10  operations  on 
education  contribute  to  reach  the  education  and  training  targets  by  supporting  systemic 
change.  This  is  highlighted  by  examples  of  operations  that  have  led  to  systemic  changes  in 
government policy approaches to education and training, showing greater potential under 10i 
(early leavers from education and training) and 10iii (lifelong learning)60.  

Finally,  the  modelling  of  ESF  impacts  based  on  RHOMOLO  simulations61  estimates  that 
operations  under  the  three  thematic  objectives  generate  positive  macroeconomic  impacts  on 
the  economy  as  a  whole,  on  top  of  the  direct  results  for  participants  in  ESF  operations62.  For 
TO8, investment in employment and mobility up to 2018 would translate into the creation of 47 
000 jobs in the long term and an increase in GDP of 0.06% compared to the baseline (2014). This 
would  be  due  to  final  macroeconomic  impacts  generated  by  human  capital  investments  and 
the  improvement  on  labour  productivity,  as  well  as  the  structural  changes  involved.  ESF 
support  to  social  inclusion  is  expected  to  increase  the  EU  GDP  by  0.037%  (which  amounts  to 

58   A  minority  of  stakeholders  replied  in  the  public  consultation  that  the  ESF  support  have  not  been 
effective at all in reducing early school leaving: 1% of managing authority respondents (1 of 104), 2% 
of beneficiary organisation respondents (5 of 227) and 5% of certifying, coordination and monitoring 
body respondents (2 of 38).  

59  See in TO10 supporting study the counterfactual positive results in both cases: contribution to reduce 
early school leavers in Galicia (Spain) operational programme and lifelong learning in West Wales and 
the Valleys operational programme (UK) – Traineeship programme. 

60  See section “5.5.5 Process effects including sustainability” for a  more detailed discussion of systemic 

change.  

61  RHOMOLO is the Commision’s spatial computable general equilibrium model, developed by the Joint 

Research Centre. See for details: https://ec.europa.eu/jrc/en/rhomolo  

62  These results describe only the macroeconomic potential impacts of ESF as it has evolved up to 2018 
and are subject to change as the programmes continue. Therefore, in order to ensure the robustness of 
results presented in this exercise, an accurate, systematic analysis with respect to the evolution of the 
ESF is required for future ex-post evaluation. The results included do not cover the  indirect impacts 
that  the  programme  has  had  throughout  society  (an  analysis  of  these  would  probably  be  nearly 
impossible  anyway,  due  to  its  complexity).  However,  it  is  clear  that  the  positive  effects  of  the  ESF 
operations extend beyond what is known and can be quantified. 

35 

 
                                                           
EUR 4 billion) by 2023 and create an additional 127 000 jobs, whereas  education and training 
operations are expected to add 0.16% to GDP (EUR 18 billion of euro) by 2023 compared to the 
baseline, and create around 170 000 additional jobs. All increases are expected to be long-term 
(until 2033), as GDP and employment are expected to still be higher relative to the baseline. At 
the same time, labour supply, i.e. the number of individuals either working or looking for a job, 
should  increase  as  well  (moving  from  inactivity  to  activity),  in  particular  as  a  result  of  ESF 
investments in employment and social inclusion (TO8 and TO9). 

Moreover, at the regional level, a considerable number of regions seem to benefit significantly 
from ESF support, in particular less developed regions and those with less favourable conditions 
in  terms  of  employment,  social  development  and  human  capital  and  education.    In  some 
regions, these positive effects can be quite significant, given the size of the policy.   

In  the  context  of  the  recent  COVID-19  pandemic  all  results  should  be  considered  with  due 
caution.  The  model  assumes  that  there  are  no  other  adverse  or  positive  shocks  affecting  the 
economy  other  than  the  ESF.  However,  COVID-19  is  already  having  a  very  strong  impact  on 
observed GDP and employment rates. Poverty and inequality are likely to rise, with the elderly, 
women  and  young  people  with  low  skills  likely  to  be  affected  the  most.  On  the  other  hand, 
RHOMOLO  estimations  show  however  that  the  expected  post-COVID-19  macroeconomic 
outcomes would be worse in the absence of ESF support63. 

5.1.3  Effects on participants 
There is evidence that ESF operations have a sustainable positive impact on participants. These 
impacts primarily relate to gains in terms of employment, education and qualifications, both in 
the immediate and longer-term (see Table 7). Although a wide range of 'soft' outcomes (such as 
soft  skills  or  increased  self-confidence)  was  also  identified  notably  for  social  inclusion 
operations, these were recorded in only a few instances, as they are not part of the common 
indicators.  

The  figures  in  Table  7  should  be  interpreted  with  considerable  caution  as  there  is  significant 
under reporting, as explained in chapter 3 - Implementation / State of play.  

Evidence  from  the  ESF  implementation  of  employment  related  operations  under  TO8 
(employment  and labour mobility)  and TO9 (social inclusion), is in line  with meta-analyses   of 
active  labour  market  policies64,  responses  received  from  the  public  consultations,  and  in  the 
case of TO8 (employment and mobility) of counterfactual evaluations conducted. 

63  For  the  purpose  of  this  exercise,  note  that  Rhomolo  simulation  results  focus  more  on  the  potential 
long-run  impact  of  ESF  support.    The  policy’s  impact  is  estimated  based  on  investments  that  took 
place up to 2018, as is detailed in the econometric analysis enclosed with this staff working document. 

64  For  example,  Butschek,  S.  &  Walter,  T.,  2014.  What  active  labour  market  programmes  work  for 
immigrants in Europe? A meta-analysis of the evaluation literature. IZA Journal of Migration (2014) 
3:48. doi  10.1186/s40176-014-0023-6;  Card,  D.,  Kluve,  J., Weber,  A.,  2015. What  Works?  A  meta-
analysis of recent active labour market program evaluations. IZA Discussion Paper No. 9236; Kluve, 
J., 2006. The effectiveness of European active labour market policy. IZA Discussion paper no. 2018; 
Vooren  et  al,  2019.  The  effectiveness  of  active  labour  market  policies:  a  meta-analysis.  Journal  of 
Economic Surveys (2019) Vol. 33, No. 1, pp. 125–149. doi: 10.1111/joes.12269. 

36 

 
                                                           
In general, in the short term, net employment effects seem to be positive for service/sanction 
type  programmes  (job  counselling,  guidance,  orientation),  whereas  for  traineeships  and  job 
incentives  they  were  stronger  in  the  medium-  to  long-term  (from  one  to  two  year  onwards). 
Public  work  typically  showed  lower  net  employment  effects.  The  effects  of  vocational  and 
educational  training  are  positive  as  long  as  this  training  is  personalised,  adjusted  to  labour 
market needs and possibly coupled with work experience. This is made easier if training offered 
to  employees  is  on  very  specific  problems  faced  by  companies  and  also  addresses  change 
management.   

The  effect  of  the  different  forms  of  support  tend  to  also  vary  based  on  target  group 
characteristics. For instance, job guidance and counselling can be very effective for low skilled 
people  and  migrants,  both  in  the  short-  and  medium-  to  long-term,  but  is  less  useful  to 
graduates. Self-employment support seems to have also a positive effect especially for the low 
skilled.  Work-based  learning  seems  to  generate  homogeneous  effects  on  the  probability  of 
employment across all skill groups in the medium- to long-term. Lighter forms of support such 
as  job counselling  can  however  be  ineffective  for  those  at  a  greater  distance  from  the  labour 
market, such as the people who are inactive, have been unemployed for more than two years or 
have multiple disadvantages. They also appear to show lower than average benefits for the high 
skilled. 

There  is evidence  that T010 (education and training) operations have  contributed to reducing 
ELET,  supporting  the  achievement  of  school  leaving  qualifications,  reducing  the  number  of 
young people at risk of becoming NEET and helping young people to achieve qualifications that 
enable them to gain access to the labour market. Operations focusing on adults with low levels 
of qualifications target both those in employment and those who are unemployed or inactive. 
Case  studies  have  highlighted  successful  examples of  ESF  operations  providing second-chance 
education opportunities for adults with low levels of qualifications. 

In  the  case  of  TO9  (social  inclusion)  the  success  rate  was  used  as  a  first  indication  of 
effectiveness.  This is the ratio of recorded successes (as  measured by result indicators) to the 
corresponding reference group (as measured by output indicators). The observed range is from 
zero to 29%. This relatively low rate can be explained by low reporting on results and the fact 
that  common  result  indicators  do  not  necessarily  correspond  to  the  desired  results,  (for 
instance  employment  outcomes  for  actions  aimed  at  parents  of  school  going  children). 
Moreover, the bulk of the operations took place under investment priority 9i (active inclusion) 
hiding, a large variety of actions with labour market, education and training or service delivery 
aims. The estimated success rates are higher when calculated for the most relevant indicator for 
each  type  of  operation  (e.g.  the  success  rate  for  employment  focused  actions  is  89%  for  the 
result  indicator  referring  to  being  in  employment  or  self-employment).  While  the  differences 
across investment  priorities and types  of actions are in line  with expectations, any analysis of 
result indicators should be treated with great caution. 

However,  the  success  of  ESF  actions  should  not  be  analysed  only  in  terms  of  employment, 
education  and  qualifications  outcomes.  Inputs  to  an  EU  level  Delphi  survey  on  thematic 
objective  9  (social  inclusion)  identified  a  number  of  areas  including  individual  behaviour  and 
social  roles  where  ESF  actions  may  have  induce  change.  The  soft  outcomes  most  commonly 
identified based on country evidence and national evaluations (e.g. in Bulgaria, Spain, Finland, 

37 

 
France, Italy, the Netherlands, Slovakia, and the United Kingdom) are soft skills, increased self-
esteem, and improved labour market prospects. More than half of the organisations involved in 
the delivery of ESF support who responded (N=295) to the public consultation on TO9 agreed 
that  participation  in  ESF  supported  actions  generated  soft-skills  (57%)  and  increased  self-
confidence (54%). Such soft outcomes are particularly important to TO9but not limited to it. A 
high proportion (41% out of 141 participants) of respondents to the public consultation on TO10 
(education and training) also felt that their self-confidence increased as a result of participation. 
The  evaluation  of  TO  8  (employment  and  mobility)  notes  that  soft-outcomes  are  quite 
widespread but underreported.  

5.1.4  Factors hindering or promoting effectiveness 
The various supporting studies identified a range of factors which affect the effectiveness of ESF 
operations as regards the delivery of support. There are broadly three groups of factors: 

(1) 

(2) 

(3) 

The  environment  in  which  the  operations take  place.  This  is  made  up  of  external 
factors   that  are  not  directly  influenced  by  the  entities  involved  in  the  delivery  of 
these operations; 
Factors  internal  to  the  management  entities.  They  relate  to  institutional  capacity, 
regulatory requirements, governance and partnerships; 
The target groups themselves. 

A  supportive  environment  is  crucial.  This  refers  both  to  how  the  operations  fit  into  a  wider 
national or regional context and to how they are perceived by the local communities in which 
they take place. It is particularly important for TO9 (social inclusion) and TO10 (education and 
training)  operations.  Shared  management  entails  that  the  actual  day-to-day  management  of 
operational programmes is entrusted to national (or regional) bodies. Operational programme 
strategies should be integrated and embedded into national (or regional) strategies if they are 
to  be  implemented  successfully.  Performance  improves  when  there  is  a  strong  strategic 
approach that informs the prioritisation and targeting of resources to address specific needs. On 
the other hand, there are several documented cases where the lack of local support or buy-in 
has  slowed  down  the  implementation  of  TO9  (social  inclusion)  measures.  For  instance,  a 
housing  programme  in  Italy  was  challenged  when  owners  remained  reluctant  to  rent  their 
property  to  vulnerable  groups  (homeless  people  and  members  of  the  LGBT  community)  even 
with  a  guarantee  of  payment.  Such  examples  show  that  there  might  be  a  need  to  envisage 
actions that build support for the proposed operations.   

The  various  supporting  studies  have  also  shown  that  institutional  capacity,  at  all  levels  of 
implementation, and regulatory requirements, are linked and cross-cutting factors influencing 
effectiveness. Managing authorities have  to lead strong coordination processes involving local 
and  regional  authorities  and  key  partners  such  as  NGOs,  civil  society  organisations  and  social 
partners.  In  turn,  beneficiaries  must  have  the  capacity  to  deliver  ESF  operations  at  the  scale 
required. They may need to rely on the managing authorities to build the capacity of their staff 
on how to access, implement and monitor the funds. Regulatory issues include those related to 
the setting up and operation of the monitoring systems, the definition of simplified cost options, 
duplications  of  controls  and  compliance  with  state  aid  regulations.  Responses  to  the  public 
consultation  tend  to  confirm  that  these  regulatory  issues  can  place  an  important  burden  on 
beneficiaries and managing authorities in the implementation of ESF supported actions. 

38 

 
For example, respondents to the  TO8 public consultation on employment and labour mobility 
were asked about factors that may hinder the implementation of actions. Approximately three 
quarters  of  them  (76%  of  238)  agreed  that  the  administrative  burden  on  beneficiaries  and 
managing  authorities  hampers  progress.  In  addition,  external  factors,  such  as  structural 
problems (i.e. lack of employment opportunities and low education and skills level) also play an 
important  role  in  respondents’  views  (for  71.3%  of  238  respondents).  Some  organisations 
specified  the  obstacles  types  they  had  encountered  in  a  related  open  question.  Out  of  80 
responses,  33  indicated  cumbersome  bureaucracy  and  lengthy  procedures  as  the  most 
important  factors  hampering  the  effectiveness  of  ESF  procedures,  which  is  aligned  with  the 
responses given to the closed question.  

The  public  consultation  for  TO9  (social  inclusion)  asked  the  organisations  involved  in  the 
management  and  delivery  of  ESF  support  to  qualify  the  administrative  arrangements  for 
implementing  operations.  The  results  suggest  that  the  most  effective  administrative 
arrangement  of  all  is  communication,  with  71%  of  respondents  rating  it  as  appropriate. 
Opinions  on  the  management  and  control  system  were,  however,  mixed,  with  45%  of 
respondents believing that it is effective and 41% considering it burdensome. 

Figure 11. How would you qualify the following administrative arrangements for the 
implementation of operations? 

N= 295 Source: TO9 supporting study report annex 7  -
https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes   

Partnerships  between  managing  authorities,  beneficiary  organisations  and  other  actors 
improve  effectiveness.  A  high  level  of  cooperation  is  especially  important  in  the  delivery  of 
personalised operations or integrated support. This success factor was highlighted in the public 

39 

 
 
consultations  as  well  as  the  country-based  analyses  of  several  Member  States  across  all 
supporting studies. Local actors, civil society organisations and NGOs, often play a crucial role in 
engaging with harder to reach target groups.  

The correct alignment of operations with the needs of the target group is also a success factor. 
An  econometric  analysis  of  TO8  (employment  and  mobility)  data  showed  that  the  socio-
economic  context  and  distance  of  the  target  groups  from  the  labour  market  might  affect  the 
‘gross’  results  of  the  operations  with  an  employment  objective.  Combined  with  a  focus  on 
targets  or  results,  this  may  lead  to  ‘creaming’.  This  finding  also  applies  to  TO9  and  TO10 
operations.  However,  it  is  the  form  and  suitability  of  the  support  offered  to  participants  that 
determines  its  net  effects,  irrespective  of  where  it  is  implemented  and  for  whom.  The 
overshooting of targets and high success rates across all ESF operations indicates that creaming 
does indeed occur. 

The precise identification of a target group makes it easier to take into account the additional 
efforts  required  to  reach  it,  as  opposed  to  those  people  for  whom  a  result  can  be  generated 
more  easily. The  econometric analysis of monitoring data for TO8 (employment  and mobility) 
suggests  that  operations  targeting  needs  that  are  comparatively  widespread  and  constant 
among the population tend to proceed at a faster pace, especially where there are no obstacles 
to participation such as disabilities or multiple disadvantages. 

Similarly, the evaluation of ESF support to TO10 showed that all Member States identified target 
groups at the design stage of operations. This is largely thanks to the programming process of 
the  European  Structural  and  Investment  Funds,  which  ensured  that  needs  assessments  were 
conducted by all Member States in the partnership agreement and reflected in the operational 
programmes.   

At a general level, 143 of the 145 operational programmes that planned for ESF TO9 operations 
were found to be consistent with the country specific recommendations (CSRs). The CSRs from 
2014 to 2019 were reviewed more closely to identify relevant recommendations for ESF support 
to social inclusion.  

In  their  replies  to  the  public  consultation,  beneficiary  organisations  and  managing  authorities 
also underlined the  importance  of knowing and understanding the specific circumstances  and 
needs of the target group to successfully deliver ESF support. However, the precise alignment of 
operations  to  the  needs  of  the  target  group  does  not  need  to  take  place  in  the  operational 
programme  itself.   The  broad  definition  of the  target  group  in  some  operational  programmes 
has allowed for flexibility in implementation, for instance in reacting to the refugee crisis (e.g. 
Belgium, Austria).   

5.2  Efficiency 

5.2.1  Assessment of efficiency 
At  this  stage  of  implementation,  efficiency  can  only  be  assessed  based  on  the  cost  per 
participation. Cost per result would be a more appropriate measure, but it cannot be used yet 
because of delayed and non-simultaneous reporting on results and costs.  

40 

 
Table 8. Cost per participation 

TO9 Social 
inclusion 
1,441 

TO10 education and 
training 
1,137 

TO8 Employment and 
mobility 
1,309 

Cost per participation  
(in EUR) 
average 
Range across investment 
priority 
Range across Member 
States 
2007 – 2013 Ex-post 
update* 
*The 2007-2013 estimate was calculated ex post at the end of the period. The current estimates for 2014-
2020 are based on end-of-2018 data and are affected by delays in reporting. 

581 – 3,048 

326 – 7,493 

343 – 6,791 

729 - 2,754 

142 - 1,935 

440 - 8,129 

1,215 

1,685 

739 

The  cost  per  participation  is  the  ratio  of  declared  expenditure  to  the  reported  number  of 
participations.  The  observed  cost  per  participation  (see  Table  8)  varies  widely,  reflecting  the 
variety of operation types implemented, the different needs of target groups and the range of 
contexts  in  which  operations  are  implemented.  The  estimates  for  TO8  (employment  and 
mobility)  and  TO9  (social  inclusion)  are  roughly  in  line  with  the  unit  costs  reported  in  the 
supporting  study  the  update  of  data  reported  in  the  2007-2013  ESF  ex-post  evaluation65.  The 
estimate for TO10 (education and training) is 50% more. This is essentially thought to reflect: (i) 
a bigger emphasis on more costly operations, indicating a wider use of individual grants, and (ii) 
high quality and longer schemes supporting higher education and training. Moreover, in 2007-
2013,  the  reporting  of  indirect  participants  was  more  widespread  (e.g.  where  a  school  or 
curriculum was directly targeted, the number of pupils was also reported). 

Overall cost-benefit analyses carried out for TO9 (social inclusion)66  and TO10 (education and 
training)67    suggest  positive  returns  particularly  under  9i  (active  inclusion),  and  significant 
private and public returns for 10i (early leavers from education and training) and 10ii (tertiary 
education). However a note of caution should be attached to the results as a number of broad 
assumptions needed to be applied, given the limitations of the monitoring data and the limited 
impact evidence. A detailed cost benefit analysis identified positive returns for four out of five 
projects investigated under TO 9 (social inclusion). The net benefits of TO9 are even higher if all 
outcomes, in particular including 'soft outcomes' are considered.   

5.2.2  Factors hindering or promoting efficiency 
As regards TO8 operations, actual and perceived administrative burden has hampered effective 
implementation,  through  delays,  and  has  also  impacted  cost-effectiveness.  New  monitoring 
systems  and  databases  were  developed  at  Member  State  level  to  comply  with  regulatory 
changes  since  the  2007-2013  programming  period.  The  most  significant  factor  is  insufficient 
administrative  capacity,  which  in  turn  affects  implementation.  Furthermore,  additional  costs 

65  https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8330&furtherPubs=yes   

66  https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes  

67  https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8346&furtherPubs=yes  

41 

 
                                                           
have been incurred for recruiting and delivering ESF to harder-to-reach groups, especially those 
in remote/rural areas. 

As regards TO9, there were mixed views on whether the application of simplified cost options 
reduced  administrative  burden  or  not.  The 
implementation  of  mono-fund  operational 
programmes  appeared  to  be  more  efficient  than  multi-fund  operational  programmes.  Gold 
plating was identified in 13 Member States68.  

 TO10  operations,  gold  plating  was  also  frequently  highlighted  by  national  stakeholders  as 
creating additional administrative barriers for ESF project managers and potential beneficiaries. 
The  particular  challenges  in  managing  ESF  TO10  operations  highlighted  in  the  case  studies 
included the following: 

• 

• 

• 

• 
• 

high  number  of  stakeholders  in  implementing  ESF  operations  which  requires  a 
strong coordination capacity to ensure efficient delivery; 
lack  of  cooperation  between  national  governmental  ministries  in  the  delivery  of 
operations under the same investment priority;  
insufficient communication and information exchange between project managers in 
the implementing organisations.  
lack of mutual coordination in aligning individual education policy actions;  
lack of consistency in implementing operations (for example in cases some activities 
are carried out in parallel and  should take place in successive stages). 

Overall,  most  organisations  that  responded  to  the  TO10  public  consultation  and  most 
stakeholders believe that, on the whole, the ESF administrative and regulatory requirements are 
proportionate  to  the  benefits  achieved;  most  managing  authorities  and  intermediary  bodies 
that  responded  (62.3%,  i.e.  65  out  of  104)  judged  the  listed  arrangements  to  be  appropriate. 
However, on average, more than 20% (23 out of 104) of respondents rated the arrangements to 
be  burdensome,  while  around  8%  (8  of  104)  judged  them  as  insufficient.  Most  respondents 
from  beneficiary  organisations  (59.5%,  i.e.  135  out  of  227)  judged  the  listed  arrangements  as 
appropriate. However, more than 22% (50 out of 227) of beneficiary organisations rated them 
to  be  burdensome,  while  6%  (13  out  of  227)  perceived  them  as  insufficient.  Moreover, 
interviewed stakeholders from countries with smaller ESF allocations felt that the requirements 
would be too onerous. The evaluation research suggests that administrative requirements can 
discourage smaller beneficiaries such as NGOs, which are crucial for reaching vulnerable groups, 
from accessing ESF funds.   

5.2.3  Visibility of ESF interventions 
Regarding  the  visibility  of  ESF  operations,  there  is  room  for  improvement.  In  particular, 
evidence  of  the  visibility  of  TO8  operations  remains  thin,  even  if  there  are  several  good 
examples of effective communication. Out of 541 respondents to the TO8 public consultation, 
70.4% seem to consider social media campaigns as the best channel of information, followed at 
a  distance  by  information  and  awareness  raising  events  (44.5%).  The  least  popular  channel 
seem  to  be  the  distribution  of  flyers  and  placement  of  advertisements  in  newspapers. 

68  Gold-plating  describes  additional  rules  and  regulatory  obligations  that  go  beyond  the  European   
Structural    and   Investment    Funds (ESIF)  requirements set out at EU  level, and   that   make  the 
implementation  of ESIF more  costly  and  burdensome for  programme bodies and beneficiaries. 

42 

 
                                                           
Interestingly, the channels of information actually encountered by surveyed ESF participants are 
slightly  different:  most  respondents  say  that  they  learned  about  the  support  measures  they 
benefited  from  through  employment  services  and  employment  information  centres  (43.4%), 
and less than a fifth through social media.  

,  a  significant  proportion  of  stakeholders  and  potential  participants  were  not  aware  of  or 
familiar with the ESF at all. Although many Member States invested in raising awareness of the 
ESF  among  beneficiaries,  target  groups  and  the  general  population,  the  level  of  awareness 
remains mixed. The picture is somewhat better for TO10, where there was positive evidence of 
the  visibility of  ESF funded  operations:  responses  to the  public  consultation  suggested  a  good 
level of the ESF’s visibility among organisations without a direct role in managing and delivering 
ESF programmes (only 12.9%, i.e 8 out of 62- organisations reported never hearing of the ESF 
before  the  survey).  Programme  managers  were  familiar  with  the  programme’s  visibility  rules 
and claimed to be applying them. Beneficiaries are aware of visibility rules as this is part of their 
contractual  obligations  and  intermediary  organisations  have  played  an  important  role  as 
multipliers in maximising the visibility of TO10 operations among beneficiary organisations69. 

5.3  Relevance 

5.3.1  Alignment with needs 
This  evaluation  criterion  examines  the  extent  to  which  ESF  operations  were  relevant  for 
addressing policy needs and the needs of target groups requiring support, at the programming 
and  implementation  levels.  Overall,  evidence  from  the  three  supporting  studies  shows  that 
there  is  a  high  degree  of  alignment  of  planned  ESF  operations  under  the  three  thematic 
objectives with employment, social inclusion and education and training needs. The evidence 
also shows  that ESF operations have  in general been  relevant for the needs  of target groups 
though  there  is  still  room  for  improvement  in  this  area.  This  could  be  achieved  by  better 
involving different actors in both the design and monitoring stages, stakeholder involvement is 
key to meeting target group needs – particularly those of the most disadvantaged groups. 

A)  At  the  planning  stage,  evidence  shows  that  the  most  important  needs  of  the  main  target 
groups of each ESF thematic objective were taken on board to a great extent. This is largely part 
thanks  to  the  programming  process  of  the  European  Structural  and  Investment  Funds  which 
ensured  that  needs  assessments  were  conducted  by  all  Member  States  in  the  partnership 
agreement and reflected in the operational programmes. The broad scope in the design of the 
investment  priorities,  covering  a  wide  range  of  policy,  areas  also  contributed  to  ensuring 
relevance at the strategic level. 

For  instance,  the  evaluation  findings  show  that  TO8  operations  on  employment  and  labour 
mobility, contributed to helping unemployed people join the labour market. In response to the 
high  level  of  unemployment  across  the  EU  at  the  start  of  the  programming  period  most 
attention  was  given  to  (immediate)  employment  measures,  such  as  providing  individual 
guidance to job seekers, integrated approaches, hiring incentives for employers, and supporting 
apprenticeship models. Less attention was dedicated to more structural employment measures, 

69  As 

the 
https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8346&furtherPubs=yes  

supporting 

detailed 

TO10 

in 

study. 

43 

 
                                                           
such  as  gender  equality,  active  ageing  or  labour  market  institutions.  Moreover,  analysis  of 
regional allocation of expenditure suggests that TO8 funding was typically directed to where it 
was most needed. This means that regions with the highest unemployment rates received not 
only the bulk of funding but also the largest share of expenditure with respect to their GDP. 

Box 1. Identifying the target group needs for social inclusion operations 

Social  inclusion  (TO9)  operations were  highly  relevant  given  that  target  groups  were  typically 
identified  at  the  planning  stage,  through  needs  assessments  and  existing  evaluations  and 
consultative processes, which promoted buy-in and helped ensure that objectives were realistic 
and adapted to the socio-economic context. 
For  instance  in  Austria,  target  groups  were  identified  through  consultations  between  the 
Managing  Authority,  the  intermediate  bodies  at  the  federal  and  regional  level,  the  social 
partners,  NGOs,  and  other  stakeholders.  In  Lithuania,  the  specific  needs  of  the  ageing 
population  were  reviewed  during  the  planning  stage  through  workshops  with 
local 
stakeholders;  TO9  actions  were  tailored  accordingly.  Delphi  survey  respondents  also 
underscored the importance of identifying the target groups in the early stages of the project to 
promote relevance. 

As regards education and training, all Member States identified the target groups at the design 
stage of operations. The design of investment priorities covered a wide range of education and 
training  objectives  and  types  of  intervention  across  all  education  levels  throughout  the  life-
cycle,  allowing  Member  States  to  tackle  diverse  education  and  training  needs  in  different 
Member States or regions. This programming flexibility has been widely praised by national and 
EU stakeholders alike for increasing the relevance of planned operations to the needs of target 
groups. 

B) During the implementation stage, ESF operations succeeded in reaching the most important 
target groups of each thematic objective and addressed their needs. This was demonstrated by 
participation  indicators  and  confirmed  by  stakeholder  views  and  other  evaluation  evidence, 
although some challenges were identified for specific target groups. 

The  objectives  and  operations  funded  by  ESF  support  to  employment  and  labour  mobility 
(TO8) were mostly aligned with the needs of the target groups addressed. Participation figures 
confirm  the  dominance  of  operations  targeting  the  unemployed.  Until  December  2018,  the 
number of participations of unemployed people across all programmes amounted to some 4.6 
million  compared  to  0.5  million  of  inactive  persons,  and  1.7  million  of  employed  people. 
According  to  organisations  that  responded  to  the  public  consultation,  job-seekers,  inactive 
people and the long-term unemployed were the most successfully supported groups. In-depth 
analysis  of  operational  programmes  and  country  case  studies  provide  evidence  that  Member 
States  have  developed  operations  that  are  relevant  to  the  needs  of  the  unemployed, 
entrepreneurs, women, employees, older workers and labour market institutions. 

As regards social inclusion, most of the participations recorded under TO9 involved people who 
were unemployed (53%) and had a low education level - primary or lower secondary - (54%). A 
large proportion of participations were from people with a foreign background or from minority 
groups  including  Roma  (28%)  and  people  with  a  disability  (16%).  The  reported  proportion  of 
people living in rural areas (16%) is far below the almost one third of the EU28 population living 

44 

 
 
in rural areas - see Table 5 and Figure 8. Several social inclusion operational programmes carried 
out  gender-based  targeting,  (e.g.  women  not  engaged  in  the  labour  market  were  targeted 
Ireland,  lone  immigrant  women  were  targeted  in  Finland  and  Roma  women  were  targeted  in 
Spain).  Moreover,  TO9  funds  were  mostly  allocated  to  less  economically  favourable  regions, 
thus addressing those geographic areas in which most people in need are living; about 68% of 
recorded participations were in less economically favourable regions. 

Education  and  training  operations  succeeded  in  reaching  and  addressing  the  needs  of  their 
target  groups,  particularly  of  young  people,  a  key  target  group  for  TO10:  approximately  6.26 
million  participations  were  by  people  under  25  years  old  (66%  of  the  total  TO10 
participations)70. Young people were the main focus of operations under investments priorities 
10i  (early  leavers  from  education  and  training  and  access  to  learning  pathways)  and  10ii 
(tertiary  education).  Evidence  shows  that  ESF  TO10  operations  made  good  progress  in 
addressing the needs of young people71. This contributes to reducing the share of early leavers 
from education and training, supporting the achievement of school qualifications, reducing the 
number  of  young  people  at  risk  of  becoming  NEET  and  helping  young  people  to  get 
qualifications that enable them to gain access to the labour market.  

Box 2. Ensuring the relevance of early leavers from education and training operations through 
risk assessment 

Planning risk assessments at the start of operations that target disadvantaged student groups, 
has allowed for more tailored support that directly addresses people’s needs. For example, in 
Latvia,  a  risk  assessment  methodology  was  developed  as  part  of  the  programme  supporting 
students  at  risk  of  early  school  leaving  in  Latvia.  This  made  it  possible  to  identify  people’s 
specific  needs  and  their  circumstances  from  the  start,  which  is  important  as  these  vary  even 
within  the  same  target  group.  Some  at-risk  students,  in  particular  those  with  learning 
difficulties, needed extra classes, while others, such as those from low-income families, needed 
transport  and  accommodation  compensation  instead.  The  risk  assessment  ensured  that  the 
programme to be tailored to these specific needs, increasing its relevance. 

C) However the evidence shows that ESF operations did not always fully succeed on reaching 
and addressing the needs of some specific target groups, particularly the most disadvantaged 
ones. This is discussed in more details below 

TO8  employment  operations  have  managed  to  reach  participants  in  high  need  of  support, 
including  inactive  people,  the  long-term  unemployed  and  migrants.  Despite  some  reported 
issues with eligibility criteria including as a result of gold-plating (e.g. for migrants) one in eight 
participant  has  a  migrant  or  minority  background.  Over  one  in  four  participations  registered 
comes from a rural background. This is less than the proportion of the rural population in the 
EU28 but there is considerable evidence of underreporting in this area. However, there are still 
some  challenges  to  address,  as  disadvantaged  groups  are  particularly  targeted  in  more 
developed regions, and in regions in which labour market conditions are improving and which 

70  People  in  employment  and  unemployment  account  for  a  relatively  small  proportion  of  the  overall 

number of participations under ESF TO10 (most participants are classified as inactive). 

71  As highlighted in the effectiveness section 

45 

 
 
                                                           
have  progressively  increased  their  focus  on  individuals  at  a  greater  distance  from  the  labour 
market. Less developed areas seldom report positive results for disadvantaged. Organisations 
that  responded  to  the  public  consultation  seem  to  moderately  agree  on  the  ESF’s  ability  to 
support migrants (49.2%) and individuals with a foreign background or people in remote areas 
(48.9%) and highly disagree about TO8 ESF support to people affected by poverty (39.1%) and 
homeless  people  (17.4%).  Moreover,  operations  have  been  less  effective  in  addressing  the 
needs of older workers (participants aged over 54) and gender equality, with limited focus on 
measures for active ageing and those that address strongly embedded gender gaps. 

Compared with ESF support under TO8 and TO10, support for social inclusion (TO9) reached a 
substantially  higher  proportion  of  people  with  a  disability,  migrants,  people  with  a  foreign 
background  and  minorities,  as  well  as  people  with  low  education72    -  see  Table  5.  However, 
evidence on whether ESF support for social inclusion reached the most vulnerable populations 
with the greatest needs is mixed. In the replies to the public consultation, 39% of respondents 
considered that the target groups that should be prioritised were in effect being reached while 
35%  believed  the  opposite.  Respondents  to  the  EU-level  Delphi  survey  noted  that  reaching 
target  groups  outside  the  reach  of  social  services,  employment  services,  health  care  and 
education  is  a  key  challenge.  Among  other  factors  (see  below),  the  assessment  identified  the 
risk  of  ‘creaming’  in  TO9  operations,  i.e.  targeting  less  vulnerable  people  with  less  complex 
needs who can get better results. For example in Poland, the managing authority of the national 
ESF operational programme believed that the focus on monitoring employment results created 
a tendency to recruit participants who were more likely to become employed rather than the 
people furthest away from the labour market. 

Education  and  training  operations  have  faced  the  biggest  challenges 
in  reaching 
disadvantaged  groups,  registering  lower  participation  rates  for  disadvantaged  people 
compared with other ESF operations (TO8 and TO9): Overall, the proportion of disadvantaged 
participants  in  TO10  is  21%,  below  the  proportion  of  disadvantaged  participants  in  all  ESF 
operations  (31%).  Multiple  stakeholders  at  the  EU  and  national  level  identified  refugees  and 
migrants,  Roma, the  long-term unemployed, older workers and people with disabilities as the 
most challenging groups to engage across all investment priorities. The responses to the public 
consultation  (see  Annex  2)  provide  evidence  of  TO10’s  overall  success  in  supporting  specific 
target  groups  but  confirm  that  Roma  and  other  minorities  were  often  not  seen  to  be 
successfully supported through the ESF.  

Moreover,  improving  access  to  higher  education  for  vulnerable  groups  is  a  big  challenge. 
Disadvantaged  groups  have  been  hard  to  reach  for  ESF  TO10  tertiary  sector  operations  (10ii) 
because of difficulties in developing collaborative relationships between the tertiary sector and 
schools, hampering the identification of suitable  groups that would widen the diversity of the 
student body. Reaching and engaging NEETs is another challenge for ESF operations supporting 
education,  as  institutions  have  limited  resources  to  implement  outreach  activities.  There  are 
some notable exceptions, as some operational programmes highlight NEETs as a priority target 
group,  e.g.  West  Wales  and  Valleys  in  the  United  Kingdom,  where  TO10  operations  aim  to 

72  However,  ESF  support  for  social  inclusion  reached  a  lower  proportion  of  people  in  rural  areas  than 

TO8 and TO10. 

46 

 
                                                           
reduce the number of pupils at risk of becoming NEET, and the Portuguese national operational 
programme. In fact, identifying marginalised young people was often seen as less of a challenge 
than getting them to take part in operations.  

Particular  reasons  why  reaching  disadvantaged  groups  is  a  challenge  vary  by  country, 
investment priority and type of operation. Common challenges include: 

1)  eligibility  criteria  set  for  participation  (e.g.  in  certain  countries  individuals  must  be 
registered  as  unemployed before  being  eligible  for  ESF  operations, making operations 
inaccessible  to  the  inactive;  a  similar  obstacle  was  mentioned  in  relation  to  Roma 
individuals, who may not have identity papers);  

2) administrative requirements for beneficiaries discourage smaller NGOs with stronger 
links to harder-to-reach groups;  

3) a need for more integrated and follow-up actions; 

4) additional funding required to target hard-to-reach groups;  

5) higher risk of targeting operations to vulnerable groups in terms of achievement of 
targets (leading to  ‘creaming’, as explained above). 

5.3.2  Observed factors that improve relevance 
Individual-centred  approaches  based  on  specific  target  groups  needs  were  considered 
especially relevant for all thematic objectives. For instance, for TO8, a German operation in the 
federal  operational  programme,  focused  on  difficult-to-reach  target  groups  and,  offered 
counselling - under the heading of gender equality measures - based on a tailor-made package 
of measures. Similarly, in Finland, personal guidance and a needs assessment were the starting 
points,  which  ensured  that  actions  met  local  beneficiary  needs.  For  TO10,  the  education 
stakeholders  interviewed  felt  that  interventions  focusing  on  NEETs  require  more  integrated 
operations and follow-up activities to improve their effectiveness and sustainability. 

Adequate  and  effective  involvement  of  a  wide  spectrum  of  key  partners  throughout  ESF 
programming  and  implementation  is  key  to  aligning  operations  with  the  specific  needs  of 
target groups and local context, improve buy-in and ensure that the objectives are realistic.  

Whilst  the  partnership  principle73  was  thoroughly  implemented  in  all  Member  States  at 
planning  level  on  paper  (i.e.  in  the  drafting  of  the  partnership  agreements  and  operational 
programmes),  most  stakeholders  consulted  expressed  concerns  about  the 
insufficient 
involvement  and  diversification  of  stakeholders.  For  instance,  in  education  ESF  operations, 
public authorities (national and regional levels), are generally overrepresented at the expense of 
education  and  training  institutions,  local  authorities,  civil  society  and  social  and  economic 

73  Article  5  of  the  Common  Provision  Regulation  (CPR)  requires  Member  State  to  organise  for  each 
ESIF fund programme a partnership with the competent regional, local authorities and other relevant 
stakeholders,  at  all  programming  and  implementation  stages  and  at  all  levels,  in  line  with  its 
institutional and legal framework. 

47 

 
                                                           
partners74. Case  studies in ESF support to employment showed that the cooperation between 
new  partners  at  regional  level  made  it  possible  to  develop  innovative  ideas  in  response  to 
specific  topics  and  changing  needs.  For  social  inclusion  operations,  overall,  the  different 
partners  were  highly  involved  in  the  programming  phase  and  to  some  extent  during 
implementation; for multi-fund operational programmes, however, the implementation of the 
partnerships  and  mobilisation  of  partners  was  identified  as  a  challenge,  possibly  due  to  the 
programme’s  complexity.  Delphi  survey  respondents  also  underlined  the  importance  of 
identifying the target groups in the early stages of a project to improve relevance. Finally, the 
capacity of key ESF stakeholders needs to be increased, to allow them to better contribute to 
the  design,  implementation  and  monitoring  of  ESF  operations  and  thus  improve  relevance  to 
target group needs. 

The use of an integrated approach, combining several operations at the same time (depending 
on needs) is particularly relevant for operations related employment, social inclusion and early 
leavers from education and training. 

Box 3. Integrated approaches for ensuring relevance of support to employment 

In  most  support  to  employment  operations,  particularly  in  investment  priority  8i,  different 
activities for individuals are combined in an integrated way (guidance, training, accreditation of 
prior  learning,  job  searching  and  matching  etc.).  This  ensures  that  the  ESF  does  not  support 
isolated operations, but those that feed into an individual, tailor-made plan for participants. In 
some cases, the supply side (increasing the qualifications and skills of job seekers) and demand 
side (incentives to hire job seekers or provide work experience places) measures are combined. 
The supporting studies provide many examples: 
• 

In the employment-related operations of the Campania operational programme in Italy, 
needs  were  addressed  by  pursuing  different  types  of  measures  related  to  recipients’ 
profiles  and  characteristics  (pathways  combining  guidance,  counselling,  upskilling  and 
reskilling training and, company incentives for the long term unemployed).  
In  Germany,  the  national  operational  programme  combined  operations  targeted  at 
SMEs,  individuals  and  intermediaries  (social  partners),  and  strengthened  regional 
structures.   
In  France,  operations 
integrated  several  areas  to  remove  obstacles  hindering 
integration  in  the  labour  market  (acquiring  relevant  skills,  guidance  for  elaborating  a 
career plan, etc.). 

• 

• 

Integrated approaches are particularly key to addressing the needs of disadvantaged groups; in 
social exclusion operations, actions were tailored to target groups by taking a holistic approach 
and  adapting  to  their  multiple  needs.  For  example  in  the  Italian  National  operational 
programme on social inclusion, actions to tackle extreme social exclusion amongst adults sought 
to  address  the  needs  of  people  experiencing  housing  exclusion.  However,  this  integrated 
approach  was  missing  from  some  operations,  for  example  in  the  Romanian  human  capital 
operational  programme,  and  Italy’s  Puglia  operational  programme,  where  the  operation  was 
weakened because of a too narrow focus on training.  

74  Non-public  partners  such  as  training  institutions/vocational  schools,  employers  and  entrepreneurs, 
NGOs, civil society appear to not have been appropriately consulted, and when they were consulted, it 
was not in a timely manner, prohibiting meaningful contribution. 

48 

 
 
                                                           
5.3.3  Flexibility in a changing context 
As regarding TO8 operations, the operational programmes have proved to be flexible enough to 
allow  changes  throughout  the  programming  period,  which  ensured  that  they  were  able  to 
respond to changing needs.  

For  most  programmes,  objectives,  operations  types  and  intended  target  groups  were  defined 
relatively broadly in. This provided the flexibility to respond to changes in the implementation 
context  without  additional  procedures.  Formal  amendments  of  operational  programmes  for 
formal  changes  in  allocated  budgets  and  revisions  to  indicators,  were  also  common.  These 
adjustments were tracked and show different patterns across regions, in line with the changing 
situation of employment and labour mobility (see Box 4).  

The  case  studies  gives  ample  evidence  of  how  programmes  were  able  to  flexibly  respond  to 
changing socio-economic contexts.  

Box 4. Examples of flexibility 

 

In the German national operational programme, for instance, the increasing importance of digital skills 
for  employment  received  additional  attention  in  the  last  calls  for  proposals,  within  the  broader 
programmed  framework  of  skills  for  employment.  This  involved  a  slight  modification  of  calls  for 
proposals, which continued to fit within the broader framework set by the operational programme.  

  Another  example  is  Denmark,  where  the  improved  economic  situation  reduced  the  possibilities  for 
firms to allow staff to participate in long-term skill upgrading that was initially planned. With lower 
demand  for  long  term  operations  than  expected,  additional  operations  were  set  up  to  support briefer 
training sessions focusing on more specialised skills. 

 

 

In  Spain,  particularly  in  regions  where  the  unemployment  rate  improved  substantially,  existing 
employment operations were reformulated to benefit not only the unemployed, but also people with a 
precarious or temporary job situation.  

In  response  to  the  rapidly  improving  youth  employment  rates  since  2014,  Czechia  was  able  to  re-
balance its operational programmes to focus  more  specifically on older unemployed  workers,  whose 
unemployment rates were not improving at the same rate. 

Managing authorities considered that TO9 operations allowed for sufficient flexibility to adapt 
to changes  in context, including the  migration crisis that started around 2015. Changes  in the 
operation types that were implemented compared with what had been planned occurred in at 
least 10 Member States. Shifts in target groups reached by TO9 operations were identified in 13 
Member States. 

Most Member States already identified priority target groups at the planning stage, but others 
started with a less precise target group definition, which was further focused and disaggregated 
during  implementation.  This  flexibility  may  have  helped  TO9  operations  to  reach  more  target 
groups than originally planned. 

Similarly,  operations  planned  under  TO10  remained  relevant  throughout  the  programming 
period,  despite  changes  in  the  socio-economic  context.  Evidence  shows  that  where  changes 
were needed to ensure continued relevance - for example when national/regional needs and/or 
political  priorities  changed,  operational  programmes  were  able  to  adequately  adapt.  Notable 

49 

 
 
 
 
 
examples  of  operational  programmes  that  were  successfully  reprogrammed  to  respond  to 
emerging needs and adapt to new realities: 

• 

• 

In Ireland, it became harder to engage unemployed people in training programmes 
due to the declining unemployment rate. The operational programme was adapted 
to address this, ESF funds for training for the unemployed (10ii) were reallocated to 
operations to promote adult literacy under 10iii.  
In Portugal, the human capital operational programme was reprogrammed in 2018 
to allocate more funding for operations targeting low-skilled adults, which had been 
less of a priority in initial programming given the strong focus on measures for the 
unemployed at the start of the period.  

There  were  very  few  examples  of  the  ESF  being  unable  to  adapt  to  a  significantly  altered 
implementation or political context. One such example comes from the United Kingdom, where 
the  adult  participation  rate  in  lifelong  learning  fell  slowly  but  consistently  between  2013  and 
2018 with the 2018 rate higher than the EU average, yet ESF TO10 actions were not adjusted to 
address this.  

Box 5. Preliminary reflections on the ESF’s possible role in addressing COVID 19 crisis 

When the COVID-19 pandemic started, the field work for the supporting studies was completed 
or about to be completed. The time frame covered, 2014-2018, excludes the pandemic and the 
subsequent crisis. Nevertheless, a number of lessons learned show that ESF operations are still 
highly relevant in the new COVID-19 context. 

As in any crisis, Europe’s more vulnerable groups suffer disproportionally. This is because they 
are  less  well  connected  to  internet,  their  jobs  tend  not  to  lend  themselves  to  tele-working, 
online learning is not accessible to their children, etc.  

Vulnerable groups have complex needs and very often the hardest to reach. Strong cooperation 
and  coordination  is  therefore  especially  important  when  deciding,  designing  and  delivering 
targeted operations or support that are known to be more efficient in addressing these needs. 
This, in turn, requires high levels of inclusive stakeholder involvement and partnerships.  

Unemployment  levels  are  already  rising  as  a  consequence  of  the  COVID-19  pandemic.  This  is 
only likely to worsen as the economic impact of the crisis unfolds. Changes to the economy may 
prove to be long terms. In this context, lifelong learning and training will be crucial in preparing 
people  to  re-enter  the  labour  market  and  adapt  to  new  working  arrangements  and  the 
economy’s changing. Issues related  to the provision of guidance  and counselling also plays an 
important role. 

5.4  Coherence 

5.4.1  ESF Internal coherence 
ESF  operations  show  high  levels  of  complementarity  and  coherence  with  each  other  across 
different investment priorities, both within thematic objective and across the different thematic 
objectives 8, 9 and 10. 

50 

 
 
 
 
 
Internal coherence within thematic objectives 

5.4.1.1 
ESF  support  to  employment  operations  is  complementary  and  coherent  across  investment 
priorities, with operations either indirectly or directly contributing to one another’s objectives 
of and/or complementing each other. The definition of the scope of each investment priority 
maximises  complementarity  and  synergy  among  all  actions  contributing  towards  the  general 
objective of improving employment. At the same time, some flexibility is allowed as regard the 
operations  that  can  be  funded.    For  instance,  there  is  complementarity  between  investment 
priority  8iv  for  activities  that  promote  equality  between  men  and  women  in  all  areas  and 
investment priority 8i on access to employment.  

However, there is room to improvement coherence to avoid overlap and maximise synergies, 
particularly:  a)  when  operations  cover  horizontal  themes  such  as  social  innovation,  and  b) 
when  national  and  regional  operational  programmes  operate  on  the  same  territories.  As 
regards point a), given the relatively flexible boundaries demarcating TO8 investment priorities, 
it  is  possible  for  operations  funded  under  different  programmes  and  strands  to  focus  on  the 
same objectives and target groups (e.g. social innovation operations, the analysis of Germany’s 
operations and Spain’ show). These operations are cross-sectoral by their very nature and thus 
an additional effort is required to fine-tune them so that they meet the needs and objectives of 
thematic  support.  As  regards  point  b)  overlap  can  happen  when  national  and  regional 
operational programmes contribute to similar objectives, such as in the case of Italy, France and 
Spain. For instance, in Italy, although the scope of the national operational programme is more 
systemic, similar operations are being implemented at national and regional level, for example 
to support the public employment services or provide employment incentives. 

As  regards  social  inclusion,  by  addressing  the  multiple  drivers  of  social  exclusion  and 
discrimination, ESF operations complement each other in helping vulnerable groups towards 
social inclusion. ESF support takes holistic approach by combining different intervention types. 
Operation  types  1-4  focus  on  groups  of  individuals  facing  social  exclusion  and  discrimination. 
Types  5  and  6  focus  on  the  demand  side  (e.g.  the  capacity  of  organisations  to  adequately 
support  socially  excluded  populations).  In  sum,  about  two  thirds  of  operational  programmes 
working  on  social  inclusion  combine  both  operation  types,  targeting  both  individuals  and 
entities75. For example, in Sweden, ESF support to social inclusion included operation types 1, 4 
and  6.  In  Spain  (under  the  Madrid  operational  programme),  ESF  support  to  social  inclusion 
included operations types 1, 4, 5 and 6).  

Overall,  coherence  is  very  strong  between  investment  priorities  within  ESF  operations 
focussing  on  education  and  training  (TO10),  as  well  as  between  national  and  regional 
operational  programmes.  Most  consulted  stakeholders  and  case  study  research  emphasised 
the  fact  that  the  sectoral  focus  of  the  different  investment  priorities  helped  ensure  clarity  of 
scope  and  coherence  between  the  different  levels  and  types  of  education  and  training:  pre-
school and school education (10i), higher education (10ii), lifelong learning (10iii) and vocational 
education  and  training  (10iv).  There  is  some  slight  blurring  of  the  distinction  between  10iii 
(lifelong learning) and 10iv (labour market relevance) in different Member States, however. In 
particular,  VET-focused  priorities  and  operations  could  be  found  under  both,  especially  for 

75  66% of planned funds, 67% of allocated funds and 67% of declared expenditures. 

51 

 
                                                           
relation  to  adult  learners  and  workers.  However,  specific  operational  programmes  generally 
used a clear intervention logic for each (this varied according to the needs of the Member State 
or region), and there is no evidence of overlaps. Still, challenges related to ensuring coherence 
were occasionally raised, in particular in terms of potential duplications between national and 
regional operational programmes. One example is Portugal, where joint work was required to 
avoid  overlaps  between  the  national  programme  for  success  at  school76  and  the  local  action 
plans to tackle school failure77 which are supported under regional operational programmes.   

5.4.1.2  Coherence across thematic objectives 
Overall,  internal  coherence  between  operations  funded  across  different  ESF  thematic 
objectives (TOs 8, 9 and 10) is good, with many examples of coherence and complementarity 
between  operations.  There  have  been  no  overlaps  in  practice.  However,  there  were  some 
challenges  related  to 
implementation  (e.g.  cases  of  over-separation  of  management 
responsibilities  and  weak  inter-sectoral  cooperation)  which  may  have  limited  the  synergies 
between operations under different thematic objectives in certain Member States.  Key factors 
contributing  to  internal  coherence  (confirmed  in  the  Policy  Delphi  validation)  included:  (i) 
rigorous  planning  at  programming  stage,  (ii)  ensuring  alignment  with  CSRs  and  national 
strategies  and  (iii)  good  coordination  and  communication  between  authorities  managing 
different TOs/investment priorities/operational programmes. 

At the planning stage there is high potential for overlaps between TO9 and the other two TOs, 
given  that  there  are  strong  similarities  between  most  types  of  social  inclusion  operations  and 
with  operations  under  other  TOs.  Indeed,  some  actions  that  could  have  been  programmed 
under other TOs may have been programmed under TO9 to meet the requirement that Member 
States  allocate at  least  20%  of ESF  funds to TO9. The potential for overlap is particularly  high 
between  some  types  of  TO9  and  TO8  operations.  For  example,  Type  1  (employed-focused 
actions)  and  Type  2  (enhance  basic  skills)  have  potential  overlaps  with  TO8  -  Employment 
Objective78. The overlap with TO10 was small, mainly because of the limited presence of Type 3 
(basic school education) operations under social inclusion. 

Although  they  often  have  a  similar  employment  goal,  operations  supporting  social  inclusion 
focus  on  the  more  disadvantaged  groups  and  take  a  more  holistic  approach  than  other 
thematic  objectives,  which  helps  avoid  potential  overlaps.  Social  inclusion  operations  often 
focused on the most vulnerable groups and people who were furthest from the labour market, 
which  ensured  that  they  were  different  -  as  well  as  complementary  -  to  TO8  and  TO10 
operations, which covered a much broader target group. There are several examples where TO9 
operations  were  distinguishable  from  TO8  operations,  primarily  regarding  Type  1  and  2 
operations.  In  Spain,  the  Acceder  project  provided  services  tailored  to  a  specific  group  in  a 
vulnerable situation (which would not  be  provided by regular public employment services). In 
Italy  participants  in  TO9  operations  were  recruited  from  social  services  or  mental  health 

76  PNPSE - Programa nacional de promoção do sucesso escolar. 

77  PiiCiE - Plano integrado e inovador de combate ao insucesso escolar. 

78  More than half of expenditures declared by beneficiaries to managing authorities (55%) and recorded 

participations (55%) for TO9 had potential overlaps with TO8 – Employment objective. 

52 

 
                                                           
services,  while  participants  in  operations  under  TO8  and  TO10  were  recruited  through  public 
employment services or training providers.   

Overall, there has been a good degree of coherence across ESF thematic objective operations, 
as well between TO10 and TO8, particularly in relation to youth employment and vocational 
training For example, several operations relating to training for young people also refer to YEI79 
or TO8 funding: (i) the Irish national evaluation of TO1080 underlines the coherence with YEI; (ii) 
in  the  regional  operational  programme  for  Lombardy,  the  ESF  contribution  to  vocational 
identifies 
education 
complementarity with TO8.  

is  coherent  with  TO8;  (iii)  a  Slovenian  case  study  also 

in  TO10 

Some  consulted  stakeholders  did,  however,  suggest  that  coherence  could  be  improved,  by 
involving stakeholders more in design and implementation phases.. Issues were raised in this 
area  due  to  difficulties  in  relationships  with  stakeholders  involved  in  implementation.  Some 
targeted  consultations  also  underlined  that,  synergies  between  the  different  TOs  were 
sometimes limited, as the actors involved in TO8 and TO10 came from ‘different worlds’. Finally, 
evidence from case studies highlights a number of some challenges for ensuring coherence, for 
example in Romania, where despite good coherence between thematic objectives on paper, a 
lack  of  inter-sectoral  cooperation  and  multi-sectoral  policy  programming  meant  that  tackling 
the  phenomenon  of  early  school  leaving,  or  adapting  education  systems  to  the  needs  of  the 
labour market, are not addressed in a sufficiently coherent way. 

5.4.2  ESF external coherence 
5.4.2.1  Coherence with EU policies 
Overall,  ESF  operations  are  well  aligned  with  EU  policies  in  the  fields  of  employment,  social 
inclusion and education and training. Moreover, evidence gathered during the evaluation shows 
that there is strong alignment and a generally high level of coherence between ESF operations 
and  the  priorities,  analyses  and  country-specific  recommendations  (CSRs)81  of  the  European 
Semester82. 

ESF  support  to  employment  operations  are  fully  aligned  with  EU  employment  policy  and 
objectives, as set in the EU2020 strategy and in the European Semester. In general, the country 
specific recommendations are addressed by the ESF TO8 investments (see point 5.4.4 below). 

In  all  Member  States,  ESF  support  to  social  inclusion  was  aligned  with  the  European  policy 
framework on social inclusion83,  at  both the  design and the planning stage,  and  remained so 

79  Youth Employment Initiative 

80  POBAL (2017), Kickboxing, Kindness and Going the Extra Mile. 

81  https://ec.europa.eu/info/business-economy-euro/economic-and-fiscal-policy-coordination/eu-

economic-governance-monitoring-prevention-correction/european-semester/european-semester-
timeline/eu-country-specific-recommendations_en  

82  https://ec.europa.eu/info/business-economy-euro/economic-and-fiscal-policy-coordination/eu-

economic-governance-monitoring-prevention-correction/european-semester_en  

83  The  European  policy  framework  on  social  protection  and  social  inclusion  encompasses  a  range  of 

policies that are specific to and cut across sectors and target groups. 

53 

 
                                                           
after reprogramming and OP modifications. The EU 2020 strategy84 and the European Pillar of 
Social Rights85, introduced in 2017, were policies the most often cited in planning documents – 
the  former  was  cited  in  the  planning  documents  in  16  Member  States86    while  the  latter  was 
cited in the planning documents in seven Member States87. Planning documents were updated 
during the programming period to reflect the introduction of the European Pillar of Social Rights 
in 2017. The social investment package (SIP)88 was only referenced in the planning documents 
for TO9 operations in Romania89. Few references to EU policies for specific target groups (e.g. 
Roma and persons with a disability) were made, although in practice these target groups were 
frequently  addressed  by  social  inclusion  operations.  The  EU  framework  for  national  Roma 
integration  strategies  was  cited  by  three  countries  (Bulgaria,  Germany,  and  Romania),  the 
European  disability strategy 2010-2020  was only  cited  by Lithuania.  In  contrast,  the  review  of 
annual  implementation  reports  found  that  operations  in  22  Member  States  aimed  to  help 
specifically  Roma  people  and  other  minorities  and  operations  in  25  Member  States  aimed  to 
help  people  with  disabilities.  Respondents  to  the  EU-level  Delphi  survey  also  highlighted  the 
value of considering the bigger picture and linking social inclusion to innovation - environment 
and digital policies, among others90  - (only one Member State (Romania) referenced the digital 
single market Strategy). 

Overall, ESF TO10 operations are well aligned with other EU policies in the field of education 
and training, including the ET 2020 strategic framework. They are also in general well aligned 
with  the  EU’s  new  policies  and  policy  priorities  which  have  emerged  since  2014,  such  as  the 
digital education action plan91, the Council Recommendation on high-quality early childhood 
education  and  care  systems92  and  the  education  priorities  of  the  action  plan  for  the 

84  European  2020  –  A  European  strategy 

for  smart,  sustainable  and 

inclusive  growth: 

https://ec.europa.eu/eu2020/pdf/COMPLET%20EN%20BARROSO%20%20%20007%20-
%20Europe%202020%20-%20EN%20version.pdf  

85  European  Pillar  of  Social  Rights  –  Building  a  more  inclusive  and  fairer  European  Union: 
https://ec.europa.eu/commission/priorities/deeper-and-fairer-economic-and-monetary-union/european-
pillar-social-rights_en  

86  Germany,  Estonia,  Ireland,  Greece,  Croatia,  Italy,  Latvia,  Luxembourg,  Lithuania,  the  Netherlands, 

Austria, Poland, Portugal, Slovenia, Finland and the UK. 

87  Belgium, Ireland, Spain, Croatia, Italy, Slovenia and Romania. 

88  European  Commission,  2015.  Policy  roadmap  for  the  implementation  of  the  social  investment 

package: 
https://ec.europa.eu/social/main.jsp?langId=en&catId=1044&newsId=1807&furtherNews=yes  

89  Communication  from  the  Commission  2013.  Towards  social  investment  for  growth  and  cohesion 
including  implementing  the  European  Social  Fund  2014  –  2020.  staff  working  document(2013)  39 
final. 

90  Annex 6.1 – EU-level Delphi survey findings. 

91  https://ec.europa.eu/education/education-in-the-eu/digital-education-action-plan_en  

92  https://eur-lex.europa.eu/legal-content/en/ALL/?uri=CELEX%3A52018DC0271  

54 

 
 
                                                           
integration of third-country nationals93, which was launched in response to  the 2015 migrant 
crisis. Moreover, evidence, from ESF-funded operations shows that the ESF has supported the 
implementation of these policies on the ground.  Investments made under TO10 also contribute 
to  the  Skills  Agenda  for  Europe94,  particular  via  interventions  funded  under  10iii  and  10iv. 
However, due principally to ESF’s national focus, few of the operations funded by ESF TO10 use 
EU policy tools (e.g. EQF95, ECVET96, EQAVET97).  

There is also evidence that the ex-ante conditionalities led to greater coherence with EU and 
national policies, fostering structural reforms in some of the Member States/regions that had 
to  fulfil  them.  The  2017  Commission  staff  working  document  on  the  added  value  of  ex-ante 
conditionalities98  highlights  some  examples:  in  Czechia,  a  detailed  action  plan  for  inclusive 
education  was  implemented  to  fulfil  the  ex-ante  conditionalities  on  early  leavers  from 
education and training, addressing the repeated country specific recommendation on the need 
to include disadvantaged children (including Roma) in mainstream schools.  

5.4.2.2  Coherence with other EU instruments 
ESF TO8 operations also complement actions aiming to improve the competitiveness of SMEs 
(TO3)  and  research  and  innovation  (TO1)  mostly  supported  by  the  ERDF.  However,  the 
integration  between  the  ESF  and  the  ERDF  is  not  always  straightforward  and  strengthened 
coordination could further encourage integrated projects. Only 8.9% of the respondents to the 
public consultation considered that they are not aligned. 

Another  important  ESF  TO8  synergy  of  is  with  the  EURES  (mobility)  axis  of  the  EaSI 
programme99; in some countries ESF directly supports the EURES network. In general, there is a 
good level of complementarity with other structural funds and EU programmes, although there 
are  not  many  instances  in  which  this  complementarity  gives  life  to  multi-fund  projects.  As 

93  https://ec.europa.eu/home-affairs/sites/homeaffairs/files/what-we-do/policies/european-agenda-

migration/proposal-implementation-
package/docs/20160607/communication_action_plan_integration_third-country_nationals_en.pdf  

94  https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52016DC0381  

95  Council  Recommendation  of  22  May  2017  on  the  European  Qualifications  Framework  for  lifelong 
learning and repealing the Recommendation of the European Parliament and of the Council of 23 April 
2008  on  the  establishment  of  the  European  Qualifications  Framework  for  lifelong  learning, 
http://data.consilium.europa.eu/doc/document/ST-9620-2017-INIT/en/pdf 

96  Recommendation of the European Parliament and of the Council of 18 June 2009 on the establishment 
of  a  European  credit  system  for  vocational  education  and  training  (ECVET),  https://eur-
lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32009H0708(02)&from=EN  

97  Recommendation of the European Parliament and of the Council of 18 June 2009 on the establishment 
of a European quality assurance reference framework for vocational education and training, https://eur-
lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32009H0708(01)&from=EN  

98  European Commission (2017), The value added of ex-ante conditionalities in the European Structural 

and Investment Funds, staff working document(2017) 127 final. 

99  https://ec.europa.eu/social/main.jsp?langId=en&catId=1081  

55 

 
                                                           
regards support to labour mobility, more specific issues like complementarity with Interreg and 
facilitation of life of frontier workers and employers could be explored. 

There  is  evidence  to  suggest  that  ERDF  funds  for  infrastructure  such  as  housing  and  social 
structures  were  complementary  to  ESF  support  for  TO9.  There  is  some  evidence  that  goods 
purchased  with  FEAD  funds  were  used  to  support  the  provision  of  more  comprehensive 
measures funded through ESF support for TO9. Though little was used under ESF, community 
led  local  development  (CLLD)  has  provided  a  way  to  channel  resources  from  different  funds 
(notably the EARDF) to support integrated, local solutions promoted by local private and public 
actors. 

The coordinated use of the Asylum, Migration and Integration Fund (AMIF) with ESF support 
was  less  common,  but  at  least  in  Italy  this  combination  has  made  it  possible  to  offer  more 
support to migrants following the 2015 crisis. Limited coordination across funds, as well as the 
complexities linked to drawing from different funds, were identified as reasons for not (further) 
pursuing the integrated use of funds.  

As  regards  ESF  TO10  operations,  there  are  many  examples  of  ESF  coherence  with  the 
European  Regional  Development  Fund  (ERDF)  and  Erasmus+  both  at  operational  programme 
and  operation  level.  However,  examples  for  the  other  European  Structural  and  Investment 
Funds are more limited (notably the EARDF and other funding instruments such as the Asylum, 
Migration and Integration Fund (AMIF)100  and the Marie Skłodowska-Curie actions (MSCA).  

There are synergies between ESF TO10 and ERDF funding, and they provide very valuable and 
complementary  support  to  major  reforms  and structural  change  in  the  field  of  education  and 
training.  Examples  include  major  reforms  in  early  childhood  education  and  care  or  in  digital 
education,  for  instance  in Croatia or  Slovenia.  The  level of  coherence  and  complementarity  is 
also high between ESF TO10 and Erasmus+, and there are several good examples of synergies in 
practice.  One  such  example  is  ESF  funding  being  used  to  provide  top-up  support  for 
disadvantaged  students  and  support  their  mobility  under  Erasmus+,  or  upscale  Erasmus+ 
projects.  

As regards coherence with the EARDF, some examples of good synergies are presented in the 
2019 Evaluation of the impact of the CAP on generational renewal, local development and jobs 
in  rural  areas101.  The  evidence  from  this  evaluation  shows  that,  coherence  between  EU  funds 
has  improved  in  the  current  programming  period  compared  with  previous  ones.  There  is 
coherence  between  the  EARDF  and  the  ESF,  particularly  in  those  Member  States  where  the 
different policies are delivered together by sub-regional delivery bodies or similar arrangements 
(e.g.  local  development  companies  in  Ireland  delivering  social  inclusion  programmes,  local 
integrated approaches in some regions of Italy, and chambers of agriculture in France mobilising 
ESF support to train candidates receiving EARDF support).  

100  https://ec.europa.eu/home-affairs/financing/fundings/migration-asylum-borders/asylum-migration-

integration-fund/union-actions_en  

101  https://op.europa.eu/en/publication-detail/-/publication/4bd0b0a2-0503-11ea-8c1f-01aa75ed71a1  

56 

 
                                                           
Examples of potential synergies with other EU funding instruments such as AMIF or the Marie 
Skłodowska-Curie  actions  (Horizon  2020)  being  used,  are  limited,  despite  high  levels  of 
coherence in the legal texts. Although alignment with the objectives of EU policy tools such as 
ECVET,  EQF  and  EQAVET  exists  on  paper,  examples  of  using  of  these  tools  in  ESF  TO10 
operations  or  programming  are  rare,  especially  compared  to  Erasmus+  projects.  This  is 
potentially  due  to  the  fact  that  transnational  cooperation  is  not  a  priority  under  ESF,  which 
limits the need to align activities with EU frameworks.  

5.4.2.3  Coherence with national policies 
ESF TO8 investments are generally underpinned by country specific recommendations and thus 
contribute  to  addressing  the  challenges  identified.  This  is  also  attributable  to  the  negotiation 
process  that  has  taken  place  at  the  programming  stage  between  the  Commission  and  the 
Member  States;  the  negotiation  process  continues  during  implementation  as  Member  States 
negotiate  with  the  amendments  and  changes  to  the  operational  programmes.  Operations 
funded  under  TO8  are  generally  coherent  with  other  policies  and  strategies  supporting 
employment and mobility at the national and regional levels.  

This coherence is confirmed by positive evidence from evaluations and the case study analyses. 
This  is  the  case  for  instance  for  evaluations  in  the  Veneto,  Hamburg,  Central  Macedonia, 
Lombardia, Umbria, Thüringen and Lubuskie operational programmes. In Veneto, an evaluation 
found  that  that  the  operational  programme  is  coherent  with  the  regional  comprehensive 
strategy for 2014-2020. In Hamburg, the ESF’s coherence with European and national strategies 
is reported as high and the link between the ESF and these strategies increased in the last two 
years.  The  evaluation  also  showed  that  the  EU  framework  for  national  Roma  integration 
strategies  up  to  2020  provides  a  useful  complement  to  the  policy  framework.  Operations 
implemented  by  the  Croatian  employment  service  and  financed  by  the  European  Social  Fund 
are complementary to the guidelines for the implementation of an active employment policy in 
the Republic of Croatia for the 2015-2017 and 2018-2020 periods. 

In France, the TO8 focus on entrepreneurship is coherent and complementary with the national 
pact  on  growth,  competitiveness  and  employment.  Together  these  actions:  (i)  support 
companies’ development and investments, by introducing entrepreneurship in the educational 
programmes  of  secondary  schools  and  higher  education,  and  (ii)  to  make  it  easier  to  start  a 
business by simplifying processes and adapting support. 

Overall ESF support for TO9 was found to be coherent with national and regional policies and 
programmes.    For  example,  in  Greece,  ESF  support  to  social  inclusion  complemented  and 
reinforced  a  number  of  means-tested  policies  aiming  to  help  people  in  vulnerable  situations 
(minimum income guarantee, child benefits, rent subsidy, heating benefits, ad hoc transfers to 
the  poorest  households  and  ad  hoc  measures  to  facilitate  access  of  the  most  vulnerable  to 
public health care services). In Lithuania, the alignment of ESF actions under TO9 with national 
and  regional  policy  measures  is  ensured  through  the  requirement  of  having  a  direct  link 
between  planned  measures  with  EU  and  national  funds  and  national/regional  strategic 
documents.   

As regards TO10 operations, there is strong coherence between investments under made TO10 
and other activities supporting education and training at national and regional level. This close 

57 

 
coherence was further confirmed by most case studies, targeted consultations and, the results 
of  the  public  consultation.  Even  greater  coherence  could  be  achieved  through  wider 
involvement  of  national/regional  stakeholders  (including  labour  market  actors  and  target 
groups), as well as through better coordination of policies and actions at national/regional level. 
Target groups themselves (or their representatives) should also be involved in order to better 
identify and monitor evolving needs.  

Overall,  Member  States  have  been  relatively  effective  in  their  use  of  ESF  funds  to  support 
national and regional strategic priorities under the various TOs.  

5.5  EU added value 

The assessment of the EU added value of ESF operations has involved the identification of key 
dimensions and types of effect (See Box 6 for a description), and the gathering of evidence to 
determine  the  extent  to  which  ESF  operations  have  had  an  effect  at  the  national  or  regional 
level  that  would  not  have  happened  without  EU  interventions.  The  supporting  evaluation 
studies in general conclude that for all three ESF thematic objectives there is a good degree of 
EU added value across all areas, with substantial evidence pointing to volume, scope, role and 
process effects. 

Box 6. Types of ESF added value  

• 

• 

• 

• 

Volume effects: ESF action adds to existing action, either by supporting national action 
in general (‘mirroring’) or in specific areas of national policy (‘boosting’);  
Scope effects: ESF action broadens existing action by supporting groups or policy areas 
that would not otherwise receive support;  
Role  effects:  ESF  action  supports  local/regional  innovations  that  are  taken  up  at 
national level or national innovative actions that are then mainstreamed;  
Process effects: ESF action influences Member-State administrations and organisations 
involved in the programmes. 

5.5.1  Volume effects 
Evidence  shows  that  ESF  support  to  employment  has  considerably  added  value  in  terms  of 
volume effects, as it made it possible to support more people to be supported than would be 
the case if only national programmes were available. According to a recent study on the impact 
assessment of human capital investments102, the ESF had a significant volume effect, especially 
through investment priority 8i (Access to employment), which received the largest share of the 
ESF budget within TO8. 

It is difficult to assess the precise  extent of the effect  but,  based on existing evidence,  a high 
proportion of people supported by the ESF would otherwise have probably simply benefited 
from  passive  unemployment  support.  This  is  particularly  true  in  Czechia,  where  most  active 
labour  market  policies  are  co-funded  by  the  ESF.  Without  ESF  support,  many  disadvantaged 
people (especially those aged over 55, people with disabilities, the long-term unemployed and 
the low-skilled) would simply receive passive unemployment support.  

102  FGB, Study supporting the impact assessment of human capital investments, Final report, May 2018. 

58 

 
 
                                                           
In  France,  ESF-funded  operations  are  recognised  to  have  a  powerful  leverage  effect.  The 
programming strategy is clearly oriented towards a volume effect: ESF funding is mostly used to 
reinforce  existing  employment  policy  (without  adding  new  operations).  In  the  Rhône-Alpes 
operational programme, for instance, the ‘Mode d’emploi’ operation (which provides intensive 
individualised  support  to  participants)  complements  the  support  provided  by  the  public 
Employment  service.  In  Spain,  the  ESF  operations  funded  under  TO8  allowed  intermediate 
bodies  to  increase  their  impact  by  reaching  a  wider  audience  which  would  not  have  been 
possible without the ESF. In Romania, operations under investment priorities 8i and 8iii added 
to the measures financed by the State budget. In Denmark, TO8 operations have added value 
by increasing the number of people who benefited from entrepreneurial training and education 
and, entrepreneurial support and consultancy, as well as the number of companies involved in 
skills upgrading.  In the  Campania operational programme  –  Italy, the volume  effect  was clear 
for  both  the  number  of  actions  and  the  number  of  participants  involved  for  all  investment 
priorities under TO8; this is due to the integration between ESF funding and national/regional 
funding for the same types of actions and groups.  

There is also evidence that ESF support to social inclusion played a significant role in funding 
measures  fighting  social  exclusion  and  poverty,  complementing  national  policies.  A  volume 
effect  was  identified  in  22  Member  States.  This  effect  was  primarily  observed  in  terms  of 
complementarity  with  national  efforts  (17  Member  States)  and  also  in  terms  of  boosting 
funding for social inclusion (5 Member States). The analysis103 provided examples for each type 
of volume effect:  

• 

• 

Complementary  to  national  efforts:  In  Spain  the  Special  Employment  Centres 
(Centros Especiales de Empleo) focus on labour market integration of workers with 
disabilities,  which    is  a  widespread  national  policy  co-funded  by  several  ESF 
operational  programmes  -it  provides  a  partial  subsidy  towards  the  wage  costs  of 
workers  with  disabilities 
in  Special  Employment  Centres.  In  Lithuania,  the 
ESF-supported  activation  measures  for  elderly  people,  complement  and  boost 
national policies. In Italy, ESF support to social inclusion boosted national measures 
implementing the national anti-poverty strategy. 
Primary  role  in  social  inclusion  funding:  In  Hungary,  most  measures  aiming  to 
support  the  social  inclusion  of  people  in  vulnerable  situations  are  funded  by  ESF. 
Similarly in Latvia, the creation and provision of community-based services helping 
people  with  mental  disorders 
lives  and,  social  care  and 
rehabilitation services for disabled children and their family members, would likely 
have not been possible at all without ESF support. Moreover, in Poland, ESF support 
to social inclusion promoted the development of the social economy. However, the 
primary  role  of  ESF  support  raises  some  concerns  about  the 
longer-term 
sustainability of these interventions. 

independent 

lead 

The  results  of  the  public  consultations  show  that  the  vast  majority  of  ESF  participants,  the 
general public, organisations managing/delivering ESF actions and other organisations all have a 
positive  perception  of  the  EU  added  value  of  ESF  support  in  terms  of  volume  effects.  For 
instance,  for  education  and  training  operations,  87.2%  out  of  370  respondents  from 
organisations  directly  involved  in  the  management  or  delivery  of  ESF  actions  considered  that 

103  https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes     

59 

 
                                                           
‘more can be done than with national resources alone’. Over 80% out of 46 respondents from 
organisations familiar with the ESF but not directly involved also held this view, as did 73.9% out 
of 173 members of the general public with an awareness of ESF. For employment support (TO8 
public consultation), 48% of all 541 respondents considered that without the ESF there would 
not  be  enough  money  to  pay  for  the  supported  programmes.  The  importance  of  the  volume 
effect of ESF-funded operations was also highlighted during the focus group discussions, in the 
Policy Delphi survey and in the various case studies.  

Only a minority of respondents to the public consultations answered that the EU should not be 
involved. In the case of social inclusion for instance, 14% (7 of 50 respondents not involved in 
ESF  delivery  and  not  aware  of  the  ESF)  felt  there  was  no  role  for the  EU  in the  promotion of 
social    inclusion, combating poverty and any discrimination. 

Box 7. Significant volume effects of ESF support to education and training 

Many of volume effects of education and training operations were identified in the supporting 
study104, including: (i) an increase in the number of participants (teachers/trainers or learners) 
and  learning  establishments  that  can  benefit  from  education,  training  or  school/college 
improvement  programmes,  which  increases  the  geographical  scope  of  implementation  of 
education and training initiatives; and (ii) implementation of innovative education and training 
programmes  at  a  larger  scale  than  would  be  possible  without  ESF  support  (e.g.  inclusive 
education,  adult  learning,  programmes  to  help  early  leavers  from  education  and  training). 
Although  volume  effects  were  most  visible  –  and  most  often  cited  –  for  large-scale 
interventions  across  schools  (or  pre-schools)  under  10i,  the  supporting  study  noted  volume 
effects under all investment priorities, with particularly strong volume effects relating to adult 
learners (10iii). 

There  are  many  examples  of  volume  effects  across  the  different  Member  States.  These  
include: (i) the ‘Time for Leaders’105  ESF operation in Lithuania which has already involved over 
20 000 educational staff (from pre-school institutions to vocational education centres), and (ii) 
the PiiCiE operation in Portugal (Plano integrado e inovador de combate ao insucesso escolar), 
which targets primary and secondary education students and develops a wide range of activities 
aiming to reduce school failure. 

In some Member States whose national education and training budgets were particularly hard 
hit by the economic crisis, ESF support allowed the education and training system to continue 
developing despite the grave impact of the crisis on all levels of education. This important EU 
added value was highlighted, for instance, in the case study on the Pomorskie region in Poland, 
where both the ESF and ERDF helped to plug the gap left by significant cuts in the education and 
training budget allocated to regions by the central government. 

5.5.2  Scope effects 
Several sources suggest106  that in some cases, ESF support to employment operations covered 
target groups which are not covered by nationally funded operations, therefore reflecting the 

104  https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes  

105  www.lyderiulaikas.smm.lt    

106  Impact Assessment study, the Labour Market Policies database, focus group and case studies. 

60 

 
 
 
 
                                                           
scope  effect  of  ESF  co-funded  operations.  Such  target  groups 
entrepreneurs/self-employed, women and older workers.  

include  young  people, 

According to the labour market policies database107, this is the case for employment incentives 
in  Estonia,  Spain,  Cyprus,  Latvia,  Lithuania,  Hungary  and  Slovenia,  where  the  long-term 
unemployed  are  specifically  targeted  only  by  ESF  co-funded  operations.  The  supporting  study 
provides  many  relevant  examples  where  ESF  operations  broadened  the  target  groups.  One 
example is Estonia, where the supported measures were extended to new target groups (low-
skilled,  young  people,  older  people,  and  people  with  disabilities).  Similarly,  in  France,  by 
targeting  groups  such  as  women  and  older  people,  ESF-funded  operations  have  a  leverage 
effect.  In  Italy,  ESF  employment  operations  co-financed  under  the  national  system  for  active 
employment  policies  operational  programme  also  broadened  the  target  populations  (e.g.  to 
include the unemployed in incentive projects).  

The ESF has also contributed to the broadening of existing operations including the adoption 
of individualised operations and targeted approaches to focus on the needs of specific groups. 
Perhaps  its  main  contribution  was  to  increasing  awareness  of  gender  equality  and  the 
introduction  of  specific  policies,  which  were  previously  noticeably  absent  in  some  Member 
States.  In  Germany,  ESF  operations  play  a  role  in  plugging  the  gap  left  by  national  actions 
related  to  specific  target  groups  that  are  hard  to  reach  and  specific  topics  related  to 
employment.  These  include,  for  instance,  ESF  funding  for  8iv  Equality  between  men  and 
women; without the ESF, these type of operations would simply not exist. Another example is 
the ‘Unternehmens Wert: Mensch’ operation, under which counselling of companies could not 
be proposed without the ESF support. Finally, there is also some evidence (e.g. Croatia) where 
the ESF has prioritised investments in hard-to-reach areas that had previously not been priority. 

Box 8. Social inclusion scope effects: broadening actions to more groups 

A scope effect of ESF support to social inclusion was identified in 17 Member States108. The ESF 
provided support for specific target groups that were not covered or that received less coverage 
in  nationally  funded  interventions.  For  example,  in  Bulgaria,  ESF  support  to  social  inclusion 
covered  Roma  children  and  children  with  special  educational  needs,  in  Denmark  it  covered 
people  with  ‘limited  attachment’  to  the  labour  market,  and  in  Croatia  operations  supported 
people with disabilities. The Netherlands used ESF to support refugees and Italy used it to help 
LGBT  people  facing  housing  exclusion.  In  Austria  the  'Youth  College  Vienna'  project  provided 
education and counselling to young refugees who would have not received support from other 
sources and who are less likely to have the skills needed to enter training or the labour market. 

For education and training operations, there is strong evidence of scope effects both in terms 
of broadening actions to new target groups and to new policy areas.  In terms of broadening 
actions  to  include  new  target  groups,  evidence  clearly  shows  that  ESF  funding  has  given 

107

https://webgate.ec.europa.eu/empl/redisstat/databrowser/explore/all/lmp?lang=en&display=card&

sort=category  

108  Belgium, Bulgaria, Denmark, Germany, Ireland, Greece, Croatia, Italy, Lithuania, Luxemburg, Malta, 

the Netherlands, Austria, Slovenia, Slovakia, Sweden and Romania. 

61 

 
 
                                                           
 
education and training providers an opportunity to address more disadvantaged and harder-to-
reach  groups.  This  is  true  in  all  Member  States  and  across  all  education  sectors  and  all 
investment priorities. Such groups include young NEETs, early school leavers, children with low 
socio-economic status, low-skilled or low-qualified adults, migrants, Roma, people with special 
educational  needs  (SEN),  people  living  in  remote  areas,  as  older  workers  and  workers  with 
obsolescent skills. The additional cost of reaching out to these groups and providing the sort of 
individualised and cost-intensive support which they require often means that they are de facto 
excluded  from  standard  education  and  training  provision.  Examples  of  good  practices  in  this 
area include the Bulgarian ‘Your Class’ ESF operation109 and the German ‘Voluntary 10th school 
year’ operation110. 

Moreover,  ESF  funding  has  allowed  education  and  training  providers  to  address  wider  policy 
areas  such  as  inclusive  education,  adult/lifelong  learning,  non-formal  education,  digitisation, 
civic education, developing vocational training in wider/future-oriented sectors and developing 
soft/transversal  competences.  Examples  include  the  ‘PuMPuRs’111  operation  targeting  early 
leavers from education and training in Latvia and the  ‘digital skills bridge’112   ESF operation in 
Luxembourg,  which  aims   to support  the  development  of  digital  skills  for workers  affected  by 
digital transformations.  

5.5.3  Role effects 
Evidence on role effect of ESF TO8 operations is mixed. There is little information on the extent 
to which the operations supported by the ESF supported innovation and the transfer of ideas, 
although there are some examples of good practice with lessons that can be learned. 

Box 9. Examples of TO8 role effects 

• 

• 

Limited role effects: The case studies provide some insight into the ESF’s role effect of 
the  ESF,  highlighting  that,  in  Spain  and  Romania,  TO8  operations  provided  little  or  no 
support to innovation. In Spain for instance, innovation was included in ESF actions only 
to  a  limited  extent  due  to  uncertainty  regarding  its  eligibility.  Most  supported 
operations complemented already existing actions, without necessarily introducing new 
approaches to tackling the problems. A lack of innovative measures was also identified 
in Romania, particularly for ESF operations focusing on the most disadvantaged groups, 
such as Roma or low educated people. 
Substantial  role  effects:  In  Austria,  it  is  reported  that  the  ESF  supported  innovative 
operations to promote gender equality (8iv) and support active and healthy ageing (8vi). 
Similarly,  in  Germany,  the  role  effects  of  ESF-funded  operations  in  the  field  of 
employment  and  mobility  were  recognised  at  the  level  of  federal  and  regional 
operational  programmes.  Examples 
innovative  federal  operations 
supported under investment priority 8v: ‘UnternehmensWert: Mensch’ (which aims to 
improve working conditions) and ‘IQ-Qualifizierungsprogramm’ (which supports people 
who  need  help  getting  recognition  of  their  certificates).  Some  elements  of  the  latter 

include  two 

109  See box A5-1 in Annex 5 of the TO10 supporting study. 

110  See Box 8 in the TO10 supporting study. 

111  http://www.pumpurs.lv/  

112  https://www.skillsbridge.lu/  

62 

 
                                                           
project were subsequently transferred into actions under regular national funding. 

The  focus  group  also  highlighted  two  examples  where  the  ESF  created  opportunities  to  test 
innovative  approaches  and  mainstream  theme.  In  Estonia,  peer  counselling  is  proposed  for 
people with disabilities (for instance a disabled worker provides advice to disabled people who 
are  looking  for  a  job  or  who  just  started  to  work  and  need  support).  In  Malta,  in  order  to 
provide  people  with  mental  disabilities  with  employment  opportunities,  work  activities  are 
divided  into  sub-tasks  to  allow  individuals  to  perform  single  tasks,  which  are  eventually 
combined. 

In general, the  role effect of the  ESF TO8 operations was  well  acknowledged. In Austria, it is 
reported  that  the  ESF-supported  innovative  operations  to  promote  gender  equality  (8iv)  and 
support  active  and  healthy  ageing  (8vi).  Similarly,  in  Germany,  the  role  effects  of  ESF  funded 
operations  in  the  field  of  employment  and  mobility  were  recognised  both  at  the  level  of  the 
federal and regional operational programmes.  

ESF TO10 investments have contributed to the transfer of ideas by funding the piloting or wider 
implementation of ideas from other Member States or from regional to national level. There is 
also evidence that ESF actions supported innovation in education and training including various 
ways of modernisation of practices and policies (e.g. digitisation, inclusive education), targeting 
skills  development  in  new  or  evolving  sectors  of  the  economy  (e.g.  sustainable  development, 
high-tech industries), testing new or alternative pedagogies (e.g. inclusive educational methods, 
use of theatre) and developing innovative training offers.  

Lastly, ESF TO10 has also led to structural changes in education and training systems including 
the roll-out or decentralisation of new approaches and policies. Examples of TO10 support for 
structural changes exist under all investment priorities; one such example is the establishment 
of the network of regional centres of VET competence in Croatia. 

5.5.4  Process effects including sustainability  
The TO8 case studies show that there are various benefits for administrations and organisations 
involved  in  ESF  operations.  In  Czechia  for  instance,  cooperation  between  the  Managing 
authority, the Ministry of Labour and Social Affairs, other state institutions, social partners and 
NGOs (in particular for actions supporting the capacity of labour offices, the social partners and 
NGOs) was highlighted as a process effect of ESF operations.  

The process effects of ESF supported operations were also largely recognised during the focus 
group  discussions.  ES  operations  increased  inter-institutional  cooperation,  for  example  in 
Bulgaria,  where  social  partners  and  social  services  now  work  together,  as  a  result  of  the 
cooperation  needed  to  implement  individual  centred  approaches  to  support  people  into  (or 
closer to) employment.  

As  regards  the  continuity  of  EU-supported  operations  after  funding  ends,  there  is  increasing 
evidence  of  them  being  mainstreamed  into  national  policies  especially  in  socio-economic 
contexts  that  is  more  favourable  compared  to  the  financial  crisis.  Thus,  the  systemic  changes 
achieved  through  the  implementation  of  ESF  TO8  support  are  likely  to  remain  even  after  ESF 
funding ends.  

63 

 
 
Managing authorities consider that it is still too early to assess the sustainability of TO9 support. 
ESF-funded  TO9  operations  generated  process  effects  by  improving  the  administrative  for 
capacity and knowledge of the design and delivery of services. 

Similarly, ESF TO10 support generated positive effects which are likely to continue after the end 
of  funding.  There  is  a  strong  likelihood  of  ongoing  benefits  at  all  three  levels:  individual, 
institutional and policy. 

However,  the  COVID19  pandemic  may  affect  the  sustainability  of  the  positive  effects  of 
operations run under the ESF’s three ESF thematic objectives. 

At  individual  level,  ESF  TO10  operations  had  an  immediate  positive  impact  on  many  people 
across the EU. The public consultation confirmed this positive impact with a large proportion of 
ESF  participants  reporting  that  the  support  had  made  a  positive  difference  to  their  situation: 
over half said that, without EU support, there would been fewer or no opportunities for people 
in  their  situation.  Key  outcomes  mentioned  by  individual  respondents  included:  learning  new 
skills and/or getting a qualification, feeling more confident, starting a new education or training 
course,  and  improving  employment  conditions,  such  as  increasing  salary  or  receiving  a 
promotion. 

These  immediate  results  of  participation  in  TO10  ESF-funded  operations  are  likely  to  have 
lasting impacts on the lives and life chances of most people involved.  

The findings also identify to sustainable impacts of ESF TO10 support at institutional level. Many 
of the effects on institutions are likely to have a long-lasting impact. For example: 

• 

• 

• 

Improvements to the competences of teachers and trainers are likely to contribute 
to  long-term  improvements  of  the  institutions  in  which  they  work,  for  example 
through improved teaching skills and methods, increased awareness of how to deal 
with diversity, better organisation and more innovation. Once new approaches and 
methods  are  introduced,  evidence  suggests  that  it  is  unlikely  that  institutions  will 
‘slip  back’  to  old  ways  and  instead  will  continue  a  positive  trajectory  of  ongoing 
improvements. 
The  links  established with other organisations and the  wider community (families, 
communities,  NGOs,  employers,  other  education  and  training  providers,  etc.)  will 
also  remain  in  place,  and  continue  to  provide  a  more  effective  and  integrated 
context for educational and training activities. 
The investments in new teaching curricula (e.g. in Croatia), resources and methods 
will benefit institutions for many years beyond the end of ESF funding. 

ESF  TO10  funding  also  had  many  policy  impacts  that  are  highly  likely  to  have  sustainable 
positive  effects.  Examples  highlighted  in  the  supporting  study  include  ESF  support  for:  the 
establishment of the  Maltese systematic approach to addressing early  leavers from education 
and training; the testing and introduction of the dual learning system in Belgium (Flanders); the 
new  start  programme  to  ensure  that  VET  schools  in  Greece  better  adapt  to  the  needs  of 
students from more vulnerable backgrounds; and the introduction of a major digital education 
reform in Croatia.. 

64 

 
6  CONCLUSIONS 

The  evidence  from  the  three  thematic  evaluation-supporting  studies  underpin  the  following 
overarching conclusions on 2014-2018 ESF operations.  

The  assessment  of  effectiveness  has  shown  that  there  has  been  progress  in  the  financial 
implementation  of  ESF  programmes  and  in  the  achievement  of  participation  and  result 
targets.  However  there  are  large  differences  in  performance  across  countries,  operational 
programmes and investment priorities. There were initial delays in implementing ESF operations 
and financial execution is lagging 7% behind the comparable stage of the previous programming 
period. Nevertheless, by the end of 2018, implementation rates were generally on track to meet 
the targets by the end of the programming period; although, administrative capacity to deliver 
operations  remains  a  constraint  in  some  countries  and  (especially)  within  less  developed 
regions.  

Overall,  across  the  EU,  27%  of  planned  resources  were  spent  by  the  end  of  2018,  and  the 
project selection rates113 were around 70%. As regards output targets they show good progress, 
as  22.2  million  participations  were  reported  covering  the  specific  target  groups  of  each  ESF 
thematic objective (with higher shares of unemployed participants in employment operations, 
of disadvantaged groups in social inclusion ones and of inactive and young people in education 
and training). 

Although  implementation  is  still  ongoing  and  comprehensive  results  data  are  still  to  be 
reported, the evaluation has nevertheless highlighted some significant achievements. There has 
been  good  progress  in  achieving  results  from  participation  in  ESF-funded  operations,  as  9.5 
million  positive  results  were  reported  in  total  by  December  2018  (this  includes  participants 
engaged 
including  self-employment;  entered 
education/training or gained a qualification). 

job  searching  or  who  found  a 

job; 

in 

There  is  evidence  that  ESF  operations  have  a  sustainable  positive  impact  on  participants.  In 
addition  to  the  reported  impacts  in  terms  of  gains  of  employment  and  education  and 
qualifications,  both  in  the  immediate  and  longer-term,  a  wide  range  of  'soft'  outcomes  were 
identified  notably  for  employment  and  social  inclusion  operations.  Such  outcomes  are  quite 
widespread  but  underreported,  as  they  are  not  part  of  the  common  indicators.  The  most 
common  soft  outcomes  are  soft  skills,  increased  self-esteem  and  improved  labour  market 
prospects.  

The three evaluation-supporting studies have identified some notable achievements at the level 
of investment priorities and types of operations: 

Within  ESF  support  to  employment,  the  most  successful  operations  have  been  access  to 
employment  for  the  unemployed.  The  assessment  confirms  varying  but  positive  effects  in 
terms of employment chances for unemployed participants due to the support they received. In 
addition, exploratory research on overall macroeconomic impacts tends to confirm that support 
is  beneficial  and  sustainable  for  several  regions  with  high  unemployment  levels.  Also,  ESF 

113  This is the ratio of funds allocated by the Managing Authority to specific projects or operations over 

the total amount available (by TO) or indicatively planned (by IP). 

65 

 
                                                           
operations  have  contributed  to  the  development  of  a  more  adaptable  workforce  and  have 
helped improve the employability of the participants. ESF support has been  most effective – in 
terms of supporting employment - for individuals who are at a certain distance from the labour 
market, i.e. neither close (such as well qualified individuals) nor too far away from it (e.g. those 
with multiple disadvantages). 

More  than  half  of  results  generated  for  ESF  support  to  social  inclusion  were  for  Type  1 
operations  (employment-focused  actions).  An  additional  35%  of  results  were  generated  for 
Type 6 operations (action influencing attitudes and systems). The result-level achievement rate 
was more moderate than that for outputs, which is in line with the fact that results take more 
time to materialise, especially for disadvantaged groups. Moreover, social inclusion operations 
helped  reduce  discrimination,  improved  integration  of  marginalised  communities,  changed 
attitudes towards education and increased soft skills. 

As  regards  education  and  training,  early  leavers  from  education  and  training  and  lifelong 
learning operations have been more successful in terms of both direct results and the potential 
for  longer-term  systemic  change.  The  relative  success  of  these  priorities  appears  to  relate  to 
their stronger alignment with national and regional strategic priorities. Many successful types of 
operations have been identified - ranging from support to vulnerable learners and the teaching 
staff  that  work  with  them,  to  digital  innovations  in  the  classrooms  and  local  action  plans  to 
address early leavers from education and training. However, many Member States have faced 
challenges preventing from achieving positive results under ESF tertiary education operations. 

On top of the  positive  results  for participants  in ESF operations, the modelling of ESF impacts 
based  on  RHOMOLO114  simulations  suggests  that  the  programme  will  have  final  positive 
macroeconomic impacts (measured as additional EU GDP and jobs compared to baseline) as a 
result of the ESF human capital investments, which enhance the productivity of the labour force 
and increase participation in the labour market, among other effects. The final estimated impact 
would add 0.26% to European GDP by 2023, and 316 000 jobs. 

Therefore, although it is not possible to fully disentangle the impact of ESF funding from other 
context  factors, there  is  clear  evidence  on  the  positive contribution  of ESF  operations  to the 
overarching objectives of ESF support to employment, social inclusion and education towards 
the  EU  2020  targets,  both  from  a  micro  and  a  macro  perspective  (based  on  results 
achievements, qualitative evidence from stakeholder views and micro and macro econometric 
modelling). 

Finally, significant potential for long-term impacts was identified through ESF support for the 
development  of  system  level  changes  and  sustainable  innovative  measures.  Qualitative 
evidence  (country-based  analyses  including  case  studies)  suggest  that  ESF  support  to  social 
inclusion  also  had  broader  level  impacts  including  enhanced  access  to  public  services, 
deinstitutionalisation,  and  cross-sectoral  collaborations  to  promote  innovative  approaches.  
Education  and  training  operations  greatly  support  systemic  change,  enhancing  changes  in 

114  RHOMOLO is the spatial computable general equilibrium model of the European Commission, see for 

details: https://ec.europa.eu/jrc/en/rhomolo  

66 

 
                                                           
government policy approaches to education, particularly visible for early leavers from education 
and training and lifelong learning operations. 

The analysis of efficiency has shown that there was a high level of variation across investment 
priorities and Member States in the cost-effectiveness of the different types of operations. The 
large  variance  in  the  cost  per  participation  reflects  the  wide  range  of  types  of  operation  and 
delivery mechanisms, as well as the different costs levels in the Member States.  

Overall  cost-benefit  analyses  attempted  for  social  inclusion115  and  education  and  training116 
suggest  positive  returns  particularly  under  active  inclusion,  early  leavers  from  education  and 
training and tertiary education operations, however a note of caution should be attached to the 
results  as  a  number  of  broad  assumptions  needed  to  be  applied,  given  the  limitations  of  the 
monitoring data and the evaluation evidence. 

Actual and perceived administrative burdens have hampered effective implementation, through 
delays, and also have impacted cost-effectiveness. New monitoring systems and databases have 
been developed at Member State level to comply with regulatory changes since the 2007-2013 
programming period, but a more significant factor is insufficient administrative capacity, which 
in turn impacts implementation.  

At  this  stage  of  the  implementation,  efficiency  can  only  be  approached  through  cost  per 
participation.  The  large  variance  reflects  the  wide  range  of  types  of  operations  and  delivery 
mechanisms,  as  well  as  the  different  costs  levels  in  the  Member  States.  The  cost-benefit 
analyses  carried  out  for  social  inclusion  and  education  and  training  support  suggest  positive 
returns particularly under active inclusion, early leavers from education and training and tertiary 
education operations. ESF administrative requirements are considered to be proportionate to 
the  benefits  achieved.  Nonetheless,  there  is  recognition  that  these  requirements  can  be 
complex  and time-consuming,  making  them onerous  for  smaller  beneficiary organisations,  for 
example those working with specific target groups or schools. 

In  support  to  employment  operations,  there  have  been  additional  costs  for  recruiting  and 
delivering  ESF  to  harder-to-reach  groups,  especially  those  in  remote/rural  areas.  As  regards 
social inclusion, other key factors that limit the efficiency of ESF operations include gold plating 
and  drawing  on  other  EU  funds  (e.g.  ERDF,  CF)  to  support  the  operational  programme.  The 
complexity  of  administering  an  operational  programme  with  multiple  funds  may  have  led  to 
delayed  implementation  and  generation  of  results.  For  education  and  training  support,  there 
was also evidence that having many partners involved in implementation can result in delays.  

Qualitative  analysis  has  shown  that  factors  that  can  foster  efficiency  include  the  use  of 
including  specialist  sectoral 
intermediary  bodies,  wider  partnership  arrangement, 
organisations,  and  simplified  cost  options  (SCOs).  The  introduction  of  SCOs  may  lead  to  an 
initial increase in administrative burden for some beneficiaries that need to adjust procedures 

115  See TO9 supporting study: 

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes  

116  See TO10 supporting study: 

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8346&furtherPubs=yes  

67 

 
                                                           
and train their staff. Over time, however, the use of SCOs can promote the take-up of ESF and 
lower the administrative burden.  

Overall,  there  was  a  high  degree  of  relevance  of  ESF  to  employment,  social  inclusion  and 
education  policy  needs  and  to  the  needs  of  target  groups  requiring  support,  at  the 
programming  and  implementation  levels.  Programming  remained  relevant  throughout  the 
programming period to date, as the operational programmes have proved to be flexible enough 
to adapt to socio-economic and policy changes. The flexibility of OPs allowed Member States to 
deal with unexpected shifts in the socio-economic context, such as the 2015 refugee crisis. 

In  terms  of  target  groups  reached,  the  evidence  shows  that  ESF  operations  were  generally 
aligned  to  their  needs  but  that  challenges  remain  in  reaching  and  addressing  the  needs  of 
some groups, particularly the most disadvantaged ones.  Support to employment managed to 
reach  participants  in  high  need  of  support,  including  the  inactive,  the  long-term  unemployed 
and migrants. The low budget available, lengthy implementation periods and a need for more 
holistic  and  informed  approaches  resulted  in  issues  linked  to  active  ageing  and  embedded 
gender  stereotypes  being  little  or  hardly  addressed.  The  supporting  study  shows  that  ESF 
support was most relevant when combined with other measures that support the participants 
(including health, housing, etc.), especially those furthest away from the labour market. 

ESF support for social inclusion reached a substantially higher share of people with a disability, 
migrants, people with a foreign background and minorities, as well as those with low education 
than did  support to employment and labour mobility and education operations, which is fully 
consistent with the focus and target groups of social inclusion. However, evidence as to whether 
ESF support to social inclusion reached the most vulnerable populations with the greatest needs 
is mixed due to the challenges in identifying these groups with the existing common indicators, 
as these  are  necessarily broad groups relevant at the EU level. The assessment also identified 
risks  of  ‘creaming’,  i.e.  targeting  less  vulnerable  people  with  less  complex  needs  who  can  get 
better results.  

The  biggest  challenges  faced  by  education  and  training  operations  have  been  to  reach  some 
disadvantaged groups, and they have registered lower participation shares for these groups as 
compared to other ESF operations. Refugees and migrants, Roma, the long-term unemployed, 
older  workers  and  people  with  disabilities  were  identified  as  the  most  challenging  groups  to 
engage. In particular, improving access to higher education for vulnerable population groups is a 
big  challenge,  as  disadvantaged  groups  have  been  hard  to  reach  for  ESF  tertiary  education 
operations because of difficulties in developing collaborative relationships between the tertiary 
sector and schools. 

Ensuring relevance to needs was fostered by a number of factors including the needs analyses 
in  partnership  agreements,  a  flexible  approach  to  programming,  allowing  adaptation  of 
operational programmes and planned actions to any changing needs of Member States/regions, 
and  close  alignment  of  ESF  priorities  with  the  European  Semester  and  its  country-specific 
recommendations. The assessment found that the European Semester is a strong framework for 
ensuring that ESF operations are relevant to target groups. Improvements to relevance could, 
however  be  made  through  enhanced  consultation  with  actors  closer  to  the  needs  of  target 
groups, such as NGOs, social partners and, training providers. While these actors were involved 

68 

 
and consulted to some extent – in part thanks to the partnership principle enshrined in the ESI 
Funds – there is room for more meaningful consultation and more engagement with a broader 
range of actors in the needs assessment, programming and implementation stages. 

The  assessment  also  showed  that  ESF  operations  have  a  good  overall  degree of  internal  and 
external coherence. They are well aligned with other EU policies and initiatives in the field of 
employment, social inclusion and education and have supported their concrete implementation 
on  the  ground.  There  is  also  a  good  degree  of  coherence  with  the  European  Semester  CSRs 
which  are  well  reflected  overall  in  the  situation  analyses  of  partnership  agreements,  helping 
address the challenges identified, thanks to the negotiation process between Commission and 
Member States. at the programming stage. Moreover, coherence was improved thanks to the 
fulfilment of ESF ex-ante conditionalities, which lead to structural reforms, although this also 
caused some delays in ESF implementation (e.g. in Romania).  

ESF employment, social inclusion and education and training operations are internally coherent 
(between  different  thematic  objectives)  and  between  investment  priorities  within  each 
thematic  objective.  ESF  support  to  social  inclusion  has  a  high  potential  overlap  with  some 
support  to  employment  operations  (e.  g.  promoting  sustainable  and  quality  employment  and 
supporting  labour  mobility),  and  actions  that  could  have  been  programmed  under  other 
thematic  objectives  may  have  been  programmed  under  social 
inclusion  to  meet  the 
requirement that Member States allocate at least 20% of ESF funds to this thematic objective. 
However,  actual overlaps are  low  due  to  the  different  target  groups  addressed  and  the  more 
holistic approach taken under social inclusion.  

Coherence with other EU funding instruments in related fields is more mixed: while  there  is 
often good coherence in the legal texts, evidence of synergies in implementation varies, and a 
variety of challenges in combining different funds were identified. ESF employment operations 
show  a  good  level  of  complementarity  with  ERDF  thematic  objective  3  (competitiveness  of 
SMEs).  However,  integration  with  ESF  can  be  improved.  Employment  operations  also  show 
significant  synergies  with  other  EU-funded  programmes,  namely  EaSI,  the  European 
Globalisation Fund and the Asylum Migration and Integration Fund. 

ESF social inclusion support was also combined with other EU funds, the most common EU fund 
being the ERDF. While strong complementarities were found among examples of OPs that draw 
on ESF and other EU funds, the complexities involved in the implementation of multi-fund OPs 
imply delays in implementation. 

In  education  and  training  operations,  good  examples  of  coherence  in  implementation  were 
nonetheless found in particular with ERDF and Erasmus+, however only to a very limited degree 
with other funding instruments (e.g. Marie Skłodowska-Curie actions). 

ESF operations are well aligned with national and regional policies in the Member States. Such 
complementarity can  take  the  form of  ESF  having  a supporting  role  for  local  policies  or  filling 
policy gaps. However, alignment and complementarity with national policies could be improved 
concerning  operations  supporting  horizontal  themes  such  as  social  innovation  or  regarding 
tertiary education ESF operations. 

69 

 
ESF operations have generated  a good degree of EU added value across all dimensions,  with 
substantial evidence pointing to volume, scope, role and process effect., There were important 
volume  effects  in  terms  of  the  increased  participation  of  target  groups  (e.g.  teachers  and 
learners in education operations), the number of actions, and widening the geographical scope 
and scale of implementation of key programmes. Therefore ESF support is playing a pivotal role 
in  funding  employment,  social  inclusion  and  education  policies  that  complement  national 
efforts.  

Scope  effects  were  also clear,  particularly  for  broadening  actions  to wider  target  groups  that 
would  have  not  been  supported  otherwise,  including  many  disadvantaged  groups  (e.g.  social 
inclusion  operations  supported  Roma  children  and  children  with  special  educational  needs, 
people with disabilities, people facing housing exclusion and refugees, whereas education and 
training  ESF  support  covered  Roma,  older  workers  and  low-skilled  adults).  There  were  also 
scope effects in addressing policy areas which were not previously high priorities on the agenda 
(e.g. inclusive education, adult learning and, non-formal education). 

ESF  operations  also  played  an  important  role  in  supporting  the  transfer  of  ideas  and  the 
introduction  of  innovations  and  structural  reforms.  In  employment  operations  the  ESF  has 
fostered  innovation  especially  in  the  support  of  harder-to-reach  groups,  including  those  with 
disabilities,  and  those  from  remote  rural  areas  and  there  is  evidence  of  mainstreaming,  with 
innovative  approaches  being  adopted  by  Member  States  as  part  of  their  labour  market 
programmes (e.g. the ESF has helped to raise the profile of gender issues and promoted social 
innovation). In social inclusion innovation has focused on the use of individualised and targeted 
approaches, especially toward migrants, older workers and women. The ESF also supported the 
improvement  of  existing  national  frameworks  (e.g.  through  the  establishment  of  monitoring 
and  coordination  mechanisms,  design  and  implementation  of  integrated  approaches  to 
combating social exclusion), and allowed for the testing of new partnerships. In education and 
training,  significant  innovation  effects  were  recorded,  with  the  use  of  digitisation  and  new 
pedagogies, as well as structural reforms such as the development of local or regional education 
strategies  or  the  rolling-out  of  new  educational  approaches  and  strategies  nationwide.  Role 
effects could however potentially be further enhanced through a greater focus on peer learning, 
transnational cooperation, visibility of ESF support and, dissemination of results of ESF-funded 
operations.  

Many  process  effects  from  ESF  operations  are  also  in  evidence.  Crucially,  the  ESF  has  helped 
build  effective  delivery  capacity  in  Member  States  in  terms  of  programme  and  cost 
management  and  monitoring  systems,  and  awareness  and  knowledge  of  target  groups. 
Relevant examples include improved administrative capacity and knowledge in the design and 
delivery of services promoting social inclusion, improvements to governance and organisation of 
the  offer  of  education  and  training,  and  establishing  closer  links  and  cooperation  between 
schools and training providers with other stakeholders at local/regional or national level. 

70 

 
6.1  Lessons learned 

The supporting studies provided evidence on those factors that seem to lead to better use of th 
ESF funding and on the main points for improvement117. 

Operational  programme  strategies  should  be  integrated  and  embedded  into  national  (or 
regional)  strategies.  Shared  management  entails  that  the  actual  day-to-day  management  of 
operational  programmes  is  entrusted  to  official  national  (or  regional)  bodies.  Performance  is 
enhanced  by  a  strong  strategic  approach  which  informs  the  prioritisation  and  targeting  of 
resources to address specific needs. This is clearly the case when the ESF is used to increase the 
volume of support offered. It is also the case when innovative approaches are tested before, if 
successful,  being  mainstreamed.  All  thematic  objectives  included  actions  intended  to  impact 
structures and systems. Success requires the active support from the national (or regional and 
local) actors concerned. 

Institutional capacity, at all levels of implementation, and regulatory requirements, are linked 
and  strongly  influence  effectiveness.  Managing  authorities  have  to  lead  strong  coordination 
processes  involving local and regional authorities  and key partners such as NGOs, civil society 
organisations  and  social  partners.  Beneficiaries  in  turn  must  have  the  capacity  to  deliver  ESF 
operations on the scale required. They may need to rely on the managing authorities to build 
the capacity of their staff on how to access, implement and monitor the ESF. The issues most 
commonly cited as placing a high demand on the capacity are the setting up and operation of 
the monitoring and control systems, the definition and roll-out of simplified cost options, and 
the application of state aid regulations.  

Strong  partnerships  are  required.  Greater  cross-sectoral  cooperation  between  education, 
health,  labour  market,  social  services  and  services  dealing  with  territorial  planning  and 
development  should  be  fostered  at  all  levels  (EU,  national,  regional)  to  address  the  complex 
needs of target groups. This is especially important in the delivery of personalised operations or 
integrated  support  of  interventions  that  need  to  follow  up  on  each  other  and  cannot  be 
conducted in parallel. Local actors, civil society organisations and NGOs, often play a crucial role 
in engaging with harder to reach target groups. Yet the partnership principle is not always fully 
implemented  in  practice,  with  stakeholders  pointing  out  some  examples  of  superficial 
consultation  processes  that  hindered  meaningful  engagement  of  partners  across  all  areas  of 
programme  design,  implementation  and  monitoring.  Moreover,  having  a  high  number  of 
partners  makes  more  demands  on  the  institutional  capacity  of  the  managing  authorities  and 
may  therefore  come  to  be  a  constraint  on  efficient  delivery.  Managing  authorities  may  be 
sometimes avoid these local actors, civil society organisations and NGOs, precisely because of 
the perceived demand on their institutional capacity. Diversifying the stakeholder groups that 
are  consulted  remains  an  important  area  for  improvement,  as  public  authorities,  especially 
from the national and regional levels, are often overrepresented. 

The correct alignment of operations to the needs of the target group is another success factor. 
It  is  the  form  and  suitability  of  the  support  offered  to  participants  that  determines  its  net 

117  The supporting studies provide additional lessons learned and points for improvement specific for each 

ESF thematic objective. 

71 

 
                                                           
effects, irrespective of where it is implemented and where the money comes from, ESF or non-
ESF. Overall, there is strong evidence that the most important needs of the main target groups 
of  each  ESF  thematic  objective  were  taken  on  board.  Nevertheless,  ESF  operations  did  not 
always  fully  succeed  in  reaching  and  addressing  the  needs  of  some  specific  target  groups, 
particularly the most disadvantaged ones. It is the precise identification of a target group that 
allows account  to be  taken of the  additional efforts required to reach it, rather than reaching 
those  for  whom  a  result  can  be  generated  more  easily.  The  alignment  of  operations  to  the 
needs  of  the  target  group  does  not  need  to  take  place  in  the  operational  programme  stage 
itself.  Some operational programmes attributed their flexibility to a broad definition of target 
groups, at the planning stage, narrowed down during implementation.  

ESF-supported actions bring about changes in number of areas including individual behaviour 
and/or  social  roles  well  beyond  employment,  labour  mobility  and  education  outcomes  but 
these  are  poorly  documented.    Outcomes  such  as  qualifications  can  be  captured  and 
adequately analysed by using the existing set of indicators. However, these soft-outcomes are 
quite  widespread  among  participants,  difficult  to  capture  in  quantitative  terms  and  thus 
underreported.  Efforts  could  be  made  to  improve  the  systems  to  this  end,  including  through 
experimenting with “distance travelled” approaches.  

More should be done to promote the visibility of ESF support and mutual learning. Adopting a 
greater focus on dissemination of ESF results and good practices, with a robust consideration of 
success factors, challenges and transferability – at EU, national, regional and operation level – 
has  potential  to  inspire  other  potential  beneficiaries,  and  ensure  that  lessons  learned  in 
addressing  key  challenges  and  reaching  specific  target  groups  are  better  shared  among  the 
community of practitioners. A focus on dissemination would also ensure better visibility of the 
contributions of ESF across the EU, and could foster greater absorption and improved funding 
applications.  As  seen  in  the  supporting  studies,  there  was  evidence  of  non-awareness  of  the 
opportunities offered by ESF support.  

Finally, in order to provide robust evidence on the impacts of the ESF-financed interventions, 
more timely availability of data is crucial. No proper cost effectiveness analysis could be done 
at  this  stage  due  to  incompleteness  of  data,  as  reporting  on  outputs,  results  and  costs  is  not 
synchronised  and  operations  are  not  completed.  Moreover,  the  counterfactual  impact 
evaluations require detailed data not only on the participants but also on the non-participants, 
and thus data collection should be planned well in advance. 

72 

 
 
ANNEX 1: PROCEDURAL INFORMATION 

Lead DG, Decide Planning/CWP references 

This evaluation was carried out by DG EMPL, as a result of three different initiatives related to 
the ESF support to policies in the areas of:  

• 

• 

• 

Employment  and  labour  mobility  (thematic objective  8)  under  the  Decide  number 
PLAN/2018/3178. The initiative was published in July 2018. 
Social inclusion (thematic objective 9) under the Decide number PLAN/2018/3177. 
The initiative was published in December 2018. 
Education  and  training  (thematic  objective  10)  under  the  Decide  number 
PLAN/2018/3147. The initiative was published on December 2018. 

Organisation and timing 

2.1. 

Thematic objective 8:  

An Inter-Service Steering Group (ISSG) was created in July 2018. The invitation launched on 17 
July 2018 was addressed to the following DGs: AGRI, BUDG, EAC, GROW, HOME, JRC, JUST, 
MARE, REGIO, RTD, SG and SJ.  

The timing of the evaluation was as follows: 

20 July 2018 - 
17 August 2018 
8 August 2018 - 
28 September 2018 

19 October 2019 

4 March 2019 

19 March 2019 

17 April 2019 

11 September 2019 

13 January 2020 
12 February 2020 

18 February 2020 

September 2020 

Publication of the roadmap and feedback period 

Two rounds of consultation of ISSG via written procedure 

Request for services for the external study supporting the 
evaluation 
Signature of the contract for the supporting study with the 
consortium led by Fondazione Giacomo Brodolini  
1st meeting of the ISSG: kick-off meeting for the supporting 
study 
2nd meeting of the ISSG: draft inception report of the supporting 
study 
3rd meeting of the ISSG: draft interim report of the supporting 
study 
Final deliverable of the Rhomolo modelling 
Draft final report of the study 
4th meeting of the ISSG: draft final report of the supporting 
study 
Written consultation on the draft staff working document 

73 

 
 
 
 
2.2. 

Thematic objective 9:  

An Inter-Service Steering Group (ISSG) was created in December 2018. The invitation was 
addressed to the following DGs: AGRI, BUDG, EAC, GROW, HOME, JRC, JUST, REGIO and SG.  

The timing of the evaluation was as follows: 

19 December 2018 - 
16 January 2019 
17 December 2018 

d1 February 2019 

16 May 2019 

17 May 2019 

26 June 2019 

14 November 2019 

28 February 2020 
5 March 2020 
September  2020 

Publication of the roadmap and feedback period 

1st meeting of the ISSG 
Request for services for the external study supporting the 
evaluation 
Signature of the contract for the supporting study with the 
consortium ICF-  
2nd meeting of the ISSG: kick-off meeting for the supporting 
study 
3rd meeting of the ISSG: draft inception report of the supporting 
study 
4th meeting of the ISSG: draft interim report of the supporting 
study 
Draft final report of the supporting study 
Final deliverable of the Rhomolo modelling 
Written consultation on the draft staff working document 

2.3. 

Thematic objective 10:  

An Inter-Service Steering Group (ISSG) was created in January 2019, which included the 
following DGs: EMPL, BUDG, EAC, HOME, JRC, REGIO and SG.  

The timing of the evaluation was as follows: 

21 December 2018 -  
18 January 2019 
25 January2019 

28 February 2019 

 6 June 2019 

12 June 2019 

12 July  2019 

20 November 2019 

 6  April 2020 
2 June 2020 
September  2020 

Publication of the roadmap and feedback period 

1st meeting of the ISSG 
Request for services for the external study supporting the 
evaluation 
Signature of the contract for the supporting study with the 
consortium ECORYS-Ismeri  
2nd meeting of the ISSG: kick-off meeting for the supporting 
study 
3rd meeting of the ISSG: draft inception report of the supporting 
study 
4th meeting of the ISSG: draft interim report of the supporting 
study 
Draft final report of the supporting study 
Final deliverable of the Rhomolo modelling 
Written consultation on the draft staff working document  

74 

 
 
 
Exceptions to the better regulation guidelines 

N/A 

Consultation of the RSB (if applicable) 

N/A 

Evidence, sources and quality 

Each of the three initiatives were based on three main instruments: 

• 

• 

• 

An  EU-wide  public  consultation,  which  was  carried  out  and  analysed  with  the 
assistance of the external team of experts. 
A supporting study carried out by an external team of experts, through a contract 
(VC/2019/0089  for  thematic  objective  8,  VC/2019/0268  for  thematic  objective  9 
and  VC/2019/0351  for  thematic  objective  10)118,  through  DG  EMPL  Multiple 
Framework  Contract  VT/2016-027  for  the  provision  of  services  related  to  the 
implementation of the better regulation guidelines. The supporting study combined 
the  results  of  the  two  other  tools  together  with  additional  work,  such  as  desk 
research,  public  consultations,  case  studies,  interviews  with  stakeholders,  focus 
groups and Delphi surveys (see Annex 2 for more details).  
An  estimate  of  macroeconomic  effects  of  ESF  supported  operations  using  the 
Dynamic Computable General Equilibrium RHOMOLO of the Joint Research Centre, 
through an Administrative agreement between DG EMPL and the JRC-Seville. 

All three Inter-Service Steering Groups assessed the quality of the final report of the external 
contractors as satisfactory. 

The conclusions and findings of the evaluation are considered as robust, within the limitations 
and the mitigating measures described in section 4 and annex 3 of this staff working document. 

118  The links to the reports are: 

For TO8: https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8330&furtherPubs=yes  

For TO9: https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes  

For TO10:  https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8346&furtherPubs=yes  

75 

 
 
                                                           
ANNEX 2: STAKEHOLDER CONSULTATION 

1. 

INTRODUCTION 

This  synopsis  report  outlines  the  consultation  activities  that  were  organised  as  part  of  the 
labour  mobility  (excluding  youth 
evaluation  of  ESF  support  to  (1)  employment  and 
employment),  (2)  social 
inclusion  and  (3)  education  and  training  for  the  2014-2020 
programming period and presents the main findings.  

2. 

CONSULTATION STRATEGIES 

2.1. 

Objectives 

To ensure  transparency  and involve  stakeholders as much as possible, the consultations were 
organised in line with the standards and methods set out in the better regulation guidelines. As 
set out in their respective roadmaps and strategies, all consultation activities aimed to collect 
views  and  evidence  from  stakeholders  on  the  visibility,  outreach,  usefulness,  relevance  and 
effectiveness of the ESF support. 

2.2. 

Consultation stakeholders 

When  identifying  the  stakeholders  to  consult,  consideration  was  given  to  the  impact  of  ESF 
support  (direct,  indirect  and  potential)  on  individuals  and  organisations,  and  their  degree  of 
involvement.  The  stakeholders  targeted  by  the  consultation  were  organisations  or  individuals 
that: 

• 
• 
• 
• 

had or might have participated in the operations; or 
had or might have run or been involved in running the operations; or 
had an interest in social inclusion operations funded under the ESF; or 
had expertise in the subject. 

Stakeholders  involved  in  managing  the  funds  were  consulted  regularly.  Member  State 
representatives involved in ESF monitoring and evaluation were informed of the progress of the 
supporting  study,  the  interim  findings  as  well  as  the  final  findings  and  conclusions  at  the 
Evaluation Partnership meetings held between February 2019 and June 2020.  

The draft reports and individual country factsheets (both in the original English version and in 
the  national  languages)  were  circulated  to  the  evaluation  partnership  members  and  desk 
officers working for the Commission’s Directorate-General for Employment, for comments. The 
comments  received  were  mostly  to  flag  inconsistencies  in  data  related  to  individual  Member 
States, which were subsequently corrected. All the supporting studies’ deliverables of (reports, 
country  factsheet  and  related  annexes  for  example  on  methodology  and  mapping)  were 
uploaded on ‘CIRCABC’, the ESF Evaluation Partnership’s online workspace 119.  

119  https://circabc.europa.eu/ui/group/58263cb1-f430-4a10-88c7-eaf436177c89/library/ba675037-3cd1-

402d-8170-57c4a269ad39?p=1&n=10&sort=modified_DESC  

76 

 
 
                                                           
3. 

THEMATIC OBJECTIVE 8 - EMPLOYMENT AND LABOUR MOBILITY 

3.1. 

The consultation strategy 

3.1.1.  Roadmap 

The evaluation roadmap was published on the Commission’s ‘Have your Say’ portal and made 
available  for  public  feedback  between  20  July  and  17  August  2018120.  No  contributions  were 
received in this context. 

3.1.2.  Consultation stakeholders  

The following groups were consulted: 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

identify  efficiency  and  delivery 

Participants:  individuals  supported  by  the  European  Social  Fund  operations  were 
consulted to obtain a direct insight into: (i) the extent to which the operations met 
the  “integration  in  the  labour  market  and  mobility”  objective,  and  (ii)  success 
factors that cannot be directly identified through the monitoring mechanisms. 
Organisations  involved  in  the  management  of  operational  programmes  such  as: 
Member  states,  managing  authorities  at  national  and/or  regional 
level, 
intermediate bodies and, social partners represented in the monitoring committee; 
Organisations  involved  in  the  delivery  of  ESF  funded  operations  such  as 
beneficiaries or project partners including public bodies, NGOs and local authorities. 
Their 
in  the 
feedback  helped  to 
implementation of ESF operations. 
Organisations  not  directly  involved  in  the  operations  but  that,  representing  the 
people targeted by the ESF - more specifically people at risk of social exclusion - and 
can therefore provide a broader view of the needs and of issues faced by recipients 
and potential recipients. 
Academic and research bodies involved the field of employment and mobility, that 
could  provide  insight  as  to  the  relevance,  effectiveness,  efficiency,  coherence  or 
sustainability of measures; 
Organisations representing employers, that can give feedback on the  factors that 
play  a  role  in  the  successful  integration  of  end  recipients,  and  young  people  in 
general, in the labour market;  
General  public:  any  individual  or  organisation  that  could  provide  useful  views  on 
how  the  EU  can  support  employment  and  labour  mobility  and  the  other  aims  of 
thematic objective 8 (TO8). 

issues 

3.1.3.  Consultation methods and tools 

To  adequately  reach  these  stakeholders,  a  range  of  consultation  activities  and  methods  were 
used  namely,  a  public  consultation, over  60  targeted  interviews  (including  interviews  with  EU 
and  national  level  stakeholders,  10  case  studies121  which  also  included  consultations  with  key 
stakeholders  in  the  form of  in-depth  interviews),  and  a  validation  process  on  the  evaluation’s 

120  https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/1862-Evaluation-of-the-
support-to-employment-and-mobility-by-the-European-Social-Fund/public-consultation  

121  Belgium, Germany, Greece, Spain, France, Malta, Portugal, Italy, Poland and Slovakia. 

77 

 
                                                           
draft findings involving experts working together in focus groups in four Member States (Spain, 
Croatia, Luxembourg & Romania) and one at EU level.  

Table 1: Overview of the stakeholders reached through each consultation tool/method 

Type of stakeholder 

Participants 
Organisations involved in the management of 
operational programmes 
Organisations involved in the delivery of the 
operational programmes 
Organisations not directly involved in the 
implementation of the operational programmes 
Academic and research bodies involved the field 
of employment and mobility 
Organisations representing employers 
General public 

Targeted 
interviews 

Public 
consultation 
x 

Validation focus 
groups 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 
x 

x 

x 

x 

x 

x 

3.2. 

Specific consultations/activities 

3.2.1.  The public consultation 

3.2.1.1. Description of the activity 

The online public consultation ran from 14 October 2019 to 6 January 2020 and took the form of 
a survey. The questionnaire comprised four strands of questions (excluding profiling and closing 
sections), structured around the main evaluation questions and including both open-ended and 
closed questions. Each strand was customised to one specific group of respondents. 

3.2.1.2. Stakeholders  

The public consultation had a separate section for each of the following groups of respondents: 

• 

• 

• 
• 

Group A: EU citizens who have benefited or are benefiting from ESF support (‘‘ESF 
participants’’); 
Group  B:  organisations  that  are  or  were  involved  in  ESF  and/or  received  support 
(i.e.  managing  authorities,  intermediate  bodies,  beneficiaries,  and  social  partners) 
that supposedly  have  a direct  and detailed knowledge  of ESF, although to varying 
degrees,  depending  on  their  specific  function/role  in  the  ESF  (‘organisations 
involved’); 
Group C: EU citizens who did not benefit from ESF support (‘other citizens’); 
Group  D:  organisations  not  involved  in  ESF  and organisations  that  did  not  declare 
their status or responded ‘I don’t know’ to the question ‘Have you been involved in 
or  have  you  benefited  from  actions  promoting  employment  and  labour  mobility 
supported by the European Social Fund?’ (‘organisations not involved’). 

78 

 
 
 
 
 
 
Figure 1: Breakdown of respondents by MS and group – TO8 

Source: Final report of the public consultation on the evaluation of the EU support to employment and 
labour mobility 

The  public  consultation  received  541  responses.  There  was  an  unbalanced  geographical 
distribution  with  four  countries  (Bulgaria,  Croatia,  Germany  and  Italy)  responsible  for  around 
60% of responses received. Almost 60% of the 541 respondents were women and around 40% 
men.  Of  the  541  respondents,  53  (9.8%)  were  from  Group  A  (ESF  participants);  238  (43.9%) 
were  from  Group  B  (organisations  involved);  140  (25.9%)  were  from  Group  C  (other  citizens) 
and  110  (20.4%)  were  from  Group  D  (organisations  not  involved).  Of  this  Group  D,  79 
organisations declared they were not involved in ESF. The remaining 31 either did not answer 
the  relevant  question  or  said  they  did  not  know.  Therefore,  almost  half  of  the  public 
consultation respondents were organisations that are or were involved in ESF and/or received 
support. However, relatively few ESF participants responded to the public consultation. Figure 1 
above summarises the respondents by country of reference and group.  

79 

 
 
3.2.2.  Member State level interviews and focus groups 

The fieldwork involved semi-structured interviews in all Member States and case studies in 10 
Member States, covering 20 operational programmes and all TO8 investment priorities (except 
youth employment). 

Four focus groups were held in addition to the EU-level focus group. The types of stakeholders 
consulted  were  managing  authorities,  beneficiaries, 
socioeconomic  partners  and 
evaluators/researchers.  The  interviews  and  case  studies  were  used  as  the  main  source  of 
information to answer the evaluation questions and feed into case study reports.  

3.2.2.1.  EU level validation focus group 

The  EU-level  focus  group  took  place  on  3  February  2020  in  Brussels.  It  aimed  to  discuss  key 
issues in some of the evaluation questions with a group of experts involved in European-level 
policy  debates  on  employment  related  matters  and  labour  mobility,  as  well  as  with 
(19)  representatives  participated. 
representatives  of  managing  authorities.  Nineteen 
Discussions were facilitated by five consultants supplied by the contractor. 

3.3. 

Overview of results of consultation activities 

3.3.1.  Effectiveness 

For respondents that received ESF support, the most common benefit related to improvement 
of the quality of the labour market position and help in looking for a job (approximately one in 
five mentioned each of these), whereas very few said it helped with achieving a better work-life 
balance  and  entering  or  going  back  to  education  and  training.  Respondents  were  also  asked 
about  their  current  and  past  labour  market  situation.  Before  ESF  support,  41.5%  of  53 
responding ESF participants were employed and after ESF support the employment rate rose to 
84.9%. Likewise, 20.8% were unemployed before support and 7.5% after support.  

Overall, most participating organisations agreed that ESF operations were effective in achieving 
TO8 objectives, in particular: i) in promoting access to employment for jobseekers and inactive 
self-employment, 
respondents  agreed); 
people 
entrepreneurship  and  business  creation  (81.1%);  and  promoting  equality  between  men  and 
women (77.3%). 

ii)  promoting 

(89.9%  of 

the  238 

The objectives that were the least mentioned as being effective were the promoting work-life 
balance and active and healthy ageing.  

For  helping  jobseekers  and  inactive  people  enter  the  labour  market,  most  respondents 
mentioned  training  and  support  in finding a  job  as the most  effective  actions. On support  to 
self-employment,  entrepreneurship  and  business  creation,  organisations  mostly  indicated 
financial support/microcredit. To support  gender equality in the  labour market, organisations 
considered  flexible  work  arrangements  and  access  to  quality  childcare  to  be  the  most 
successful  actions,  whereas  less  than  half of them  considered training,  gender  mainstreaming 
and  awareness  campaigns  to  be  successful.  To  promoting  active  and  healthy  ageing,  flexible 
work  arrangements  and  health  and  safety  at  the  workplace  were  identified  as  effective, 
although the number of respondents not knowing/not answering is particularly high.  

80 

 
Respondents consider career guidance and vocational training of individuals as most successful 
measure  to  help  workers  and  companies  adapt  to  change,  whereas  awareness  raising 
campaigns were mentioned by less than a quarter of them. On the promotion of employment 
support  systems,  strengthening  cooperation  between 
institutions  and 
stakeholders,  improvement  of  public  employment  services,  and  training  of  staff  were 
considered  to  be  the  most  successful  measures  by  the  highest  proportion  of  respondents, 
whereas  performance  management  and  transnational  partnership  were  mentioned  least. 
Finally,  on  the  promotion  of  geographic  and  occupational  mobility  of  workers,  organisations 
considered language training, recognition of qualifications and validation of competences and 
acquired skills as successful. 

labour  market 

On effective support to target groups, the majority of responding organisations believe that ESF 
is  successful  in  supporting  job  seekers  and  inactive  people  (77.6%  of  the  348  respondents 
agreed  or  strongly  agreed),  and  long-term  unemployed  (64.7%  agreed).  More  than  half  also 
believe that ESF effectively supports people leaving education without a qualification (58%), and 
people at risk of social exclusion (56.5%). They seem to be more sceptical about ESF ability to 
support migrants or individuals with a foreign background (less than half of respondents believe 
ESF  was  successful  in  supporting  them:  49.2%  out  of  348  respondents)  or  people  in  remote 
areas  (48.9%).  Regarding  people  affected  by  poverty  and  homeless  people,  a  number  of 
respondents  said  they  did  not  know  (33.4%  and  46.1%  respectively).  Considering  this,  the 
assessment  seems  to  be  particularly  negative  about  the  ESF’s  effectiveness  in  reaching 
homeless people with 36.6% saying it did not reach them vs 17.4% saying it did). Respondents 
referring to Italy and Croatia tended to give a less positive assessment of the success of ESF in 
addressing different target groups compared to the other countries. 

Overall,  most  respondents  seem  to  consider  social  media  campaigns  as  the  best  channel  of 
information  (70.4%  of  541  the  respondents),  followed  by  information  and  awareness  raising 
events  (44.5%).  The  least  popular  channels  seem  to  be  the  distribution  of  flyers  and 
advertisements in newspapers. Most respondents said they learnt about the support measures 
they benefited from through employment services and  employment info centres  (43.4%), and 
less than a fifth through social media. 

Regarding  potential  hindering  factors  for  individuals,  information  and  ability  to  reach  out  to 
potential participants  seemed to be  the main factors. In fact,  individual respondents who did 
not  take  part  in  ESF  support  mostly  said  that  they  were  not  aware  of  it,  followed  by 
respondents saying they did not know where to ask for information.  

The  focus  groups  discussions  highlighted  the  fact  that  the  ESF  may  not  be  very  visible  to  the 
public,  as  it  is  hard  to  ‘show’  the  contribution  of  ESF  given  that  it  has  an  essentially  human 
dimension,  as  opposed  to,  the  ERDF  and  the  Cohesion  Fund  for  instance,  which  deal  with 
infrastructure.  

Organisations were also asked about factors that may hinder the implementation of actions and 
approximately three quarters of them pointed to administrative burden for beneficiaries (76% 
of  the  238  respondents)  and  managing  authorities.  External  factors,  such  as  structural 
problems (i.e. lack employment opportunities and low education and skills level of participants) 
are also considered to play an important role (by 71.3%). Some organisations specified the types 

81 

 
of  obstacles  they  have  encountered  in  a  related  open  question.  A  total  of  33  responses 
indicated cumbersome bureaucracy and lengthy procedures as the significant factors hampering 
the effectiveness of ESF procedures, which supports the responses given in the closed question. 
In addition, nine respondents mentioned that targets and measures are not carefully tailored to 
needs, especially when it comes to people facing multiple barriers and discrimination. In the EU-
level focus groups, it was highlighted that both managing authorities and beneficiaries continue 
to face a high administrative burden (because of both the ESF and CPR Regulations and the so 
called  ‘gold  plating’  of  rules  at  national  level).  This  is  particularly  limiting  for  smaller 
beneficiaries (e.g. NGOs representing vulnerable individuals) and discourages them to access to 
ESF funds. Issues with inter-institutional cooperation were also highlighted especially in the case 
of integrated pathways.  

Regarding  success  factors,  more  than  nine  in  ten  responding  organisations  agreed  that 
personalised  services  and  opportunities  help  promote  participation  (95.3%  out  the  238 
respondents), followed by services that are in line with labour market needs (92.7%), flexibility 
of duration (92.1%), flexibility in the way actions are implemented (91.5%) and financial benefits 
for participants (89%).  

In a related open question, 12 of the 40 that responded mentioned the importance of creating 
partnerships,  believing  that  they  help  bringing  together  different  experiences  to  help  design 
meaningful solutions that address different needs, as well as tailored and holistic approaches 
to  support  end  users.  Ten  respondents  also  mentioned  the  quality  of  staff  and  dedication  of 
grant recipients and delivery partners. 

The focus groups participants considered that the net effect of different measures, should not 
to be judged by the seemingly low employment rates after the end of support, as this might not 
reflect  the  strong  net  effects  for  individuals  who  are  further  away  from  the  labour  market.  A 
discussion  on  target  groups  highlighted  the  importance  of  tailored  approaches,  focusing  too 
much on target groups may cause certain individuals to be overlooked, notably those that do 
not fit under pre-defined categories but are nonetheless in need of support.  

Regarding  employers,  it  was  mentioned  that  it  could  be  difficult  to  get  employers  to  use 
financial incentives unless they are  aligned to the actual labour market needs.  

It  was  also  mentioned  that  ‘standard’  employment  measures  pose  fewer  obstacles  to 
implementation and that ‘newer’ types of measures might take more time to implement. 

The importance of integrating the support process between regional and national level public 
employment  services  was  mentioned  as  an  important  ingredient.  Regional  and  national  level 
public employment services sometimes serve the same clients, but refrain from sharing detailed 
information to avoid the risk of duplication.  

3.3.2.  Efficiency 

Most respondents consider training and education to be a cost-efficient measure (77.4% of the 
541 respondents), followed by basic skills training and incentives for employers (73.6%). Open 
include  reducing  bureaucracy  and  simplifying 
suggestions  to 
procedures  to  reduce  the  cost  of  operations,  increasing  the  flexibility  of  budget  and 

improve  efficiency  point 

82 

 
procedures, and involvement of final beneficiaries earlier and more deeply in the design of the 
measures. 

Administrative  arrangements  can  have  a  bearing  on  the  efficiency  of  operations  and 
organisations  were  asked  their  opinion  on  them.  Approximately  two-thirds  considered  the 
communication  arrangements  appropriate.  A  smaller  share  (50%-60%),  considered  the 
evaluation and project selection procedures, the application of simplified cost options, project 
implementation  and  the  reporting  and  monitoring  arrangements  to  be  appropriate.  Some 
respondents  declined  to  answer  ranged  (12.9%  on  reporting  and  monitoring  and  24.6%  on 
audit) on reporting and monitoring in particular, there seems to be room for improvement since 
33.8% of the 348 respondents said they were burdensome or insufficient.  

Opinions  on  the  audit  and  management  and  control  system  arrangements  were  even  more 
negative.  Less  than  half  of  the  respondents  considered  them  to  be  appropriate.  However, 
organisations  involved  in  ESF  –  which  arguable  have  a  more  informed  opinion  –  were  more 
likely to consider the arrangements appropriate. Among these organisations, the highest share 
of  respondents  saying  that  arrangements  were  burdensome  were  referring  to  for  the 
management and control system and reporting and monitoring (38.7% and 31.1% of the 238 
respondents  respectively).  Among  the  four  countries  with  the  higher  response  rates, 
respondents  referring  to  Germany  and  Italy  seem  to  be  relatively  more  negative  about  the 
management and control system compared to those in Croatia and Bulgaria and in general the 
rest of respondents. 

3.3.3.  Relevance 

Overall, respondents rated the usefulness of the various actions positively. In particular, almost 
90%  considered  that  actions  to  place  a  person  in  a  job  to  be  useful  (88.9%  out  of  the  541 
respondents).  Actions  aimed  at  increasing  the  human  capital    strengthen  the  workforce 
(improving the quality and welfare of human resources in companies and organisations, and 
helping a person perform better in an existing job) were deemed useful, by approximately 80% 
of the respondents. Actions promoting organisational change and increasing equality between 
men and women received a less favourable assessment, 17% and 19.2% respectively said that 
they  have  little  or  no  usefulness.  Organisations  involved  in  ESF  and  ESF  participants  seem  to 
have a slightly more positive opinion on the usefulness of these actions. In addition, all actions 
were generally considered to be useful by most  respondents in the  different countries. When 
asked  about  additional  useful  actions,  respondents  mentioned  further  enhancing  the  VET 
systems and improving training modules, developing better targeting mechanisms, and, a more 
coherent EU-wide qualification scheme. 

Approximately 70% of the 348 responding organisations believe that the ESF was able to adapt 
to  the  evolving  socio-economic  conditions  and  participants’  needs.  A  higher  proportion  of 
positive  responses  was  received  for  Italy  and  Bulgaria  and  a  lower  proportion for  Croatia  and 
Germany.  A  few  respondents  indicated  that the  ESF was  flexible  in  terms of  adjusting  targets 
and adapting to the changing needs of the target groups and less flexible in terms of excessive 
procedures and requirements and cumbersome bureaucracy. 

83 

 
According  to  focus  groups  discussions,  three  types  of  support  (not  necessarily  limited  to  ESF 
support) can be useful for labour mobility: namely support for:  

• 

• 

• 

The system and the network, including cross-cutting activities at EU-level, technical 
systems for information sharing, etc.  
Service  providers,  to  support  activities  such  as  recruitment,  outreach  activities, 
information to employers, matching CVs with jobs, etc. 
Individuals  and  employers,  for  example  regarding  relocation,  recognition  of 
competences, language trainings or the creation of a ‘welcome package’. 

Focus group members  also made  the  point  that the ESF’s support  to labour mobility through 
ESF  seems  limited.  Although  some  countries  use  ESF  funds  for  the  EURES122  network,  the 
network  is  mainly  supported  through  EaSI123  funds.  The  fact  that  ESF  has  traditionally  had  a 
national  focus  also  means  that  a  somewhat  lower  level  of  attention  has  been  paid  to  cross-
border activities. 

It was suggested that it could be useful to encourage ‘return mobility’ and get workers to move 
back to their country of origin with the new skills they acquired abroad. It was noted that third 
country nationals have the same needs (in terms of language for example), so the same services 
could be provided to all. However, funding streams are often separate for EU and third country 
nationals, and more synergies could be exploited 

3.3.4.  Coherence 

In this section, respondents often answered that they did not know or did not have an opinion. 
For  example  when  asked  about  the  ESF’s  coherence  with  existing  national  and  regional 
programmes, 40.8% and 72.7% of the 541 respondents  respectively did not  have  an opinion). 
Respondents  were  slightly  more  likely  to  have  an  opinion  on  the  European  Regional 
Development  Fund  (ERDF),  but  less  than  half  of  them  (45.1%)  considered  the  ERDF  to  be 
complementary with the  ESF, with 41.3% saying they did not  know  and only 8.9%  saying that 
they were not aligned.  

Focus  groups  participants considered  that the  coherence  and  complementarity  of  ESF’s  TO8 
with  other  funds  had  improved,  particularly  with  the  Youth  Employment  Initiative,  the 
European Globalisation Adjustment Fund (EGF)  (where the ESF plays a ‘preventive’ action and 
the  EGF  a  ‘corrective’  one),  and  the  AMIF  and  also  vis  à  vis  labour  mobility.    They  also 
mentioned  high  complementarity  between  the  thematic  objectives  and  synergy  with  the 
European semester cycle of economic policy coordination and particularly the country-specific 
recommendations.  However, they also reported difficulties in the concrete integration of funds. 

3.3.5 

EU added value 

Approximately half (47.7% out of the 541 respondents) believed that funding provided by the 
ESF  would  not  otherwise  have  been  available  (the  volume  effect).  A  further  28.5%  said  that 

122  For more info on EURES check the home page at: https://ec.europa.eu/eures/public/en/homepage  

123  EU  Programme  for  Employment  and  Social  Innovation  (EaSI).  More  information  is  available  at: 

https://ec.europa.eu/eip/ageing/funding/EaSI_en  

84 

 
                                                           
without  EU  support  there  would  be  less  attention  paid  to  unemployed  people  (the  scope 
effect).  Finally  10.2%  stated  that  ESF  support  does  not  make  a  difference  (to  national 
programmes) and the rest (11.3%) said they didn’t know. 

Similarly, most respondents believe that there is an ‘EU value added’ aspect to ESF TO8 support 
to  labour  mobility.  Just  above  40  per  cent  (42.9%  of  the  541  respondents)  believe  that  is 
because there would be not enough money to pay for such actions, and the rest (27.7%) said 
that without EU support there would be less or no attention paid to unemployed people. Finally 
11.6% stated that ESF support does not make a difference and 16.3% did not know. 

Unsurprisingly,  in  both  cases,  organisations  and  individuals  that  do  not  have  first-hand 
experience with the ESF were more likely to respond that they did not know or that ESF did not 
make a difference.  

Participants in the EU-level focus groups agreed that TO8 support has brought about  clear EU 
added value in terms of: 

• 
• 
• 

• 

• 
• 

• 

providing additional resources; 
covering new target groups; 
creating  opportunities  to  test  innovative  projects  and  upscale  them  (provided 
there’s sufficient innovation readiness); 
spreading good ‘policies’ across countries and steering attention to key challenges 
also through country specific recommendations;  
strengthening partnerships, including private public partnerships; 
increased  attention  to  social  innovation  and  social  entrepreneurship:  ESF  helps  to 
tailor the approaches, and test mechanisms and roll out successful methods;  
increased inter-institutional cooperation. 

4. 

THEMATIC OBJECTIVE TO9 – SOCIAL INCLUSION 

4.1. 

The consultation strategy 

4.1.1.  Roadmap 

The evaluation roadmap  was published on the Commission’s ‘Have your Say’ portal and made 
available  for  public  feedback  between  19   December 2018  and  16 January 2019124.    Six 
contributions were  received. Two were  anonymous and four came  from organisations (ENSIE, 
FEANTSA, EuroHealthNet, and the European Microfinance Network).They expressed support for 
the evaluation but did not suggest changes to the roadmap.  

4.1.2.  Consultation stakeholders  

The following individuals and groups were consulted: 

(1) 

Participants:  individuals  supported  by  the  European  Social  Fund  operations  were 
consulted to obtain a direct insight into: (i) the extent to which their own objectives 

124 

https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/2030-Evaluation-of-the-

support-to-social-inclusion-by-the-European-Social-Fund  

85 

 
 
 
                                                           
 
(2) 

(3) 

(4) 

(5) 

(6) 

(7) 
(8) 

(9) 

for  participation  have  been  achieved;  and  (ii)  success  factors  that  could  not  be 
directly identified through the monitoring mechanisms.  
Potential  participants:  individuals  that  did  not  received  ESF  support  but  who 
according  to  the  ESF  regulation  might  have  qualified  were  consulted  to  obtain  a 
direct insight into the reasons for their non-participation. 
Beneficiaries are the ultimate recipients of funds. They cover organisations actually 
delivering  ESF  support  to  participants  and  organisations  using  ESF  support  to 
enhance  access  to  affordable,  sustainable  and  high  quality  services  of  general 
interest, in particular in the fields of health care, employment and training services, 
services for the homeless, out of school care, child care and long-term care services. 
They were consulted to obtain a direct insight into: (i) the extent to which their own 
objectives for participation have been achieved; and (ii) success factors and barriers 
that could not be directly identified through the monitoring mechanisms.  
Potential  beneficiaries  are  organisations  that  could  have  benefited  from  ESF 
support but did not. They were consulted to obtain a direct insight into the reasons 
for their non-participation. 
Organisations  or  institutions  involved  in  the  management  of  the  operational 
programmes  such  as:  Member  States,  managing  authorities,  intermediate  bodies, 
organisations or institutions sitting on the monitoring committees. 
Organisations not directly involved but that represent the people targeted by the 
ESF - more specifically those at risk of social exclusion, poverty and discrimination - 
who  can  therefore  provide  a  broader  view  of  the  needs  and  issues  at  stake  with 
recipients and potential recipients. 
Social partners who could give feedback on important success factors;. 
Academic and research bodies involved in the area of social exclusion, poverty and 
discrimination  -  including  the  access  to  affordable,  sustainable  and  high  quality 
services  of  general  interest  –  that  could  provide  insights  into  the  relevance, 
effectiveness, efficiency, coherence or sustainability of measures. 
General  public:  any  individual  or  organisation  that  could  provide  useful  views  on 
how  the  ESF  can  social  inclusion,  and  combat  poverty  and  any  discrimination 
(thematic objective TO9).  

4.1.3. 

Consultation methods and tools 

To  reach  adequately  these  stakeholders,  a  range  of  consultation  activities  and  methods  were 
used, namely: a public consultation, targeted consultations (including interviews with EU and 
national level stakeholders, case studies of 19 Operational Programmes in 11 countries125 which 
also  included  in-depth  interviews  with  key  stakeholders,  focus  groups  in  each  case  study 
Member  State,  and  a  Delphi  survey  to  validate  the  evaluation’s  draft  findings  with  selected 
experts.  

125  Austria,  Bulgaria,  Cyprus,  Spain,  Hungary,  Italy,  Latvia,  the  Netherlands,  Poland,  Portugal  and 

Sweden 

86 

 
                                                           
Table 2: Overview of the stakeholders reached through each consultation tool/method 

Type of stakeholder 

Participants 
Potential participants 
Beneficiaries 
Potential beneficiaries 
Managing authorities 
Organisations or institutions 
involved in the management of 
the operational programmes 
Organisations not directly 
involved in the implementation 
of the operations 
Social partners 
Academic and research bodies 
General public 

Targeted 
interviews 
(EU and 
national 
level) 

X 

X 

X 

X 

X 
X 

Targeted 
interviews 
(case studies) 

Public 
consultation 

Focus 
groups 

Delphi 
Survey 

X 

X 

X 

X 

X 
X 
X 
X 
X 

X 

X 

X 
X 
X 

X 

X 

X 

X 
X 
X 

X 

The consultation methods were implemented, in line with the agreed consultation strategy. The 
only  major  deviation  was  that  the  focus  group  planned  in  Italy  could  not  be  held  because  it 
coincided  with  the  start  of  the  COVID-19  pandemic.  Instead  an  EU-level  Delphi  survey  with 
selected experts was organised to validate the draft findings and to gather their views on two 
issues that came out from the field work, (i) the documentation of “soft-outcomes” and (ii) the 
access to EU-funds for small, local organisations.  

4.2. 

Specific consultations/activities 

4.2.1.  The public consultation 

4.2.1.1. Description of the activity 

The  online  public  consultation  ran  from  26 September 2019  to  19 December 2019  and  took 
form of a survey. The questionnaire comprised six strands of questions (excluding profiling and 
closing  sections),  structured  around  the  main  evaluation  questions  and  including  both  open-
ended and closed questions. Each strand was customised to one specific group of respondents. 

4.2.1.2. Stakeholders  

A  total  of  574  individuals  and  groups  responded.  Bulgaria  was  the  Member  State  with  the 
highest  number  of  responses  (123  representing  21.4%  of  the  total),  followed  Hungary  (82 
responses-  14.3%  of  the  total),  and  Croatia  (79  responses=  13.8%  of  the  total)  from. 
Importantly,  responses  were  received  from  all  28  EU  Member  States.    Five  of  the  574 
respondents came from outside the EU.  

87 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 2: Number of respondents to the public consultation for ESF support to social inclusion 
by Member State. 

Source: TO9 supporting study report Annex 7 

The  public  consultation  had  a  separate  section  for  each  of  the  following  six  groups  of 
respondents: 

• 
• 

• 

• 
• 

• 

Strand I: organisations with no knowledge of or involvement in ESF TO9; 
Strand II: organisations with knowledge of ESF TO9 but not involved in its delivery: 
e.g. NGOs and think-tanks; 
Strand  III:  organisations  involved  in  the  delivery  of  ESF  TO9:  e.g.  Managing 
Authorities, Intermediate Bodies, beneficiary organisations or project promoters; 
Strand IV: individuals who have received ESF TO9 support; 
Strand  V:  individuals  who  have  not  received  ESF  TO9  support  but  who  have 
knowledge of it: e.g. academic experts; 
Strand VI: individuals with no knowledge of or involvement in ESF TO9.  

Both individual respondents and those responding on behalf of organisations were directed to 
specific strands of questions corresponding to their level of knowledge or involvement with 
ESF.    The  number  of  questions  varied  by  strand.    See  Table  3  below  for  the  number  and 
proportion oof respondents by group. 

Table 3: Number of respondents by group 

Strand 

Strand I: Organisations not aware of ESF 
Strand II: Organisations aware of ESF but not playing a direct role in the 

No. of 
responses 
10 
59 

% 

1.7% 
10.3% 

88 

 
 
Strand 

delivery 
Strand III: Organisations directly involved in the delivery of ESF 
Strand IV: Individuals having received ESF support 
Strand V: Individuals aware of the ESF but not having received support 
Strand VI: Individuals not aware of the ESF and not having received support 
Total 

No. of 
responses 

% 

295 
38 
132 
40 
574 

51.4% 
6.6% 
23.0% 
7.0% 
100% 

Looking  at  the  type  of  respondent,  the  574  responses  were  split  unevenly  between 
organisations (364 responses or 63% of the total) and citizens (210  responses or 37%). Of the 
organisations, the most responses came from public authorities (168), followed by NGOs (70). 
All respondents, except two, identified as EU citizens. More than 60% identified as female and 
more than 80% were aged between 25 and 54 years old.  

Of the 574 respondents, more than half responded on behalf of organisations that are directly 
involved  in  ESF  (295  or  51.4%).  Of  the  individual  respondents,  the  biggest  group  was  those 
aware of ESF but not having received ESF support (132 or 23%).  

More than a third (136 out of 364) of those responding on behalf of organisations work in large 
organisations (with 250 employees or more) that predominantly specialise in the labour market 
and social inclusion area, of which just over 40% were involved in the management of EU funds.  
On their involvement in ESF delivery, 41.9% (152 out of 364) identified as beneficiaries, having 
received ESF funding for projects. A total of 110 responses (30.3%) were submitted on behalf of 
Managing Authorities or Intermediate Bodies. Only a few of the 364 respondents (30 or 8.3%) in 
the ‘organisations’ strand indicated that they did not play a no role in ESF delivery. 

Turning  to  individual  respondents’  experience  or  involvement  with  the  ESF,  nearly  three 
quarters of 210 respondents said that they had never received ESF support; i.e. had never taken 
part in an ESF activity. 33 said they had recently received ESF support and only 5 said that they 
are currently receiving ESF support. These results appear to coincide with the answers on socio-
economic status in that  nearly three-quarters of individuals (155 out of 210 or 73.8%) say they 
are employed full-time, and ESF social inclusion activities tend to target the unemployed or the 
irregularly in employed.  

A total of 25 respondents also attached a position paper.  

4.2.2.  Targeted interviews with stakeholders 

A total of 141 interviews were carried out with EU and national level stakeholders, covering all 
included  managing  authorities, 
Member  States.  The  types  of  stakeholders  consulted 
beneficiaries,  socioeconomic  partners,  evaluators/researchers  and 
representatives  of 
participants.  

4.2.3.  Case studies 

Twenty (20) operational programmes (OPs) in 11 Member States were selected for case study. 
Within  these  20  OPs,  country  experts  -  in  consultation  with  the  Managing  Authorities  - 

89 

 
 
identified a total of 62 projects that could be appropriate for an in-depth analysis. However, one 
of the OPs was dropped from the list as an interview could not be organised with the Managing 
Authority. Focus groups were held in each of these Member States except Italy due to the onset 
of the Covid-19 pandemic. 

OPs  and  projects  were  selected  on  the  basis  of  criteria  agreed  with  Commission  during  the 
inception phase as well as consultations with the Commission and the Managing Authorities.  

The interviews and case studies were the main source of information to answer the evaluation 
questions. They also fed into case study reports. 

4.2.4.  Policy Delphi validation process 

Instead of an eleventh focus group, a Delphi survey was  conducted between 9 April 2020 and 
13 May 2020. It aimed to gather a range of perspectives from experts involved in EU-level policy 
debates on social inclusion, poverty and discrimination as well as from managing authorities not 
consulted  in  the  previous  stages,  one  auditor  and  social  partners.  Invitations  were  sent  to 
representatives  of  15  organisations  of  which  a  total  of  10  participated  in  at  least  one  round. 
These were: 

• 
• 
• 
• 
• 
• 
• 
• 
• 
• 

Solidar  
A Belgian ESF managing authority 
National Audit Union of Social Cooperatives – Poland 
European Roma Grassroots Organisations Network 
Inclusion Europe 
European Disability Forum (EDF) 
EuroHealthNet 
ENISE 
ESN 
European Network on Independent Living 

4.3. 

Overview of results of consultation activities 

4.3.1.  Effectiveness 

Overall, the vast majority of respondents from Strands II, III and V (80% of the 486) believe that 
all ESF actions are either very useful or mostly useful. The most useful actions (>80%) are:  

• 
• 
• 

• 

• 

basic skills training (e.g. social skills, IT, language) (90% or 438 respondents);  
training and education (including vocational training) (89% or 433);  
support to people with disabilities (e.g. promotion of community-based care) (86% 
or 419);  
support to overcome barriers to job search actions (e.g. transport or childcare) (85% 
or, 413);  
information, guidance, tutoring in the search for a job (85% or 412). 

The  organisations  involved  in  the  actual  delivery  (295  respondents)  deemed  that  the  most 
successful actions related to the following target groups:  

• 
• 

Persons having a disability (61% or 180);  
Unemployed for 12 months or more (60% or 178);  

90 

 
• 
• 

Unemployed for less than 12 months (59% or 174); and  
Low-skilled (57% or 168). 

Most  respondents  said  that  these  actions  brought  about  job-related  changes  for  participants, 
improved  their  soft  skills  and  self-confidence,  and  increased  awareness  of  social  inclusion 
poverty and discrimination. Partnerships between managing authorities and NGOs and targeted 
or individualised support are some of the main factors identified as contributing to the success 
of  ESF  actions.  Nearly  all  the  respondents  that  had  received  ESF  support  said  that  their 
expectations  from  participating  in  ESF  activities  had  been  partially  or  fully  met.  The  most 
frequently  expectations  were  obtaining  better  job  conditions,  finding  a  job  or  being  actively 
included in society.  

4.3.2.  Efficiency 

The efficiency question was only put to the Strand II and III respondents. Most (48-80% of the 
354 answers) said that the various ESF activities are cost-effective (the resources invested are 
proportionate to the results achieved), in particular: basic skills training; training and education; 
information, guidance and tutoring in the search for a job; and internships and traineeships to 
learn a trade. The only action deemed inefficient by a majority of respondents was ‘studies and 
evaluations of institutions’. 

involved 

Most  organisations 
indicate  direct,  practical  and 
in  the  delivery  of  the  ESF 
individualised  support  as  the  most  cost-effective,  followed  by  flexible  and  multidimensional 
support,  and  support  for  integration  to  employment.  The  factors  identified  as  contributing  to 
inefficiency,  included  the  so-called  ‘creaming’  effects  (i.e.  projects  focusing  on  achieving  set 
targets rather than on real social change), and the standardisation and rigidity of ESF delivery 
structures and timeframes.   

4.3.3.  Relevance 

Most respondents agreed that the EU should be involved in promoting social inclusion and in 
combating poverty and all kinds of discrimination. 

91 

 
Figure 3: What kind of support is provided with ESF funding to promote social inclusion, 
combat poverty and any discrimination? Answers by strand III respondents. 

N= 295 Source: TO9 supporting study report Annex 7 

ESF actions to get people into employment were deemed relevant by the highest proportion of 
respondents, regardless of their knowledge of or experience with ESF. Two-thirds of those that 
had received ESF support mentioned having participated in ESF training and education actions 
or had received information and guidance when searching for a job. Of those who answered on 
behalf of organisations directly involved in ESF delivery, most (51% of the 295) said that their 
support is directed at people who have been unemployed for 12 months or more. Many, (42% 
of the 295) said it was also directed at people with a disability.  

The  review  of  position  papers  suggests  that  the  ESF’s  Thematic  Objective  9  is  still  generally 
considered  to  be  appropriate,  as  it  supports  actions  related  to  social  inclusion  and  anti-
discrimination, which are as relevant today as they were in 2014.  

4.3.4.  Coherence 

Overall,  the  ESF  actions that  promote  social  inclusion,  combat  poverty or  discrimination  were 
considered  to  be  coherent  with  other  schemes,  in  particular  with  national,  regional  or  local 
programmes. 

92 

 
 
Figure 4: In your opinion to what extent are ESF actions promoting social inclusion, combating 
poverty and any discrimination coherent with other schemes? Respondent strands II, III and V 

National, regional or local programmes

10% 

4% 

Erasmus+

3% 

1% 

ERDF/CF

5% 

2% 

FEAD

6% 

1% 

European Solidarity Corps

4% 

58% 

34% 

46% 

43% 

52% 

52% 

34% 

27% 

60% 

68% 

They complement or reinforce each other

They do the same

They are contradictory / They hinder each other

I do not know / I do not wish to answer

N=486 Source: TO9 supporting study report Annex 7 

Regarding  EU  schemes,  Erasmus+  was  most  frequently  identified  as  complementing  and 
reinforcing  ESF  actions  promoting  social  inclusion  or  combating  poverty  and  discrimination.  A 
trend  noticeable  in  the  review  of  position  papers  is  that  there  will  be  scope  under  ESF+  to 
further strengthen coherence with the European Pillar of Social Rights (on access to education 
and sustainable employment) and with Interreg (on social innovation). 

4.3.5.  EU added value 

An  overwhelming  majority  of  respondents  indicated  that  one  of  the  advantages  of  EU 
interventions  to  promote social  inclusion,  combat  poverty  and  discrimination is  that  they  can 
achieve  more  could  be  done  than  national  or  local  resources  only.  More  than  half  of  the 
respondents  also  thought  that  ESF  measures  allow  new  social  issues  to  be  covered  and  new 
social  services  to  be  tested.  Very  few  respondents  thought  that  ESF  or  even  EU-supported 
interventions  make  no  difference  in  promoting  social  inclusion  or  combating  poverty  and 
discrimination. 

93 

 
 
Table 4: In your view, what would be the advantage of having European Union interventions? 
(broken down by Strand III respondents’ role in ESF delivery) 

Beneficiary org. 
receiving ESF 
funding 
(N=152) 

MA / IB 
(N=109) 

Member of an 
ESF Monitoring 
Committee 
(N=18) 

EU funds 
Coordinating 
body 
(N=12) 

Certifying or 
Audit 
Authority 
(N=4) 

88% 

84% 

94% 

50% 

100% 

55% 

62% 

78% 

57% 

58% 

61% 

2% 

3% 

2% 

1% 

2% 

6% 

0% 

6% 

0% 

58% 

33% 

0% 

0% 

17% 

75% 

50% 

0% 

0% 

0% 

More could be done 
than with national 
or local resources 
only 
New issues could 
be covered 
New ways of 
delivering services 
could be tested 
None. I do not think 
it would really 
make a difference 
Other 
I do not wish to 
answer / I do not 
know 

N=295. Source: PC results; multiple answers possible.  

4.3.6.  Overall conclusion 

The  public  consultation  suggests  that  ESF  support  for  TO9  is  rated  positively  value  by  a  wide 
range  of stakeholders –  both individuals and organisations that are not fully aware of the ESF 
and  those  who  actively  involved  in  its  delivery.  ESF  actions  are  generally  considered  to  be 
effective  and  to  offer  value  for  money  that  could  not  be  achieved  with  national  or  local 
resources  only.  However,  some  replies  highlight  areas  of  concern  and  of  potential 
improvements, including 'creaming' effects that hinder the efficiency of actions and the scope 
for greater coherence with the European Pillar of Social Rights. 

5. 

THEMATIC OBJECTIVE TO10 – EDUCATION AND TRAINING 

5.1. 

The consultation strategy 

5.1.1.  Roadmap 

The evaluation roadmap was published on the Commission’s ‘Have your Say’ portal and made 
available  for  public  feedback  between  21 December 2018  and  18 January 2019.      Six  replies 
were received of which 3 were anonymous. The other three were from the organisations: ENSIE 
(European  Network  of  Social  Integration  Enterprises),  Masaryk  University  (Czechia)  and  EAPN 
Nederland (Non-governmental organisation). They expressed support for the evaluation but did 
not suggest changes to the roadmap. 

5.1.2.  Consultation stakeholders 

The following groups were consulted: 

94 

 
 
(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 
(8) 

(9) 

Participants:  individuals  supported  by  the  European  Social  Fund  operations  were 
consulted  to    obtain  a  direct  insight  into:    (i)  the  extent  to  which  their  own 
objectives for participation have been achieved; and (ii) success factors that could 
not be directly identified through the monitoring mechanisms. 
Potential  participants:  individuals  who  did  not  received  ESF  support  but  who 
according  to  the  ESF  regulation  might  have  qualified  were  consulted  to  obtain  a 
direct insight into the reasons for their non-participation; 
Beneficiaries  are  the  ultimate;  and  (ii)  the  success  factors  and  barriers  that  could 
not be directly identified through the monitoring mechanisms.; 
Potential  beneficiaries  are  organisations  that  could  have  benefited  from  ESF 
support but did not. They were consulted to obtain a direct insight into the reasons 
for their non-participation; 
Organisations  or  institutions  involved  in  the  management  of  the  operational 
programmes  such  as:  Member  States,  managing  authorities,  intermediate  bodies, 
organisations  or  institutions  sitting  on  the  monitoring  committees  (see  Table  1, 
rows 5 to 9)  
Organisations  not  directly  involved  in  the  implementation  of  the  operations  but 
that represent the people targeted by the ESF - more specifically those in education 
and training - and could therefore provide a broader view of the needs and issues 
faced by the recipients and potential recipients. 
Social partners who could give feedback regarding important success factors. 
Academic and research bodies involved in the area of education and training that 
could  provide  insights  as  to  the  relevance,  effectiveness,  efficiency,  coherence  or 
sustainability of measures. 
General  public:  any  individual  or  organisation  that  could  provide  useful  views  on 
how the ESF can support education and training. 

5.1.3. 

 Consultation methods and tools 

To adequately reach these stakeholders, a range of consultation methods were used, namely: a 
public  consultation,  targeted  consultations  (including  interviews  with  EU  and  national  level 
stakeholders, case studies of 20 Operational Programmes in 12 countries126  which also included 
in-depth  interviews  with  key  stakeholders  ),  and  a  policy  Delphi  validation  process  on  the 
evaluation’s draft findings. The supporting study made use of a meta-analysis of six pre-existing 
national ESF participant surveys127.  

126  Poland,  Portugal,  Germany,  Spain,  Italy,  United  Kingdom,  France,  Czechia,  Romania,  Lithuania, 

Ireland and Slovenia 

127  Six specific ESF participant and beneficiary surveys on TO10 from six Member States (Spain, France, 

Italy, Croatia, Lithuania and the UK). 

95 

 
                                                           
Table 5: Overview of the stakeholders reached through each consultation tool/method 

Type of stakeholder 

Participants 
Potential participants 
Beneficiaries 
Potential beneficiaries 
Managing authorities 
National and regional education 
ministries and VET institutions 
National and regional labour and 
employment ministries and institutions 
European institutions involved in ESF or 
related education and training policy 
development 
Other relevant national, regional and 
European bodies responsible for 
management / implementation or match 
funding of education, training or 
employment funds 
Organisations not directly involved in 
implementation/ associations representing 
those in education or training  
Social and economic partners at national 
and EU level 
Research  and academic organisations  
General public 

Targeted 
interviews 
(EU and 
national 
level) 

Targeted 
interviews 
(case studies) 

Public 
consultation 

Policy Delphi 
validation 
process 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 
X 
X 
X 
X 

X 

X 

X 

X 

X 

X 
X 

X 
X 
X 

X 

X 

X 

X 

X 

X 

The  consultation  methods  were  implemented  in  line  with  the  agreed  consultation  strategy. 
There were some challenges in securing all the interviews for the targeted consultation as some 
EU-level  stakeholders  declined  to  participate  due  to  limited  or  no  knowledge  on  ESF  or  were 
non-responsive despite reminders. The impact of this on the consultation strategy was minimal, 
however,  as  the  targeted  consultations  allowed  the  in-depth  views  of  key  stakeholders  to  be 
gathered.  

96 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5.2. 

Specific consultations/activities 

5.2.1.  Public Consultation 

5.2.1.1. Description of the activity 

The  public  consultation  ran  for  14  weeks  and  took  the  form  of  an  online  survey.  The 
questionnaire included both closed and open questions, addressing the key evaluation criteria, 
and was tailored to the different stakeholder groups using a routing system.  

5.2.1.2. Stakeholders 

A  total  of  817  individuals  and  groups  responded  to  the  public  consultation.  The  geographical 
distribution  was  somewhat  unbalanced  with  most  responses  (153)  coming  from  Portugal, 
followed by Spain (105), Poland (102), Italy (86) and Croatia (68). All other countries had fewer 
than 60 respondents as shown in Figure 5. 

Figure 5: Number of respondents by country of origin – TO10 

The public consultation had a separate section for each of the following groups of respondents:  

• 

• 

• 

Group A: ESF participants: individual EU and non-EU citizens receiving ESF support 
now or since 2014; 
Group B: Individual EU and non-EU citizens who did not  take  part  in the ESF: ‘the 
general  public’,  reporting  either  having  an  awareness  of  the  ESF  or  no  direct 
knowledge of the ESF measures; 
Group C.1: Managing Authorities and Intermediate Bodies, organisations involved in 
the management of the ESF that supposedly have a direct and detailed knowledge 
of the ESF support; 

97 

 
 
• 

• 

• 

Group  C.2:  Beneficiaries:  organisations  or  entities  receiving  ESF  funding  for  the 
implementation of a project; 
Group C.3: EU Funds Coordinating bodies, Certifying/Audit authorities, Members of 
a Monitoring Committee directly involved in the delivery of ESF. 
Group D: Other organisations: organisations with no direct role in the management, 
monitoring and implementation of the ESF including civil society. 

Of  the  817  responses,  the  highest  number  (370  or  45.3%  of  the  total)  was  from  the  group 
representing  organisations  managing/delivering the  ESF,  followed  by the  general  public  (244 
respondents or 29.9% of the total), ESF participants (141 - 17.3%) and other organisations (62 - 
7.6%).  Of  the  organisations  managing/delivering  the  ESF,  ESF  beneficiaries  provided  the  most 
responses (227 or 61.4% of the total) followed by managing authorities and intermediate bodies 
(104 responses or 28.1%), and EU funds coordinating bodies, certifying or audit authorities and 
members of monitoring committees (39 responses or 10.5%). Figure 6 presents the breakdown 
of respondents by stakeholder/profile group128. 

Figure 6: Breakdown of respondents by country of origin and group – TO10 

5.2.2.  Targeted interviews with EU and national level stakeholders 

Targeted  interviews  were  conducted  with:  (i)  EU=level  stakeholders  directly  and  indirectly 
linked  to  the  themes  of  education  and  training  and  (ii)  national  stakeholders  involved  in  ESF 
operations, primarily managing authorities and representatives of Education/Labour Ministries; 
and (iii) other stakeholders where  relevant. A total of 25 interviews were conducted between 
October 2019 to March 2020 (See Table 6 below). 

Table 6: Targeted interviews completed 

Level 
EU 

Organisation 
DG EMPL Unit F1 ESF Coordination 
DG EAC group interview with: 

  Unit A2 Policy Strategy and Evaluation - Country analysis  
  Unit C2 Marie Skłodowska Curie actions  

128  The  figure  only  includes  countries  with  at  least  20  responses.  The  countries  with  fewer  than  20 

respondents were grouped together under ‘Other’. 

98 

 
 
 
                                                           
Level 

Organisation 

  Unit B4 Erasmus+ Coordination 

DG EMPL Unit E3 Vocational education, apprenticeships and adult learning 
EU Lifelong Learning Platform 
European University Association 
Marie Curie Alumni Association (MCAA) 
European Federation of Education Employers 
European Parliament: EMPL Committee 
European Student's Union 
European Association of Institutions in Higher Education 
European Trade Union Committee for Education 
Business Europe 
SME United 
European Forum of Technical and Vocational Education and Training 
European Vocational Training Association 
Eurochild 
Ministry of Education Science and Research 

Austria 
Luxembourg  ESF Managing Authority; Ministry of Education Childhood and Youth 
Denmark 
Malta 

Danish Business Authority (ESF MA) 
ESF Managing Authority 
Ministry for European Affairs and Equality 
ESF Managing Authority of OP Human Resource Management, Education and 
Lifelong Learning 
ESF Managing Authority 
Ministry of Education and Science 
ESF Managing Authority 

Greece 

Hungary 
Latvia 
Sweden 

5.2.3.  Case studies 

Case studies of 20 Operational Programmes in 12 Member States were conducted as part of the 
evaluation research.  

An  average  of  seven  interviews  were  conducted  per  case  study  with  stakeholders  from  the 
following categories:  

• 
• 
• 
• 

officials in charge of the specific OP;  
authority responsible for TO10 operations;  
major beneficiaries of ESF TO10 operations; and 
other  stakeholders,  including  evaluators  of  ESF  T10  measures,  vocational  training 
organisations, agencies dealing with illiteracy and early leavers from education and 
training, universities, etc.  

The case study interviews took place between October 2019 and April 2020. 

5.2.4.  Policy Delphi validation process 

A  validation  process  involving  workshops was  planned  in  order to validate  the findings  of  the 
evaluation. Due to the COVID-19 restrictions, these workshops were  replaced by online policy 
Delphi consultations, one for each investment priority, which took place between 27 April and 
27  May  2020.  The  consultees  included  thematic  experts,  policy  experts  from  relevant 

99 

 
 
Commission  departments,  ESF  managing  authorities,  ESF  TO10  beneficiaries,  social  partners, 
and EU-level NGOs.  

During this process, 22 responses were received from a range of stakeholder groups as detailed 
in Table 7. 

Table 7: Respondents to the policy Delphi validation 

Type of stakeholder 
NGOs 
Research and academic experts 
National and regional education ministries and VET institutions 
European institutions involved in education and training policy development 
Social and economic partners at national and EU level 
Beneficiaries 
Managing authorities 

Number of respondents 
6 
4 
4 
3 
3 
1 
1 

5.3. 

Overview of results of consultation activities 

5.3.1. 

 Effectiveness 

Overall,  there  was  consensus  among  stakeholders  that  progress  has  been  made  in  ESF  TO10 
operations,  but  differences  in  performance  were  noted  between  countries,  OPs  and  IPs.  ESF 
participants’  views  were  positive:  of  the  114  who  responded  to  the  public  consultation 
effectiveness  question  79  (or  56%)  stated  that  ESF  activities  had  fully  met  their  initial 
expectations and 25 (or 17.7%) said that they had surpassed them. Of the 141 respondents, 74 
(52.5%) reported that without EU support there would be less or no opportunities for people in 
their situation and 62 (44%) said that there would not be enough money to pay for such actions 
without the ESF. Only 3.5% (5 of the 141 respondents) considered that the ESF does not make a 
real  difference.  However,  a  notable  minority  of  ESF  participants  felt  that  the  support  only 
partially  met  expectations  (26  of  the  141  respondents  or  18.4%),  while  2  respondents  (1.4%) 
stated that the support did not meet their expectations.   

Of the ESF participant surveys analysed, five out of six support this generally positive conclusion 
on  the  impact  of  the  ESF.  EU-level  and  national  stakeholders  consulted  in  the  policy  Delphi 
process  also  shared  the  view  that  the  ESF’s  TO10  has  contributed  to  structural  changes  and 
reforms  of  education  and  training  systems  at  all  levels.  However,  they  also  agreed  that 
significant improvements could be made upscaling ESF initiatives.  

There is some discrepancy in views on the extent to which the ESF was effective in reaching all 
target groups.  Respondents  replying  to the  public  consultation on  behalf  of organisations  felt 
that,  overall,  ESF  operations  were  ‘very’ or  ‘mostly’  successful  in  providing  support  to  a  large 
number  of  target  groups.  Of  the  104  managing  authorities/intermediate  bodies  that 
participated,  69  (or  66.3%)  considered  that  the  most  effectively  reached  groups  were: 
individuals with no or low qualifications and skills had been effectively reached;  , 62 (or 59.6%) 
said  the  same  about  people  with  disabilities  and  60 (or 57.7%  -  60 out  of  104)  said  the  same 
about older workers. Of the 227 replying on behalf of beneficiary organisations, 133 (or 58.6%) 
considered  teachers  and  trainers  at  all  levels  to  have  been  reached  effectively,  while  126 

100 

 
 
(55.5%)  said  the  same  about  students  (,  118  (52%)  about  individuals  with  no  or  low 
qualifications or skills (, and 115 (50.7%) individuals with disabilities.  

A small proportion of respondents who felt that ESF support was ‘mostly’ unsuccessful was low 
(10%or less) for all target groups listed. However, EU-level and national stakeholders consulted 
in  the  targeted  interviews  and  in  the  case  studies,  highlighted  several  challenges  in  reaching 
some  of  these  target  groups,  particularly  the  individuals  who  are  most  disadvantaged.  Older 
workers,  the  low-skilled,  Roma,  refugees  and  migrants,  the  long-term  unemployed  and  the 
disabled  were  identified  by  EU-level  and  national  stakeholders  as  not  having been  sufficiently 
reached, both in the interviews and the policy Delphi responses. In the latter, respondents said 
that measures risked focusing on ‘low hanging fruit’, supporting people who need only minimal 
support anyway and leaving aside the more problematic cases.  

By  contrast,  the  factors  increasing  the  effectiveness  of  ESF  TO10  operations  were  generally 
agreed  upon  by  all  stakeholders  responding  to  the  public  consultation’s  qualitative  open 
questions and being interviewed. They mentioned in particular: (i) operations that bring about 
systemic change and that are strategically planned; (ii) strong coordination and the involvement 
of  local  and  regional  stakeholders  and  non-public  partners;  and  (iii)  operations  that  focus  on 
individuals and their specific support needs. Reducing the administrative burden and simplifying 
the  governance  structure  of  the  fund  was  also  frequently  mentioned  by  stakeholders  in  the 
open questions and the targeted interviews. Respondents to the policy Delphi and the targeted 
interviews  also  strongly  agreed  that  vulnerable  groups  could  be  reached  more  effectively  by 
increasing the participation of civil society organisations, NGOs and local stakeholders.  

5.3.2. 

 Relevance  

There was strong consensus among stakeholders that TO10 investment priorities and  planned 
operations  have  been  and  remain  relevant  to  education  and  training  needs.  Of  the  71 
respondents  that  answered  the  relevant  question  58  (82%)  agreed  that  the  ESF  should  be 
involved in improving skills and supporting the development of education and training. Only 5 
(or 7%) disagreed and would prefer to support education and training on the national or local 
level  (the  remaining  11%  did  not  know  or  did  not  wish  to  answer).  Survey  respondents  also 
pointed to the relevance of ESF operations. This is supported by EU and national stakeholders 
who all considered that ESF TO10 operations were relevant to the education and training needs 
in Member States. EU-level stakeholders said that the breadth of the investment priorities (IP) 
contributed to this relevance, as it allowed a range of operations for different education levels 
and groups in need to be programmed. EU-level and national and stakeholders also cited this as 
a  reason  for  the  continued  relevance  of  ESF  TO10  operations  throughout  the  programming 
period.  

According  to  the  public  consultation,  the  most  relevant  actions  included:  (i)  support  for  the 
unemployed  to  learn  new  skills  to  help  find  work;  (ii),  support  to  link  tertiary  education  and 
training institutions with businesses and with local communities; and (iii) support for students 
with fewer opportunities to complete higher education scholarship studies. Those consultees in 
the  case  studies  largely  agreed,  highlighting  in  particular  the  relevance  of  upskilling  and 
reskilling  measures.  EU  stakeholders  stressed  that  all  investment  priorities,  but  in  particular 
IP10.iii (lifelong learning) and IP10.iv (labour market relevance) are increasingly relevant given 

101 

 
the need to ensure the whole population is equipped with the right skills for a changing world of 
work.  

However,  stakeholders  involved  in  IP10.iii  and  IP10.iv  operations  pointed  out  in  the  policy 
Delphi  a  decrease  in  partnership  involvement,  especially  in  VET,  during  the  programming 
period, citing this as an obstacle to the relevance of operations. The need for employers, trades 
unions and NGOs to be more involved was stressed, and also emerged strongly in the targeted 
interviews,  particularly  from  EU  level  stakeholders.  There  was  consensus  across  a  range  of 
stakeholders  that  this  was  necessary  to  increase  relevance  of  ESF  TO10  to  the  needs  of 
vulnerable groups, in particular. 

5.3.3. 

 Efficiency  

There  was  a strong overall consensus among stakeholders  that most of the  ESF activities are 
cost-effective.  Of 17 activities presented in the public consultation survey, 13 were considered 
cost-effective  (more  than  50%  of  respondents  agreed  or  strongly  agreed  was  the  case).  In  all 
subgroups  that  were  asked  about  the  administrative  burden,  most  respondents  said  the 
arrangements were appropriate: 62.3% of managing authority respondents (65 of 104), 59.5% 
of beneficiary organisation respondents (135 of 227) and 56.1% of certifying, coordination and 
monitoring body respondents (22 of 39).  

However, the management and control system and reporting and monitoring were judged as 
the most burdensome arrangements by more than a third of respondents to the question How 
would  you  qualify  the  following  administrative  arrangements  for  the  implementation  of 
operations?’  (370  total  respondents).  This  is  reflected  in  the  views  of  EU-level  and  national 
stakeholders  consulted.  They  all  mentioned  ESF  regulatory  requirements  –  with  most 
highlighting  the  heavy  reporting  requirements  –  as  posing  challenges  to  efficient 
implementation.  There  is  strong  consensus  that  these  heavy  requirements  are  particularly 
burdensome  for  smaller  beneficiaries.  Stakeholders  consulted  in  the  targeted  interviews,  and 
through  the  case  study  interviews  shared  the  view  that  voluntary  organisations  and  NGOs 
involved  in  social  inclusion  activities  find  it  more  difficult  to  deal  with  the  funding  and  data 
collection requirements. On the other end of the scale, communication arrangements were the 
most  commonly  deemed  appropriate  across  respondents  (by  264  of  370  or  71%  of 
respondents). 

The  vast  majority of  national  stakeholders  interviewed  for  the targeted  consultations  and  the 
case studies stated that simplified cost options (SCOs) help improve efficiency. Respondents to 
the  policy  Delphi  validation  agreed.  However,  these  stakeholders  also  identified  challenges  in 
implementing  them.  Several  mentioned  that  developing  SCOs  still 
involves  a  heavy 
administrative burden, and one that takes time to implement. Stakeholders spoke about delays 
from  the  European  Commission  in  approving  SCOs  proposed  by  Member  States.  Public 
consultation  respondents  considered  the  application  of  SCOs  to  be  the  most  inefficient 
administrative arrangement although several national respondents said that SCOs help simplify 
project preparation.  

Several  respondents  to  the  public  consultation  suggested  to  focus  more  on  qualitative 
indicators rather than quantitative measures to assess the results of the ESF measures. This was 

102 

 
 
also expressed by some national stakeholders, who suggested that results indicators should be 
more relevant to the IP goals. 

5.3.4.  Coherence 

Overall stakeholders expressed the view that the ESF’s TO10 operations are generally coherent 
with other EU funding programmes, though views vary significantly per fund. There is consensus 
that Erasmus+ and ERDF are the funds that are most coherent with ESF. Of all funds, Erasmus+ 
was rated as the most coherent with TO10 by the highest proportion of respondents, 62.3% of 
the general public (107 of 173); 63.8% of organisations managing ESF (236 of 370); and 74% of 
other  organisations  (34  of  46),    followed  by  the  ERDF.  These  two  funds  were  also  the  most 
frequently highlighted in the targeted interviews.   

However,  many  of  the  stakeholders  interviewed  felt  that  coherence  was  a  challenge  during 
implementation. This was confirmed by respondents to the policy Delphi validation. There was 
agreement on the complexity of having to apply under both funds for a joint project, as well as 
on the factors influencing coherence,  namely: (i)  the lack of cooperation between institutions 
responsible  for  each  fund  on  the  national  level;  (ii)  different  implementation  and  delivery 
modes (direct vs shared management); (iii) the difference in target groups of the funds; and (iv) 
insufficient communication to potential beneficiaries on the possibility of combining funds.  

Views on coherence with other EU funding instruments are more mixed. Of the members of 
the public that replied to the public consultation, many did not know or did not wish to provide 
an answer on whether ESF activities are coherent with a number of other schemes (i.e. COSME- 
112  of  173  (65%),  EURES-  110  of  173  (64%),  AMIF-  102  of  173  (59%),  other  ESI  Funds  and 
Horizon  2020/MSCA  100  of  173  (58%)).  The  EU-level  and  national  stakeholders  interviewed 
were also less able to comment on the degree of coherence of ESF with the Marie Skłodowska 
Curie Actions and Horizon 2020.  However, the views expressed though they appear to agree on 
the limited coherence with Horizon 2020 (MSCA). Stakeholder views from targeted interviews, 
case study interviews and the policy Delphi validation showed that coherence between ESF and 
MSCA  was  challenging  in  practice  and  only  29%  from  the  general  public  and  organisations 
managing/delivering ESF felt that there was coherence between the two funds.  

5.3.5.  EU added value 

The  perception  of  EU  added  value  resulting  from  ESF  TO10  operations  was  positive  overall 
among the range of stakeholders consulted. The vast majority of ESF participants responding 
to  the  public  consultation  agreed  that  having  ESF  support  made  a  difference,  though 
organisational  respondents  tended  to  be  more  positive.  323  out  of  the  370  organisation 
respondents (or 87.2%) indicated that more can be done than with national or local resources 
only. 73.9% of respondents from the general public (128 of 173) agreed. Other positive aspects 
highlighted were that new issues can be covered 41% of general public (71 of 173) and 61.8% of 
organisations (229 of 370) and that the ESF enables experimenting with new ways of delivering 
services (60 of 173 (39.3%) of general public and 212 of 370 (57.2%) of organisations) 

A majority of stakeholders interviewed shared this view, highlighting in particular that ESF TO10 
supported disadvantaged groups that  would have otherwise  not  been targeted. Both EU-level 
and  national  stakeholders  confirm  that  the  ESF  TO10  operations  produced  effects  at  national 

103 

 
and/or  regional  level  that  would  not  otherwise  have  taken  place.  However,  a  minority  of 
stakeholders  interviewed,  coming  particularly  EU-level  stakeholders,  expressed  the  view  that 
the  ESF  TO10  support  to  innovation  in  education  and  training  has  been  limited.  This  is  also 
reflected in the public consultation responses where ‘ESF enables experimenting with new ways 
of delivering services’ was the least selected advantage of ESF TO10 operations by the general 
public and individuals aware of the ESF but not receiving support (68 out of 173 respondents or 
39.3%). Respondents to the policy Delphi agreed, highlighting that where innovation is achieved 
through  the  ESF  it  needs  to  be  better  embedded  into  national  legislation  to  ensure  spill  over 
effects of the good practice. 

104 

 
ANNEX 3: METHODS AND ANALYTICAL MODELS 

1. 

OVERALL APPROACH TO THE EVALUATION WORK 

In line with the Better Regulation guidelines, DG EMPL decided to base this evaluation on the 
work carried out by external evaluators and took the following approach: 

• 
• 
• 

collect and analyse the relevant evidence; 
provide answers to all evaluation questions; 
present evidence-based conclusions. 

COVID-19 pandemic: It shall be noted that the fieldwork for these evaluations was carried out 
before  the  COVID-19  pandemic  outbreak  reached  Europe.  The  Corona  Response  Investment 
Initiative  (CRII)  will  affect  the  support  to  Employment  for  the  remainder  of  the  current 
implementation  period  and  the  proposals  for  the  next  programming  period  will  also  aim  at 
mitigating  the  consequences  of  this  pandemic.  This  COVID-19  pandemic  is  a  major  negative 
shock to the global and European economy. Already at the end of March of 2020, a substantial 
negative  economic  impact  on  Europe  has  materialised,  at  least  for  the  first  half  of  2020  and 
possibly  longer  if  the  pandemic  is  not  contained  rapidly.  For  the  future,  the  degree  of  the 
negative outlook will depend on a number of parameters such as the lack of supply of critical 
materials, the effectiveness of containment measures, the downtime in manufacturing in the EU 
work  days  lost  in  companies  and  public  administrations,  and  demand  effects  (e.g.  mobility 
restrictions, travel cancellations). 

2. 

RATIONALE OF THE EVALUATION 

The main objective of this evaluation is to assess the effectiveness, efficiency, coherence and EU 
value added of ESF support to the thematic objective 8, 9 and 10. The evaluation includes both 
an  individual  assessment  of  each  country  and  a  cross-cutting  and  comparative  assessment. 
External  contractors  collected  and  examined  data  covering  2014-2018  and  prepared  three 
studies129.    

3. 

EVALUATION QUESTIONS AND STRUCTURE OF THE REPORT 

The evaluations were based on the following overarching evaluation questions: 

• 

• 

• 

Effectiveness:  How  effective  has  the  ESF  been  in  achieving  the  objectives  of 
thematic objective 8/9/10? 
Efficiency: How efficient has the ESF been in the achievement of the objectives of 
thematic objective 8/9/10? 
Relevance:  How  relevant  have  the operations  funded  by  the  thematic objective  8 
been? How relevant are the ESF operations under thematic objective 9/10? 

129  Supporting studies:  

TO8:    https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8330&furtherPubs=yes;    TO9: 
TO10: 

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes; 
https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8346&furtherPubs=yes  

105 

 
 
 
                                                           
• 

• 

• 

Coherence: How coherent have the operations funded by thematic objective 8/9/10 
been among themselves and with other actions in the same field? 
EU Value added: What is the EU added value of the ESF-funded operations in the 
field of employment and mobility / of social inclusion, combating poverty and anti-
discrimination / of education? 
Sustainability: How sustainable the ESF-funded operations under TO8 have been?  

Each evaluation question was further detailed in sub-questions presented in Table 9 below. The 
evaluation work and the report were structured around the conclusions reached on each of the 
questions.  

4. 

METHODOLOGY AND DATA SOURCES 

The evaluation was based on a complex methodology aimed at collecting solid evidence and 
providing well-informed answers to the evaluation questions. This included:  

• 

• 

• 

• 

• 

• 

Desk research: 
o  analysis  of  the  monitoring  data  in  the  SFC2014  database  (System  for  Fund 

o 

Management in the European Union); 
in-depth  analysis  of 
(OPs)  and  Annual 
the  operational  programmes 
Implementation  Reports,  and  specifically  the  2018  Annual  Implementation 
Reports that include reported data as of 31st December 2018; 

o  The  mapping  of 

investment  priorities,  target  populations  and  types  of 
operations,  based on  the SFC2014  database,  operational  programmes,  Annual 
the 
Implementation  Reports  and 
Commission’s Country-specific Recommendations. 

related  documentation, 

including 

o  econometric  analysis  and  provision  of  data  to  the  Joint  Research  Centre  (JRC) 

o 

for the RHOMOLO simulations; 
literature  review  (policy  documents,  regulations,  national  and  EU-wide 
evaluations, ESF websites and publications);  

o  Synthesis  of  evaluation  reports  by  member  states  using  the  database  of 
evaluations  compiled  by  the  Evaluation  Helpdesk,  Annual  Implementation 
Reports and additional desk research. 

Public consultation: assisting the Commission in developing the questionnaire, and 
analysing the results of the public consultation.  
Cost-effectiveness  analysis  using  a  combination  of  SFC2014  programme  data  (for 
unit  costs),  relevant  counterfactual 
impact  evaluations,  feedback  from  the 
stakeholders both through the public consultation as well as EU level focus group, 
case  studies and a study of micro-data where available. In addition, first  attempts 
were  made  to  conduct  cost  benefit  analyses  for  TO9  (Social  inclusion)  and  TO10 
(education and training).  
Case  studies:  addressing  the  research  questions  at  national  and  regional  level, 
based on desk research, interviews and focus groups. 
Interviews  with  desk  officers,  Managing  Authorities  and  other  stakeholders,  in 
particular for the case studies. 
Holding focus groups in the Member States and at the EU level  or Delphi surveys 
when the EU level focus group has not been possible due to the Covid-19. 

5. 

OVERALL ASSESSMENT OF STRENGTHS AND WEAKNESSES OF THE METHODOLOGY AND DATA 

The various sources were triangulated, and where the evidence was insufficient or inconclusive, 
the  approaches  were  combined:  data-based,  documentary  and  perception-based  sources  as 

106 

 
well  as  quantitative  and  qualitative  techniques,  depending  on  the  nature  of  the  evaluation 
question and the strengths of the relevant data and approaches. 

The  evaluation  drew  on  a  number  of  sources  providing  opinions  and  perceptions,  including 
surveys, interviews and the public consultation. Data on opinions and perceptions are important 
evidence  where  evaluation  questions  elicit  the  views  of  stakeholders.  In  some  cases, 
respondents  or  interviewees  may  be  the  only  source  of  knowledge  or  witness  accounts  of 
events when no other sources are available. 

Under  the  Better  Regulation  guidelines,  the  public  consultation  is  an  important  tool  for 
collecting stakeholder input and views on EU policy initiatives. It cannot be expected to provide 
a fully representative view of EU public opinion, but it does offer a channel for the people who 
care  about  a  given  issue  to  voice  their  opinion.  As  expected,  respondents’  knowledge  and 
involvement  in  running  employment  operations  differed,  but  the  design  of  the  public 
consultation made it possible to distinguish between the respondents who were well-informed 
and others. This distinction fed into the analysis of the responses, and triangulated with other 
sources of evidence. 

In conclusion, while acknowledging that there are some data gaps and methodical limitations, 
the  evaluation  presents  well-informed, evidence-based  and  reliable  answers to  the  questions, 
to the extent possible.  

107 

 
  
Table 1: Evaluation questions and sub-questions for each specific supporting study 

TO8 – Employment and labour mobility 

TO9 – Social inclusion 

TO10 – Education and training 

How  effective  has  the  ESF  been  in  achieving  the 
objectives of thematic objective 8? 

the  achievement  of 

1.1. To what extent have the financial implementation 
the  expected  outputs 
and 
progressed  according 
the 
programmes?  What  are  the  main  factors  involved 
(delays in implementation, ESF absorption…? 

targets  set 

the 

to 

in 

How  effective  has  the  ESF  been  in 
achieving  the  objectives  of  thematic 
objective 9? 
1.1.  To  what  extent  have  the  financial 
implementation and the achievement of the 
expected  outputs  progressed  according  to 
the  targets  set  in  the  programmes?  What 
were  the  main  factors  involved  (delays  in 
implementation, ESF absorption…)? 

10  How  effective  has  the  ESF  been  in  achieving 
the objectives of thematic objective 10? 

the 

extent  have 

the  achievement  of 

financial 
1.1.  To  what 
implementation  and 
the 
expected outputs progressed according to the output 
targets  set  in  the  programmes?  What  are  the  main 
factors  involved  (delays  in  implementation,  ESF 
absorption…)?  In  particular,  what  is  the  progress 
made  by  ESF  on  target  indicators  in  higher 
education  and  secondary  education  among  all 
countries? 
1.2. How and to what extent does ESF contribute to 
the investment in education, training and vocational 
training  for  skills  and 
learning  and 
education targets of Europe 2020? 

lifelong 

to 

poverty 

combating 

the  general  objectives 

How and to what extent does ESF 
1.2. 
the  promotion  of  social 
contribute 
inclusion, 
and 
discrimination  and  the  social  inclusion 
target of Europe 2020? 

1.2. How and to what extent does ESF contribute to the 
achievement  of 
thematic 
objective 8? In particular, to what extent have the ESF 
operations, contributed to the positive evolution of the 
employment  situation,  the  labour  mobility,  the  self-
employment  and  the  other  objectives  of  thematic 
objective  8  up  to  now?  How  did  it  contribute  to 
addressing problems faced by target groups? 
1.7.  To  what  extent  has  the  ESF  contributed  to 
structural  changes  in  national  education  systems, 
vocational 
training  systems,  public  employment 
systems or employment policies? 
1.8.  What  was  the  concrete  contribution  of  the  TO8 
operations  to  the  promotion  of  gender  equality  and 
active ageing which are also objectives of TO8? 
1.3. To what extent were the target groups reached by  1.3.  How  were  relevant  national  strategies  1.3.  How  were  relevant  national  strategies  and 

108 

Section in 
the staff 
working 
document 
6.1 

6.1.1 

Data 
and info 
sources 

(1) (2) 
(3) (4) 
(5) 
(1) (2) 
(3) (5) 

6.1.2 

(1) (2) 
(3) (4) 
(5) 

6.3 

(1) (2) 

 
 
 
 
TO8 – Employment and labour mobility 

TO9 – Social inclusion 

TO10 – Education and training 

and  policy 
translated into operations? 

contexts 

and 

challenges 

policy  contexts  and  challenges 
operations? 

translated 

into 

including  disadvantaged  persons, 
the  operations, 
especially  those  from  marginalized  communities  and 
those  leaving  education  without  qualifications?  To 
what  extent  higher  education  institutions/universities 
were  beneficiaries  of  TO8  investments  (in  terms  of 
numbers  and  size  projects)?  Have  any  Member  States 
invested  in  developing  graduate  tracking  systems  or 
tracer  studies  or  similar  measures  for 
graduate 
measuring  outcomes  of  graduates  on 
labour 
market? 
1.4.  What  was 
employment obtained? 
1.5. Which types of operations were the most effective 
and  most  sustainable,  for  which  groups  and  in  which 
contexts  (e.g.  more  developed,  less  developed  and 
transition regions, urban and rural areas etc.)?  

the  quality  and 

timeliness  of 

the 

changes 

(intended 

1.4.  Which 
and 
unintended)  did  the  ESF  support  bring  to 
the 
these 
target  groups?  How  were 
changes,  notably  soft  outcomes,  assessed 
and 
of 
operations  are  or  were  the  most  effective 
and most sustainable, for which groups and 
in which contexts? 

documented?  Which 

types 

Section in 
the staff 
working 
document 

Data 
and info 
sources 

(3) (5) 

6.1.3 

(1) (3) 
(5) 

6.1.4 

(1) (3)  
(5) 

on 

low 

adults, 

people 

qualified 

advantaged 

groups  with 

1.4.  Which  changes  (intended  and  unintended)  did 
the ESF support bring to each specific target group, 
e.g.  young  people  in  initial  education  and  training, 
in 
NEETs, 
employment, unemployed, inactive? Which types of 
operations are or were the most effective and most 
sustainable,  for  which  groups  and 
in  which 
contexts? In particular, did the ESF support impact 
and 
equally 
disadvantaged  backgrounds?  To  what  extent  was 
access  to  the  different  sectors  of  education  (e.g. 
higher  education)  for  underrepresented  groups 
facilitated thanks to ESF? What kind of efficiency-
enhancing  measures  did  ESF  finance  in  higher 
education  as  part  of  investment  priority  TO10(ii)? 
To  what  extent  have  non-formal  pathways  been 
used  to  prevent  and  address  early  school  leaving 
and  support 
in  education  and 
training? 
1.5.  Which 
the 
effectiveness  of  ESF  operations  under  thematic 
objective  10,  by  type  of  operation  and  by  main 
target group? 

facilitate  or  hinder 

re-integration 

factors 

1.6.  What  main  factors  (geographical,  socioeconomic, 
organisational…)  had  a  bigger 
the 
thematic 
effectiveness  of  ESF  operations  under 
objective 8, by type of operation? 

impact  on 

1.5.  Which  factors  facilitate  or  hinder  the 
effectiveness  of  ESF  operations  under 
thematic objective 9, by type of operation? 

109 

 
 
TO8 – Employment and labour mobility 

TO9 – Social inclusion 

TO10 – Education and training 

How efficient has the ESF been in the achievement 
of the objectives of thematic objective 8? 

In  what  contexts?  What  were 

2.1.  To  what  extent  were  operations  cost-effective? 
What  types  of  operations  were  more  and  less  cost-
effective? 
the 
determining factors? 
2.2.  Are  there  significant  cost  differences  between 
Member  States/Regions  in  the  implementation  of  the 
operations? What are these differences related to? 

the  organisational 
2.3.  To  what  extent  were 
arrangements, 
including  management  and  control 
systems  at  all  levels  conducive  to  the  effectiveness  of 
operations?    Was  there  administrative  burden,  in 
particular gold plating involved?  
2.4.  In  particular,  how  timely  and  cost-efficient  were 
the procedures for reporting and monitoring? 

2.5. How visible were TO8 funded operations? 

How  relevant  have  the  operations  funded  by  the 
thematic objective 8 been? 
3.1.  To  what  extent  were  the  objectives  and  the 
operations funded by the ESF relevant to the needs of 

 How  efficient  has  the  ESF  been  in  the 
achievement  of 
the  objectives  of 
thematic objective 9? 
2.1.  To  what  extent  were  operations  cost-
effective?  What  types  of  operations  were 
more  and  less  cost-effective?  In  what 
contexts?  What  were 
the  determining 
factors? 
2.2.  Are  there  significant  cost  differences 
between  Member  States/Regions  in  the 
implementation  of  the  operations?  What 
are these differences related to? 

2.3.  How  do  organisational  arrangements 
influence  service  delivery  by  beneficiaries 
or,  eventually,  lead  to  non-take  up  by 
potential  beneficiaries?  To  what  extent  is 
non-take-up  a  choice  or  due  to  non-
awareness of the instrument? 
2.4. To what extent were the organisational 
arrangements,  including  management  and 
control  systems  at  all  levels,  conducive  to 
the  effectiveness  of  operations?  Is  there 
gold  plating?  Were  the  procedures  for 
reporting  and  monitoring 
timely  and 
efficient?  

2.5. How visible are ESF funded operations 
under TO9? 
How  relevant  are  the  ESF  operations 
under thematic objective 9? 
3.1. Were the objectives and the operations 
funded by the ESF relevant to the needs of 

110 

in 

the  ESF  been 

the  objectives  of 

the 
thematic 

How  efficient  has 
achievement  of 
objective 10? 
2.1. To what extent were operations cost-effective? 
To  what  extent  are  the  costs  of  the  intervention 
proportionate,  given  the  changes/effects  it  has 
achieved? What types of operations were more and 
less  cost-effective  for  which  target  group?  In  what 
contexts? What were the determining factors?  
2.2.  Are  there  significant  cost  differences  between 
Member  States/regions  in  the  implementation  of 
similar  operations?  What  are  these  differences 
related to? 
2.3. How do organisational arrangements influence 
service delivery by beneficiaries or, eventually, lead 
to  non-take  up  by  potential  beneficiaries?  To  what 
extent  is  non-take-up  a  choice  or  due  to  non-
awareness of the instrument?  
2.4.  To  what  extent  were 
the  organisational 
arrangements,  including  management  and  control 
systems at all levels, conducive to the effectiveness 
of  operations?  Is  there  gold  plating16?  Are  there 
opportunities to  simplify  the  national legislation or 
regulatory  costs  without 
reduce  unnecessary 
undermining 
the 
intervention? Were the procedures for reporting and 
monitoring timely and efficient? 
2.5.  How  visible  are  ESF  funded  operations  under 
TO10 
How  relevant  are  the  ESF  operations  under 
thematic objective 10? 
3.1. Were the objectives and the operations funded 
by  the  ESF  relevant  to  the  needs  of  target  groups? 

intended  objectives  of 

the 

Section in 
the staff 
working 
document 
6.2 

6.2.1 

Data 
and info 
sources 

(1) (2) 
(3) (5) 

(1) (2) 
(3) (5) 

6.2.2 

(1) (3) 
(5) 

6.2.3 

6.3 

6.3.1 

(1) (3)  
(5) 
(1) (3) 
(4) (5) 
(1) (3) 
(4) (5) 

 
 
TO8 – Employment and labour mobility 

TO9 – Social inclusion 

TO10 – Education and training 

Section in 
the staff 
working 
document 

Data 
and info 
sources 

target groups?  
3.3.  Were  the  most  relevant  groups,  in  the  different 
socio-economic  contexts  (e.g.  more  developed,  less 
developed and transition groups; urban and rural areas 
etc.), targeted starting from the design stage? Were the 
most important needs of these groups addressed? 
3.4.  Were  the  TO8  operations  undertaken  suitable  to 
support  active  ageing  and  gender  equality?  How  the 
TO8  operations  implemented  fall  within  the  broader 
context of TO8? 

target  groups?  How  were  the  different 
target  groups  prioritised  and  the  actions 
tailored  to  their  specific  needs?  Were  the 
most  important  needs  of  these  groups 
addressed? 
3.2. Were  the  most relevant  groups, in the 
different  socio-economic  contexts  targeted 
starting  from  the  design  stage?  How  was 
the partnership and multi-level governance 
implemented? 

To  what  extent  are  they  still  relevant?  How  were 
the  different  target  groups  prioritised  and  the 
actions  tailored  to  their  specific  needs?  Were  the 
most important needs of these groups addressed? 
3.2. Were the most relevant groups, in the different 
education  and  socio-economic  contexts  targeted 
starting  from  the  design  stage?  How  was  the 
partnership 
governance 
implemented? 

multi-level 

and 

3.2. To what extent were OPs flexible and able to adapt 
to  changes  in  the  implementation  context,  notably  the 
evolution in the situation of employment and mobility? 

How  coherent  have  the  operations  funded  by 
thematic  objective  8  been  among  themselves  and 
with other (actions in the same field? 

4.1. In which manner were the ESF operations of TO8 
complementary  with  each  other?  What  were  the  main 
factors in this regard? 

4.2.  To  what  extent  were  they  complementary  and 
coherent with other thematic objectives and with other 
programmes  (e.g.  EURES,  EaSI,  Erasmus+,  ERDF, 
EAFRD,  EMFF,  EGF…)  oriented  to  employment  and 
mobility at the EU level? 
4.3.  To  which  extent  are  the  investments  under  TO8 
contributing  and  how  consistent  these  are  with  the 
National  Reform  Programmes  and, 
the  Country 
Specific  Recommendations  in  the  framework  of  the 

9 

adapt 

thematic 

objective 

changes 
or 

3.3. To what extent were OPs flexible and 
in 
to 
able 
the 
to 
political 
context 
implementation 
priorities? 
How coherent are the operations funded 
by 
among 
themselves and with other actions in the 
same field? 
4.2.  Were  the  ESF  operations  of  TO9 
complementary  with  each  other  and  with 
interventions 
thematic 
objectives?  What  were  the  main  factors  in 
this regard?  
4.1.  Were  ESF  interventions  in  line  with 
EU policies on social inclusion? 
they 
4.3.  To  what 
complementary  and  coherent  with  other 
EU  funding  instruments  such  as  ERDF, 
EARDF, EaSi, FEAD or AMIF? 
4.4.  To  which  extent  are  the  investments 
under TO9 consistent with the analyses and 
priorities  identified  in  the  context  of  the 

extent  were 

under 

other 

6.3.3 

6.4 

6.4.1 

(1) (3)  
(5) 

(1) (3)  
(5) 

(1) (3)  
(5) 

6.4.2 

(1) (3)  
(5) 

3.3.  To  what  extent  were  OPs  flexible  and  able  to 
adapt  to  changes  in  the  implementation  context  or 
political priorities? 

How  coherent  are  the  operations  funded  by 
thematic  objective  10  among  themselves  and 
with other actions in the same field? 

each  other 

the  ESF  operations  of  TO10 
4.2.  Were 
complementary  with 
and  with 
interventions under other thematic objectives? What 
were  the  main  factors  fostering  and/or  hindering 
complementarity? 
4.1.  Were  ESF  interventions  in  line  with  EU 
policies on education and training? To which extent 
are  the  investments  under  TO10  contributing  to 
other EU initiatives such as the New Skills Agenda 
for Europe? To what extent did they take advantage 
of  or  were  aligned  to  EU  policy  tools,  such  as  the 
Europass  documents,  the  European  Qualifications 
Framework (EQF), the European Quality Assurance 
Framework  in  Vocational  Education  and  Training 

111 

 
 
 
TO8 – Employment and labour mobility 

TO9 – Social inclusion 

TO10 – Education and training 

Section in 
the staff 
working 
document 

Data 
and info 
sources 

European Semester? 
4.4.  To  what  extent  were  they  complementary  and 
coherent  with  other  activities  supporting  employment 
and mobility at national/regional level? 

the 

Country 

European Semester notably  in the  Country 
Reports, the National Reform Programmes 
and, 
Specific 
Recommendations? 
4.5.  To  what 
they 
complementary  and  coherent  with  other 
activities  supporting  social  inclusion  and 
combating  poverty  and  discrimination  at 
national/regional level? 

extent  were 

What  is  the  EU  added  value  of  the  ESF-funded 
operations in the field of employment and mobility? 

5.1.  To  what  extent  did  the  ESF-funded  operations 
produce  effects  at  the  national  and  regional  level  that 
the  EU 
would  not  have 
intervention? This question shall be addressed from the 
following perspectives: 

taken  place  without 

Volume  effect:  Have  the  operations  added  to  existing 
actions or directly produced beneficial effects that can 
be measured in terms of volume?  

What is the EU added value of the ESF-
funded  operations  in  the  field  of  social 
inclusion,  combating  poverty  and  anti-
discrimination? 
5.1.  To  what  extent  did  the  ESF-funded 
operations  produce  effects  at  the  national 
and regional level that would otherwise not 
the  EU 
have 
taken  place  without 
intervention?  This  question 
shall  be 
addressed from the following perspectives: 
Volume  effect:  Have  the  operations  added 
to  existing  actions  or  directly  produced 
beneficial  effects  that  can  be  measured  in 

112 

extent  were 

the  European  Credit 

transfer 
(EQAVET)  and 
system  for  Vocational  Education  and  Training 
(ECVET)?4.3.  To  what 
they 
complementary and coherent with other EU funding 
instruments  such  as  ERDF,  EARDF,  ERASMUS+ 
or AMIF? 
4.4.  To  which  extent  are  the  investments  under 
TO10  consistent  with  the  analyses  and  priorities 
identified  in  the  context  of  the  European  Semester 
notably  in  the  Country  Reports,  the  National 
Reform  Programmes  and  the  Country  Specific 
Recommendations? 
To which extent fulfilling ex ante conditionalities in 
the  different  sectors  led  to  structural  reforms  or 
legislative  changes  in  those  countries/regions  that 
had to work to fulfil them? 
4.5.  To  what  extent  were  they  complementary  and 
coherent with other activities supporting education, 
training and vocational training at national/regional 
level? 
What  is  the  EU  added  value  of  the  ESF-funded 
operations in the field of education? 

5.1  To  what  extent  did  the  ESF-funded  operations 
produce  effects  at  the  national  and  regional  level 
that  would  otherwise  not  have  taken  place  without 
the  EU 
intervention?  This  question  shall  be 
addressed from the following perspectives: 

6.5 

6.5 

(1) (3) 
(5) 

(1) (3) 
(5) 

Volume  effect:  Have  the  operations  added  to 
existing  actions  or  directly  produced  beneficial 
effects that can be measured in terms of volume? 

6.5.1 

(1) (3) 
(5) 

 
TO8 – Employment and labour mobility 

TO9 – Social inclusion 

TO10 – Education and training 

Scope  effect:  Have  the  operations  broadened  existing 
actions by addressing groups or policy areas that would 
otherwise not have been addressed? 

Role effect: Have the operations supported innovation 
and  the  transfer  of  ideas  that  have  been  subsequently 
rolled out in different contexts?  

Process  effect:  Have  Member  State  administrations 
and  participating  organisations  derived  benefits  from 
being  involved  in  the  operations?  (Sustainability 
question) 

How  sustainable  the  ESF-funded  operations  under 
TO8 have been? 
6.1.  To  what  extent  the  effects  of  ESF  support  under 
TO8 are likely to continue after the end of the funding, 
both at individual and policy level? 

effect:  Have 

terms of volume? 
Scope 
the  operations 
broadened  existing  actions  by  addressing 
groups or policy areas that would otherwise 
not have been addressed?  
Role effect: Have the operations supported 
innovation  and  the  transfer  of  ideas  that 
have  been  subsequently  rolled  out 
in 
different contexts?  
Process  effect:  Have  Member  States 
administrations 
participating 
organisations  derived  benefits  from  being 
involved in the operations? To what extent 
the effects of ESF support under TO10 are 
likely  to  continue  after  the  end  of  the 
funding,  both  at  individual,  institutional 
and policy level? 

and 

Scope  effect:  Have 
the  operations  broadened 
existing  actions  by  addressing  groups  or  policy 
areas 
that  would  otherwise  not  have  been 
addressed?  
Role  effect:  Have 
the  operations  supported 
innovation and the transfer of ideas that  have been 
subsequently rolled out in different contexts?  

effect: 

Have  Member 

States 
Process 
administrations  and  participating  organisations 
derived  benefits  from  being 
the 
operations?  To  what  extent  the  effects  of  ESF 
support under TO10 are likely to continue after the 
end  of  the  funding,  both  at  individual,  institutional 
and policy level? 

involved 

in 

Section in 
the staff 
working 
document 

Data 
and info 
sources 

6.5.2 

6.5.3 

6.5.4 

(1) (3)  
(5) 

(1) (3) 
(5) 

(1) (3) 
(5) 

6.5.4 

6.5.4 

(1) (3) 
(5) 
(1) (3)  
(5) 

Data and info sources to reply evaluation questions: (1) – external study; (2) – SFC2014; (3) – national evaluations; (4) – EUROSTAT data; (5) public consultations 

113 

 
 
 
 
 
 
 
 
ANNEX 4: ESF INTERVENTION LOGIC 

Overall objectives 

Specific objectives 

Treaties objectives: 
 
Strengthening 
economic and 
social cohesion 
Improve 
employment 
opportunities 

 

TO 8 ‘Employment 
and labour mobility’  
 
Access to 
employment* 
Support to 
businesses 
  Modernisation of 

 

labour markets 

TO9 ‘Social 
Inclusion’  
 

 

Support for 
specific groups 
Community led 
development 
strategies 
TO 10 ‘Education 
and Training’  
 

ELET and access 
to education and 
training 
Tertiary 
education 
Lifelong learning 
Labour market 
relevance of 
E&T systems 

 

 
 

Europe 2020 strategy 

Integrated guidelines 

Country Specific 
Recommendations 

National Reform 
Programmes 

European Pillar of 
Social Rights 

Inputs 

ESF, CPR 
Regulations 

EU: 37 billion € 
MS: 14 billion € 

EU: 21 billion € 
MS: 10 billion € 

EU: 27 billion € 
MS: 14 billion € 

Outputs 

Results 

Impacts 

  Workers, enterprises, including actors in 

the social economy, and entrepreneurs, 
supported, as well as systems and 
structures with a view to facilitating their 
adaptation to new challenges including 
reducing skill mismatches and promoting 
good governance, social progress. 
People benefited, including disadvantaged 
people such as the long-term unemployed, 
people with disabilities, low skilled, 
migrants, ethnic minorities, marginalised 
communities and people of all ages facing 
poverty and social exclusion. 
Implementation of reforms, in particular 
in the fields of employment, education, 
training and social policies. 

 

 

 

 

 

 

Employment, 
enhanced 
qualifications, 
other positive 
results such as 
increased self-
confidence, 
improved 
employability, 
Labour market 
institutions 
modernised and 
strengthened 
Better 
performing 
education and 
training systems 
Improved 
administrative 
capacity 

 

 

 

 

 

 

 

 

 

Increased levels of 
employment 
Improved access to the 
labour market 
Increased geographical 
and occupational 
mobility of workers  
Enhanced adaptability 
to change 
Increased levels of 
education and training 
Transition from 
education to 
employment facilitated 
Contribution to 
reduced poverty 
Enhanced social 
inclusion 
Gender equality, non-
discrimination and 
equal opportunities 
promoted 

TO 11 ‘Institutional 
capacity’* 

 

EU: 4 
billion €* 

  MS: 1 

billion €* 

114 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
115