COMMISSION STAFF WORKING DOCUMENT Evaluation of the EU Framework for National Roma Integration Strategies up to 2020 Accompanying the document COMMUNICATION FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL Report on the evaluation of the EU Framework for National Roma Integration Strategies up to 2020
Date: 2018-12-04
Institutions: Directorate-General for Justice and Consumers, European Commission
Subjects: Roma, access to education, aid to disadvantaged groups, equal treatment, ethnic discrimination, health policy, housing, marginalisation, rights of minorities, social integration

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 4.12.2018  SWD(2018) 480 final  COMMISSION STAFF WORKING DOCUMENT Evaluation of the EU Framework for National Roma Integration Strategies up to 2020     Accompanying the document COMMUNICATION FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL  Report on the evaluation of the EU Framework for National Roma Integration Strategies up to 2020 {COM(2018) 785 final}  
 
 
TABLE OF CONTENTS 

1. 

2. 

3. 

INTRODUCTION ............................................................................................................................... 2 

BACKGROUND TO THE INTERVENTION ................................................................................. 4 

IMPLEMENTATION / STATE OF PLAY .................................................................................... 11 

4.  METHOD .......................................................................................................................................... 13 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS ...................................... 16 

5.1 EFFECTIVENESS ................................................................................................................................. 16 

5.2. COORDINATION ................................................................................................................................. 26 

5.3. RELEVANCE........................................................................................................................................ 30 

5.4. EQUITY ................................................................................................................................................ 35 

5.5. COHERENCE ....................................................................................................................................... 38 

5.6. EFFICIENCY ........................................................................................................................................ 45 

5.7. EU ADDED VALUE ............................................................................................................................ 50 

5.8 SUSTAINABILITY ............................................................................................................................... 52 

6.  CONCLUSIONS ............................................................................................................................... 55 

ANNEX 1: PROCEDURAL INFORMATION ........................................................................................ 59 

ANNEX 2: STAKEHOLDER CONSULTATION ................................................................................... 63 

ANNEX 3: EVALUATION CRITERIA AND QUESTIONS ................................................................. 72 

ANNEX 4: METHODS AND ANALYTICAL MODELS ....................................................................... 76 

ANNEX 5: COSTS AND BENEFITS ....................................................................................................... 84 

ANNEX 6: OBJECTIVES AND MEASURES 2011 AND 2013 ............................................................. 95 

ANNEX 7A: INTERVENTION LOGIC I .............................................................................................. 101 

ANNEX 7B: INTERVENTION LOGIC II-ZOOM ON THE EU FRAMEWORK ........................... 102 

ANNEX 8: THEMATIC MAPPING OF INSTRUMENTS AND PROCESSES ................................ 103 

 
 
 
 
Glossary 

Term or acronym 

Meaning or definition 

CSR 

ECEC 

ERDF 

EPSCO 

ESF 

ESIF 

ESL 

EU-MIDIS 

EURoma 

FRA 

IP 

IPA 

NGO 

NRCP 

NRIS 

OP 

OPC 

OSCE 

PA 

REC 

SFC 

UNDP 

Country-specific recommendation  

Early childhood education and care 

European Regional Development Fund 

Employment,  Social  Policy,  Health  and  Consumer 
Affairs Council 

European Social Fund 

European structural and investment funds 

Early school leaving 

European Union minorities and discrimination survey 

European  network  on  social  inclusion  and  Roma 
under structural funds 

Fundamental Rights Agency 

Investment priority 

Instrument for pre-accession assistance 

Non-governmental organisation 

National Roma contact point 

National  Roma  integration  strategies  and  integrated 
sets of policy measures 

Operational Programme 

Open public consultation 

Organisation for Security and Cooperation in Europe 

Partnership agreement 

Rights, Equality and Citizenship Programme 

System for fund management in the EU 

United Nations Development Programme 

1 

 
 
 
1. 

INTRODUCTION    

This staff working document (SWD) is a midterm evaluation of the ‘EU framework for 
national  Roma 1  integration  strategies  up  to  2020’  (EU  framework),  adopted  as  a 
Communication2 by the Commission on 5 April 2011 and welcomed by the Council3 in 
May 2011 and the European Council in June 20114. 

This midterm evaluation responds to the request to the Commission5 made in the EPSCO 
Council  conclusions  of  8 December  2016  entitled  ‘Accelerating  the  Process  of  Roma 
Integration6’.  It  aims  to  provide  information  to  the  Commission’s  departments  and  to 
stakeholders to  help  steer the work for the remaining implementation  period  of the EU 
framework (2020) and to prepare for the period thereafter. 

The evaluation assesses the relevance, effectiveness, efficiency, coherence and EU added 
value  of  the  EU  framework.  It  also  assesses  three  further  evaluation  criteria,  namely 
coordination, equity and sustainability. 

In terms of geographic scope, the evaluation mainly focuses on EU Member States, with 
the exception of Malta, which does not have a Roma community. However, it also covers 
the  enlargement  region  (Bosnia  and  Herzegovina,  the  former  Yugoslav  Republic  of 
Macedonia, Serbia, Turkey, Albania, Kosovo*and Montenegro7). 

Regarding the thematic scope, the evaluation focuses on how the EU framework and its 
objectives  are  working,  notably  in  the  areas  of  poverty,  discrimination,  education, 
employment, health and housing. The evaluation covers in particular: 

  changes  in  the  situation  of  Roma  in  the  areas  of  poverty,  discrimination  and 
access to education, employment, healthcare and housing since the EU framework 
was adopted and the extent to which changes are attributable to this initiative;  
the  existence  of  preconditions  and  structures  for  Roma  integration  at  national 
level  as  provided  for  in  the  EU  framework,  such  as  national  Roma  integration 
strategies (NRIS) and national Roma contact points (NRCPs); 

 

  alignment  of  EU  level  and  national  policies  and  mobilisation  of  legal  and 
financial instruments (e.g. the European Semester and EU funding programmes) 
to help meet the objectives of the EU framework; 

1 Here, the term ‘Roma’ is used in line with the terminology of European institutions and international organisations, 
to  refer  to  a  number  of  different  groups  (such  as  Roma,  Sinti,  Kale,  Gypsies,  Romanichels,  Boyash,  Ashkali, 
Egyptians, Yenish, Dom, Lom, Rom, Abdal) and also includes travellers, without denying the unique features and 
varieties of lifestyles and situations of these groups. 

2 COM(2011) 173 final. 
3 Council Conclusions (EPSCO) of 19 May 2011 on an EU Framework for national Roma integration strategies up to 

2020. 

4 https://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/ec/123075.pdf. 
5 The Council conclusions urge the Commission to ‘carry out a mid-term evaluation of the EU framework for national 
Roma  integration  strategies  up  to  2020  and  to  propose  a  post  2020  strategy  on  Roma  integration,  and  include 
therein a proposal for a revision of the Council Recommendation.’. 

6 Council Conclusions (EPSCO) of 8 December 2016:  http://data.consilium.europa.eu/doc/document/ST-15406-2016-

INIT/en/pdf 

*  This  designation  is  without  prejudice  to  positions  on  status,  and  is  in  line  with  UNSCR  1244/1999  and  the  ICJ 

Opinion on the Kosovo declaration of independence. 

7 The last four countries through desk research only. 

2 

 
 
 
 
 
 
 
                                                           
 

 

 

 

the  role  of  the  EU  framework  and  NRIS  in  integrating  Roma  inclusion  into 
mainstream  policies  and  in  stimulating  cooperation  and  dialogue  between 
stakeholders, including civil society; 
the  use  of  EU  and  national  funding  to  implement  NRIS  and  Roma  integration 
measures; 
the  impact  of  the  EU  framework  on  promoting  Roma  integration  in  the 
enlargement region; and 
lessons  learnt  for  the  remaining  implementation  period  and  for  the  period  after 
2020. 

The  thematic  scope  does  not  include  an  evaluation  of  other  legal,  policy  or  funding 
instruments  that  are  directly  or  indirectly  relevant  for  Roma  integration,  such  as  the 
Racial Equality Directive, the European Semester or the European structural funds. For 
these  instruments,  only  their  alignment  with  the  EU  framework’s  objectives  and  their 
mobilisation  to  advance  Roma  integration  and  fight  discrimination  is  assessed8.  Where 
available and relevant, the SWD draws on existing evaluations9 of these instruments. 

The evaluation covers the period April 2011 (start of the intervention) to the end of 2017. 
Comparable10 data  on  the  living  conditions  of  Roma  for  the  period  2011-2016  for  EU 
Member States and 2011-2017 for enlargement countries have been used. 

The evaluation has been informed by an external study requested by the Commission and 
prepared by ICF Consulting Services Ltd and Milieu (hereafter ‘ICF’).11 

8 The scope therefore also excludes an assessment of efficiency at project level, which does not allow for reporting on 

policy indicators at population level. 

9 In particular, ex post and midterm evaluations of EU funding programmes and implementation reports on the Racial 

Equality Directive. 

10 With the limitations explained in Annex 4. 
11 Add link once published. 

3 

 
 
 
 
 
 
                                                           
2.  BACKGROUND TO THE INTERVENTION   

2.1. Description of the intervention and its objectives 

Vice-President  Viviane  Reding,  08  April  2011:  ‘The  EU  Framework  comes  at  a 
moment when the situation of Roma has become the centre of attention — and a matter of 
collective shame for Europe. (…) Action is needed both at national and at EU level. We 
need to join forces and step up our efforts to end discrimination against the Roma and to 
make sure they enjoy the same rights as any other EU citizen. That is why the European 
Commission  has  come  up  with  a  new  and  innovative  instrument:  an  EU  Framework 
designed especially for Roma economic and social integration in all Member States12’. 

Wider policy developments at the time of adoption of the EU framework include: 

From  an  economic  perspective,  the  financial  and  economic  crisis  wiped  out  years  of 
economic  and  social  progress,  leaving  millions  of  people  unemployed,  exposing 
structural  weaknesses  in  Europe’s  economy  and  putting  new  pressure  on  social 
cohesion13.  In  response,  in  its  Europe  2020  strategy  adopted  in  2010,  the  Commission 
proposed five measurable EU targets  for 2020 to be converted into national  targets.  Of 
these targets, the three related to employment, education and poverty are directly relevant 
for Roma integration.  

From  an  equality  perspective,  a  decade  had  passed  since  adoption  of  the  two  anti-
discrimination  Directives14 which  Member  States  were  required  to  implement  by  2003 
and which, among other grounds, prohibit discrimination on grounds of race and ethnic 
origin.  The  Framework  Decision  combating  racism  and  xenophobia15 was  adopted  in 
2008 and implementation of all three pieces of legislation was being monitored16. With 
regard  to  other  equality  policies,  the  Commission  adopted  its  2010-2015  strategy  for 
equality  between  women  and  men  in  September  2010.  This  strategy  underlined  the 
cumulative  effect  of  discrimination  on  two  or  more  grounds,  for  example,  for  being 
Roma and a woman17. 

Concerning  the  free  movement  of  people,  the  dialogue  between  the  Commission  and 
some national  authorities on how the relevant  EU law should be applied to  address  the 
situation of the Roma had intensified in 2010. The Commission had called upon Member 
States  to  respect  the  rules  laid  down  in  the  2004  Directive  on  free  movement18 with 
respect to the fundamental rights of EU citizens and avoiding discrimination, notably on 
the grounds of nationality or ethnic group19. In addition, transitional arrangements on the 
free movement of workers that had been agreed in the Accession Treaties of Bulgaria and 

12 Speech of European Commission Vice-President Viviane Reding, 8 April 2011, Budapest. 
13 COM(2010) 2020 final; EUROPE 2020 — A strategy for smart, sustainable and inclusive growth. 
14 Council Directive 2000/43/EC and Council Directive 2000/78/EC of 29 June 2000. 
15 In addition to the above: Council Framework Decision 2008/913/JHA of 28 November 2008 on combating certain 

forms and expressions of racism and xenophobia by means of criminal law, OJ L 328, 6.12.2008. 

16 See in particular Commission Staff Working Document ‘Roma in Europe: The Implementation of European Union 
Instruments and Policies for Roma Inclusion — Progress Report 2008-2010’; SEC(2010) 400 final of 7 April 2010 
for detail. 

17 COM(2010) 491 final. 
18 Directive 2004/38/EC of 29 April 2004 on the right of citizens of the Union and their family members to move and 

reside freely within the territory of the Member States. 
19 http://europa.eu/rapid/press-release_IP-10-1207_en.htm 

4 

 
 
 
 
                                                           
Romania still allowed the remaining Member States to temporarily restrict workers from 
Bulgaria  and  Romania  from  working  in  their  country.  This  restriction  had  an  effect  on 
Roma from these two countries, as the last restrictions on the free movement of workers 
from Bulgaria and Romania were only lifted on 1 January 2014. 

International, European and national policies and initiatives for Roma integration started 
from 2005 and continued in the years preceding the EU framework. These include: 

 
 

 

the Decade of Roma Inclusion 2005-201520; 
the  2008  Commission  Communication21 highlighting,  among  other  things,  that 
the tools to advance the social inclusion of Roma should be better applied; 
the common basic principles on Roma inclusion attached to the EPSCO Council 
conclusions on Roma inclusion22 of June 2009; 

  a  2010  Commission  Communication  focusing  on  the  social  and  economic 

 

integration of Roma in Europe23; and 
the  European  Parliament’s  Resolution  of  9 March  2011  on  the  EU  strategy  on 
Roma inclusion24. 

Despite  these  efforts,  the  economic  and  social  marginalisation  of  Roma  persisted  and 
many  of  the  estimated  10-12  million  Roma  in  Europe25 continued  to  face  prejudice, 
intolerance,  discrimination  and  social  exclusion  in  their  daily  lives.  EU  countries  were 
(and are) characterised by different sizes of the Roma population living in their territories 
and  different  starting  points26 in  terms  of  Roma  integration  policies.  The  estimated 
shares  of  Roma  in  the  EU  countries  in  201227 range  from  10.3%  in  Bulgaria,  9.1%  in 
Slovakia, 8.3% in Romania, 7% in Hungary, 2.5% in Greece, 2% in the Czech Republic, 
1.6% in Spain to less than 1% in most of the other countries.  

The  fact  that  Roma  were  marginalised  and  lived  in  very  poor  socio-economic 
conditions28 was  increasingly  seen  as  an  economic  disadvantage,  resulting  in  welfare 
dependency and limiting the labour markets and tax revenues29 30, in addition to being a 
question of European values and human rights 

The  Commission’s  policy  response  was  to  adopt  the  EU  framework.  This  was  the 
first  EU  initiative  specifically  concerning  Roma  which  included  a  follow-up 

20 The Decade of Roma Inclusion 2005-2015 was an initiative that brought together governments, intergovernmental 
and  nongovernmental  organisations,  as  well  as  Romani  civil  society.  Formally  established  in  February  2005  in 
Sofia,  it  brought  together  the  governments  of  Albania,  Bosnia  and  Herzegovina,  Bulgaria,  Croatia,  the  Czech 
Republic,  Hungary,  Macedonia,  Montenegro,  Romania,  Serbia,  Slovakia  and  Spain  (Slovenia,  the  United  States, 
Norway  and  Moldova  had  observer  status)  in  a  joint  effort  to  eradicate  discrimination  and  ‘close  gaps  between 
Roma and the rest of society’. The initiative was formally closed in 2015. 

21 COM(2008) 420 final. 
22 http://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/lsa/108377.pdf. 
23 COM(2010) 133 final. 
24 European Parliament, Resolution of 9 March 2011 on the EU strategy on Roma inclusion, 2010/2276(INI). 
25 Council of Europe estimates included as an Annex to COM(2011) 173 final. 
26 In particular in terms of participation in the Decade of Roma Inclusion since 2005 
27 Council of Europe estimates 
28 COM(2011) 173 final. 
29 InGrid  —  Integrating  expertise  in  inclusive  growth;  Methodological  and  Data  Infrastructure  Report  on  Roma 

Population in Europe; August 2016. 

30 World  Bank  (2010)  —  Roma  Inclusion:  An  economic  opportunity  for  Bulgaria,  Czech  Republic,  Romania  and 

Serbia. 

5 

 
 
 
 
                                                           
mechanism31. The Commission proposed that Member States, in proportion to the 
size of the Roma population living in their territories and taking into account their 
different starting points, design national Roma integration strategies (NRIS) and set 
achievable national goals for Roma integration or, where they already exist, adapt 
them to meet four EU Roma integration goals: 

1. Ensure that all Roma children complete, as a minimum, primary school education. 
2. Cut the employment gap between Roma and the rest of the population. 
3. Reduce the gap in health status between the Roma and the rest of the population. 
4. Close the gap between the share of Roma with access to housing and to public utilities 
(such as water, electricity, and gas) and that of the rest of the population. 

The  framework  did  not  set  a  specific  EU  Roma  integration  goals  in  the  area  of  non-
discrimination. Instead, it underlined that Member States are already under an obligation 
to  ensure  that  Roma  are  not  discriminated  against,  as  laid  down  in  the  Racial  Equality 
Directive,  i.e.  to  give  Roma  non-discriminatory  access  to  education,  employment, 
vocational training, healthcare, social protection and housing. Its conclusions stress that 
the framework is  complementary to  existing EU legislation and policies in  the areas of 
non-discrimination, fundamental rights, the free movement of persons, and the rights of 
the child.  

The framework underlines that non-discrimination is not sufficient to combat the social 
exclusion  of  Roma.  Member  States  have  the  primary  responsibility  as  well  as  the 
competence  to  improve  the  situation  of  marginalised  populations.  Action  to  support 
Roma  integration  lies  first  and  foremost  in  their  hands,  taking  into  account  specific 
national circumstances, needs and required solutions. The EU framework acknowledges 
this,  but  due  to  shared  values  (such  as  equality  and  fundamental  rights)  and  common 
objectives (such as economic prosperity, social cohesion and solidarity between Member 
States)32,  it  calls  for  a  European  role  in  policies  for  Roma  integration.  A  key  driving 
force  for  the  EU  framework  was  transnational  mobility  of  Roma  in  the  context  of 
freedom of movement, making Roma integration a joint EU-level objective. 

To  achieve  each  of  the  four  Roma  integration  goals  mentioned  above  (the  specific 
objectives),  the  EU  framework  took  a  non-binding  approach,  listing  a  set  of  measures 
Member States should take to make progress. These measures take account of different 
situations and starting points in terms of national strategies already in place and the size 
of  the  Roma  population.  The  EU  framework  also  requests  that  when  developing  their 
NRIS Member States: 

  allocate  sufficient  funding  from  national  budgets,  complemented  where 

 

appropriate by international and EU funding;  
include  robust  monitoring  methods  to  evaluate  the  impact  of  Roma  integration 
actions, and a review mechanism so that the strategy can be adapted if necessary; 
  design, implement and monitor the strategy in close cooperation and continuous 

dialogue with Roma civil society and regional and local authorities; and 

31 European  Court  of  Auditors,  Special  Report:  EU  policy  initiatives  and  financial  support  for  Roma  integration: 
significant  progress  made  over  the  last  decade,  but  additional  efforts  needed  on  the  ground,  2016, 
https://www.eca.europa.eu/Lists/ECADocuments/SR16_14/SR_ROMA_EN.pdf. 

32 COM(2012) 226 final. 

6 

 
 
 
 
 
                                                           
  appoint a national contact point for the NRIS, with the authority to coordinate the 
development  and  implementation  of  the  strategy  or,  where  relevant,  rely  on 
suitable existing administrative structures. 

Overall,  the  EU  framework  was  ambitious  in  terms  of  EU  guidance  and  new 
features (such as EU Roma integration goals and national goals, development of the 
National  Roma  Integration  Strategies,  appointment  of  National  Roma  Contact 
Points). And it was flexible in terms of concrete approaches and responses expected 
from  Member  States,  taking  into  account  the  size  of  the  Roma  population  and 
different starting points and having a non-binding approach for national measures. 

In addition, the EU framework: (i) encouraged Member States to make use of available 
EU  funds  to  address  the  needs  of  Roma;  (ii)  stressed  the  importance  of  making 
European  and  national  policies  more  sensitive  to  Roma  needs,  including  through 
empowering  civil  society;  and  (iii)  highlighted  the  need  to  put  in  place  a  robust 
monitoring  system.  With  regard  to  the  enlargement  region,  the  EU  framework 
emphasised  that  the  EU  Roma  integration  goals  are  equally  relevant  to  enlargement 
countries.  It  included  three  main  goals:  i)  improve  the  delivery  of  pre-accession 
assistance, ii) strengthen the involvement of the civil society, iii) enhance the monitoring.  

2.2 Intervention logic and baseline 

The EU framework’s theory of change began with the observation that discrimination on 
the  basis  of  racial  or  ethnic  origin  in  education,  employment,  health  and  housing  and 
other  areas  was  already  prohibited  by  EU  law,  but  that  non-discrimination  alone  is  not 
sufficient  to  combat  the social  exclusion  of Roma33.  Therefore,  to  fight  discrimination 
and  combat  poverty  (general  objectives),  countries  need  to  develop  and  implement  an 
integrated  and  sustainable  approach,  with  efforts  spanning  different  policy  areas  — 
education,  employment,  health  and  housing34 in  particular  (Roma  integration  goals  
specific objectives). The expectation was that in the long term this approach would lead 
to social and economic benefits for Roma and non-Roma35 alike. 

When the EU framework was adopted, no intervention logic had been prepared. It 
has  therefore  been  developed  for  this  evaluation  and  has  two  elements:  Annex  7a 
(intervention  logic  I 36 )  provides  an  overview  of  key  instruments,  governance 
structures  and activities available to advance  Roma integration at EU, national and 
local levels. Inputs and activities directly related to the EU framework are highlighted by 
colouring  them  in  red.  This  annex  illustrates  that  in  working  towards  the  Roma 
integration  goals,  the  EU  framework  does  not  operate  independently  of  other  legal, 
policy  and  financial  instruments  but  mobilises  and  aligns  these  instruments  and  their 
concrete activities to reach its specific objectives. It highlights that outputs, outcomes and 
impacts of the EU framework depend on the smooth running of these other instruments 
as  well  as  on  commonly  shared  external  factors.  The  annex  also  illustrates  that  the 
expected key outputs included increased funding for Roma integration, a higher number 
of  Roma  beneficiaries,  and  more  effective  targeted  and  mainstream  policy  measures.  It 
shows that the key expected outcomes of the intervention were the achievement of the 

33 COM(2011) 173 final. 
34 COM(2012) 226 final. 
35 COM(2011) 173. 
36 This intervention logic was included in the terms of reference agreed by the ISSG. 

7 

 
 
 
 
 
                                                           
Roma  integration  goals  in  2020,  increased  capacity  of  those  involved  at  national  and 
local level to implement Roma integration measures as well as strengthened cooperation 
between  stakeholders;  and  that  the  expected  key  impacts  include  socio-economic 
inclusion, less discrimination of Roma and economic, fiscal and societal benefits such as 
a  rise  in  GDP  through  better  education  outcomes  and  a  more  skilled  workforce.  Some 
external factors influencing effectiveness were already apparent when the framework was 
adopted. These include the effect of the economic crisis on the labour market, high levels 
of  discrimination  preventing  some  goals  from  being  quickly  or  fully  achieved  and  the 
political  will  required 
the  EU  framework’s  objectives.  Annex  7b 
(intervention  logic  II)  focuses  more  specifically  on  the  EU  framework,  setting  out  its 
objectives, its interaction with Member States and enlargement countries and its role in 
mobilising and aligning other EU and national policy, legal and financial tools for Roma 
integration. The expected outputs, outcomes and impacts remain the same. 

to  achieve 

Information about the baseline situation of Roma in the EU and the enlargement region 
is still incomplete, with gaps in most Member States, at least for certain subgroups such 
as  migrant  or  EU-mobile  Roma37.  The  fact  that  no  impact  assessment  was  carried  out 
when  the  EU  framework  was  prepared  also  affects  this  evaluation.  Furthermore,  early 
studies  and  reports  by  European  institutions  showed  a  lack  of  systematic  and  in-depth 
knowledge  of  the  situation  of  the  Roma38.  Most  information  was  collected  and  made 
available  thanks  to  efforts  of  the  United  Nations  Development  Programme  (UNDP)  in 
central  and  eastern  Europe 39 .  The  baselines  sources  mentioned  in  the  2011 
Communication therefore refer to  partial studies covering a limited number of Member 
States40.  The  EU  institutions’  first  systematic  (but  still  incomplete)  mapping  of  the 
situation  of  the  Roma  (in  Member  States  where  this  could  be  done  using  quantitative 
methods) followed the adoption in 2011 of the EU framework. This first attempt was the 
survey  of  11  EU  Member  States41 by  the  Fundamental  Rights  Agency  (FRA),  which 
found that in 2011, Roma people were significantly discriminated against across the 
EU.  Eurobarometer  surveys  from  2012  also  provide  insights  into  the  prevalence  and 
nature of discrimination  faced by Roma. When asked how comfortable citizens in their 
country would feel if their children had Roma schoolmates, 34% thought that citizens in 
their country would feel uncomfortable with such situation (28% fairly comfortable and 
31% comfortable). The 2011 UNDP/World Bank/EC Regional Roma survey42 revealed 
significant gaps between Roma and non-Roma in Western Balkan countries. 
The table below shows the situation of EU Roma in 2011, versus the general population, 
for a number of indicators. As explained above, such data on Roma did not yet exist at 
the  time  of  adoption  of  the  EU  framework.  Eurobarometer  findings  on  perception  of 
Roma in the general public were not included as questions in the 2012 and 2015 surveys 
were different and findings thus not comparable. 

37 Roma from the EU using their right to freedom of movement within the EU 
38 E.g. European Commission Directorate-General for Employment and Social Affairs, Unit D3, (2004) ‘The Situation 
of Roma in an Enlarged European Union’; FRA (November 2009) The Situation of Roma EU Citizens moving to 
and Settling in other EU Member States. 

39 UNDP, Bratislava 2002 ‘Avoiding the Dependency trap — the Roma in Central and Eastern Europe’. UNDP later 

conducted two rounds of quantitative data collection (in 2004 and 2011). 

40 E.g.  Open  Society  Institute,  international  comparative  data  set  on  Roma  education,  2008.  World  Bank,  Roma 
Inclusion:  An  Economic  Opportunity  for  Bulgaria,  the  Czech  Republic,  Romania  and  Serbia,  September  2010. 
Fundación Secretariado Gitano, Health and the Roma community, analysis of the situation in Europe, 2009. 

41 European  Union  Agency  for  Fundamental  Rights,  The  Situation  of  Roma  in  11  EU  Member  States  —  Survey 

results at a glance, Publications Office of the European Union, 2012, p. 26. 

42 http://www.eurasia.undp.org/content/rbec/en/home/ourwork/sustainable-development/development-planning-and-

inclusive-sustainable-growth/roma-in-central-and-southeast-europe/roma-data.html 

8 

 
 
                                                           
Table 1: Comparing the situation of Roma and the general population in 2011 

Baseline indicators 

Roma43 

General 
population44 

10 % 

17 % 

86 % 

47 % 
87 % 
86 % 

Poverty 
At risk of poverty  
Education 
Share of children +4 attending ECEC 
Early leavers from education and training 
Share of children in compulsory schooling age (5-17 
depending on the country) attending education 
Share of people that felt discriminated in the past 5 
years  because  of  being  Roma  when  in  contact  with 
school 
Share of Roma children aged 6-15 attending classes 
where all classmates are Roma 
Employment 
Share  of  people  with  ‘paid  work’  as  their  self-
declared main activity status, 16+ (%)49 
Share  of  young  people  aged  16-24  years  old  with 
current  main  activity  as  neither  employment, 
education  or  training  (NEET),  household  members 
(%)51 
Share of people that felt discriminated in the past 5 
years because of being Roma when looking for a job 
Share  of  people  that  felt  discriminated  against  at 
work in the past 5 years because of being Roma  
Health 
Share in ‘very good’ or ‘good’ health (self-reported)  55 % 
Share with medical insurance coverage 
78 % 
Housing 
Share  of  people  living  in  households  having  neither 
toilet, shower, nor bathroom inside the dwelling 
Average  number  of  rooms  per  person  in  the  0.6 

26 % 

36 % 

56 % 

19 % 

50 % 

19 %45 

90 %46 
21 %47 
98 %48 

- 

- 

70 %50 

16 %52 

- 

- 

68 %53 
94.5 %54 

7 %55 

1.4656 

43 Based  on  SWD  (2017)  458  —  FRA  Roma  integration  scoreboard  and  FRA  (2018)  A  Persisting  Concern:  anti-

Gypsyism as barrier to Roma inclusion 

44 Source is Eurostat, covering the same countries as FRA for Roma and weighted. 
45 General population: Own calculations (weighted average) based on Eurostat (t2020_52) EU-SILC 2014; Eurostat, 

EU-SILC 2011 

46 General population: Own calculations (weighted average) based on Eurostat: 2011 - educ_ipart (downloaded 

07/06/2018) 

47 General population: Own calculations (weighted average) based on Eurostat, Labour Force Survey (LFS) 2011 

((edat_lfse_14, downloaded 06/07/2018 ) 

48 FRA (2018) A Persisting Concern: anti-Gypsyism as barrier to Roma inclusion 
49 FRA uses paid work as a proxy for employment rates. 
50 General  population:  Own  calculations  (weighted  average)  based  on  Eurostat,  Labour  Force  Survey  (LFS)  2011 

([lfsi_emp_a, downloaded 06/07/2018); and Eurostat edat_lfse_20 (downloaded 06/07/2018). 

51 Comparability  between  EU-MIDIS  II/Roma  Survey  and  Eurostat  NEET  rate  is  restricted  due  to  the  different  age 
bands. Taking 15 year-olds into account would show values lower by a few percentage points for those who are not 
in employment, training or education. The Eurostat NEET rate is based on the ILO concept, which refers to having 
worked at least 1 hour in the past week, whereas EU-MIDIS II asked about the self-declared main activity and any 
paid work in the past 4 weeks (FRA, 2018). 

52 idem 
53  General  population:  Own  calculations  (weighted  average)  based  on  Eurostat,  EU-SILC  2011  ([hlth_silc_01] 

(download 06/07/2018);   

54 OECD Health Database; Health at Glance: Europe reports 2010; (download 06/07/2018) 
55 General population: Own calculations (weighted average) based on Eurostat **2011 [env_wat_pop]  (download 

06/07/2018) 

56 General population: Own calculations (weighted average) based on Eurostat EU-SILC ilc_lvho04d. 

9 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
                                                           
household 
Share of people that felt discriminated against in the 
past  5  years  when  looking  for  housing  because  of 
being Roma  

45 % 

- 

For the Roma integration goals more specifically, this translates into the following 
reconstructed baseline not available at the time of adoption of the EU framework: 

Table 2: 2011 reconstructed baseline for the Roma integration goals 

Roma integration goal 

Value/Gap 
201157 

Access  to  education:  Ensure  that  all  Roma  children  complete  at  least  primary  
school 

86 %58 

Access  to  employment:  Cut  the  employment  gap  between  Roma  and  the  rest  of 
the population 

44 pps59 

Access to healthcare: Reduce the gap in health status between the Roma and the 
rest of the population 

13 pps60 

Access  to  housing:  Close  the  gap  between  the  share  of  Roma  with  access  to 
housing  and  to  public  utilities  (water,  electricity,  gas)  and  that  of  the  rest  of  the 
population 

29 pps61 

0.66 
rooms62 

57  Based  on  SWD  (2017)  458  —  FRA  Roma  integration  scoreboard  and  FRA  (2018)  A  persisting  concern: 

antigypsyism as barrier to Roma inclusion for Roma; based on Eurostat for general population 

58 Shown value (participation rate) is for education attendance when in compulsory schooling age (5-17 depending on 

the country); Source: SWD (2017) 458 — FRA Roma integration scoreboard 
59 Gap in the share of people 16+ who self-declared main activity status "paid work" 
60 Gap in the share of people 16+ in 'very good' or 'good health' (self reported) 
61 Gap  in  the  share  of  people  living  in  households  having  neither  a  toilet,  nor  shower,  nor  bathroom  inside  the 

dwelling. 

62 Gap in the average number of rooms per person in the household. 

10 

 
 
 
 
 
 
 
 
                                                           
3. 

IMPLEMENTATION / STATE OF PLAY  

The  EPSCO  Council  conclusions  from  May  201163 welcomed  the  EU  framework  and 
‘invited  Member  States  to  set  or  continue  working  towards  their  goals,  in  accordance 
with the Member States’ policies, in the fields of education, employment, healthcare and 
housing  with  a  view  to  closing  the  gaps  between  marginalised  Roma  communities  and 
the general population’. As such, the conclusions did not explicitly endorse the EU Roma 
integration  goals.  The  political  importance  of  Roma  integration  was  recognised  by  the 
European  Council  in  June  2011 64 which  called  for  the  Council’s  conclusions  to  be 
implemented  rapidly.  It  particularly  requested  that    Member  States  prepare,  update  or 
develop NRIS or integrated sets of policy measures for improving the situation of Roma 
within their broader social inclusion policies, by the end of 2011. 

By  2012,  all  Member  States  except  for  Malta 65  had  submitted  their  strategies  or 
integrated  sets  of  policy  measures66.  That  same  year,  the  Commission  presented67 the 
results of a first assessment of all NRISs and invited Member States to consider a number 
of  adjustments  for  the  future.  The  assessment  concluded  that  much  more  needed  to  be 
done  at  national  level.  In  particular,  more  concrete  measures,  explicit  targets  for 
measurable  deliverables,  clearly  earmarked  funding  at  national  level  and  a  robust 
national monitoring and evaluation system68 would be necessary. In its Communication 
of  26 June  2013  entitled  ‘Steps  forward  in  implementing  NRIS’69,  the  Commission 
stressed the need for further action on the preconditions necessary to promptly carry out 
measures to speed up Roma integration. 

In  response,  in  December  2013  the  Council  Recommendation 70  on  effective  Roma 
integration  measures 
the  Member  States  was  adopted  unanimously.  The 
Recommendation,  which  was  the  first  EU  soft  law  instrument  explicitly  targeting 
Roma: 

in 

  provided guidance to Member States to make their measures for Roma integration 

 

 

more effective and strengthen implementation of their NRIS; 
recommended that Member States take effective policy measures to ensure equal 
treatment  for  Roma  and  the  respect  of  their  fundamental  rights,  including  equal 
access to education, employment, healthcare and housing; 
indicated  how  each  of  the  four  goals  —  ensuring  equal  access  to  education, 
employment, healthcare and housing — could be met; 

  set out that these goals could be achieved either through mainstream or targeted 
measures, including specific ones to prevent or compensate for disadvantages, or 
by a combination of both, paying special attention to the gender dimension; 

63 Council Conclusions (EPSCO) of 19 May 2011 on an EU Framework for National Roma Integration Strategies up 

to 2020. 

64 https://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/ec/123075.pdf. 
65 Malta does not have a Roma community. 
66 Or  integrated  sets  of  policy  measures  in  line  with  the  Council  Conclusions  (EPSCO)  of  19 May  2011  on  an  EU 
Framework for National Roma Integration Strategies up to 2020. Strategies and integrated sets of policy measures 
are referred to as NRIS. 

67 COM(2012) 226 final. 
68 COM(2012) 226 final. 
69 COM(2013) 454 final. 
70 Council  Recommendation  (EPSCO)  of  9 December  2013 on  effective  Roma  integration  measures  in  the  Member 

States 

11 

 
 
 
 
                                                           
 

reinforced the focus on anti-discrimination, referring explicitly to antigypsyism71, 
and  extended  the  work  on  Roma  integration  to  new  horizontal  and  structural 
areas72 beyond employment, education, health and housing; and 

  called on Member States to communicate to the Commission on an annual basis, 
starting  from  2016,  the  measures  taken  in  line  with  the  Recommendation  and 
progress achieved in implementing the strategies. 

For the enlargement countries it is important to note that in 2013, the Communication on 
the  enlargement  strategy  and  main  challenges  2013-2014  identified  access  to  civil 
documentation73 as a priority for enlargement countries, in addition to the four areas of 
employment, education, housing and health74. 

Individual  assessments  of  Member  States’  NRIS  were  presented  by  the  Commission  in 
2012, 201475 and 2016. The 2016 assessment identified some positive trends, such as the 
growing focus on early childhood education, but underlined that — overall — efforts had 
not prevented further deterioration in living conditions of Roma or widespread hostility 
among the general population. In its Communication, the Commission called on Member 
States to reaffirm political commitment to Roma integration so that the legal, policy and 
financial  instruments  in  place  could  be  fully  used  to  bring  tangible  results  on  the 
ground 76 77 .  In  response,  the  Council  adopted  Conclusions  in  December  2016  on 
accelerating  the  process  of  Roma  integration 78 .  Member  States  reconfirmed  their 
commitment  to  the  Roma  integration  process  and  their  determination  to  ensure  that  all 
policy, legal and financial instruments put in place at European and national level be used 
to close the gap between Roma and non-Roma. The Commission’s 2017 midterm review 
of the EU framework79 took stock of progress achieved80 since 2011 by presenting the 
findings of the Roma integration scoreboard. It urged Member States to intensify efforts 
and  provided  guidance  on  how  to  prioritise  actions.  It  announced  that  a  midterm 
evaluation would be launched in order to analyse the EU framework's achievements and 
challenges in more depth, consult stakeholders and use external expertise. 

The  EU  framework  led  to  the  adoption  of  NRIS  and  the  establishment  of  NRCPs  at 
national level (see table below). In total, 15 Member States81 have adopted strategies and 
the remaining 1282 have adopted integrated sets of policy measures within their broader 

71 The Council of Europe’s European Commission against Racism and Intolerance (ECRI) defines antigypsyism as a 
‘specific  form  of  racism,  an  ideology  founded  on  racial  superiority,  a  form  of  dehumanisation  and  institutional 
racism  nurtured  by  historical  discrimination,  which  is  expressed,  among  others,  by  violence,  hate  speech, 
exploitation, stigmatisation and the most blatant kind of discrimination’. 

72 Examples  of  ‘horizontal  areas’  are:  the  protection  of  Roma  women  and  children,  including  human  trafficking, 
poverty reduction, empowerment; examples of ‘structural areas’ are: bodies for the promotion of equal treatment, 
transnational cooperation. 

73 Such as ID cards,birth, marriage and death certificates 
74 COM(2013) 700 final. 
75 COM(2014) 209 final. 
76 COM(2016) 424 final. 
77  The  2016  Communication  also  included  for  the  first  time  detailed  information  on  Roma  integration  for  the 
enlargement region. The report included country fiches presenting the state of play in the four key priority areas, 
the EU dialogue and monitoring structures as well as the funds allocated under IPA I for Roma integration projects. 

78 14294/16. 
79 COM(2017) 458 final. 
80 For a short summary, see press release: http://europa.eu/rapid/press-release_IP-17-2961_en.htm 
81 BG, CZ, IE, EL, ES, HR, IT, HU, PL, PT, RO, SI, SK, FI, SE. 
82 BE, DK, DE, EE, FR, CY, LV, LT, LU, NL, AT, UK. 

12 

 
 
 
 
                                                           
social  inclusion  policies83.  The  reasons  provided  by  these  12  Member  States  for  not 
having developed strategies are: (i) the small size of the Roma population84; (ii) the legal 
impediments  to  adopting  a  strategy  that  targets  one  specific  ethnic  group85;  and  (iii) 
insufficient  evidence  of  discrimination  against  Roma,  which  deemed  a  strategy 
unnecessary86.  

Most Member States that adopted NRIS did so in response to the EU framework. As of 
2016,  all  enlargement  countries  had  adopted  an  NRIS  and  action  plans  and  had  put  in 
place NRCPs87. Some Member States had bodies in place prior to the EU framework to 
coordinate  matters  on  ethnic  minority  issues,  including  Roma  issues  but  they  were 
formally nominated as NRCPs following adoption of the EU framework88.  

Table  3:  Which  of  the  following  instruments  and  governments  structures  were 
established at Member State in response to the EU framework for NRIS89? 

NRIS 

NRCP 

Yes 

No 

NL, UK, EE, SE 

AT,  BE,  BG,  CY,  CZ, 
DE,  DK,  EL,  ES,  IE, 
LT,  LU,  LV,  PT,  SK, 
Former 
Yugoslav 
Republic  of  Macedonia, 
Montenegro, Turkey 

HR 

(1999), 

Pre-existed  (year  of 
adoption) 
FI 
(2003), SI 
Albania (2003), Bosnia and 
(2005), 
Herzegovina 
Kosovo 
Serbia 
(2009) 

(2009), 

SE 

BG,  ES,  FI  (1956), 
HR (1991), SI, SK 

Bosnia 

AT,  BE,  CY,  CZ,  DE, 
DK,  EE,  EL,  IE,  LT, 
LU,  LV,  NL,  PT,  UK, 
and 
Albania, 
Herzegovina, 
former 
of 
Yugoslav 
Kosovo, 
Macedonia, 
Serbia,  
Montenegro, 
Turkey 

Republic 

Source: Interviews with NRCPs; *The table is based on interviews with NRCPs — not all NRCPs responded 
to  these questions  and  therefore  not  all  Member States  are  covered.  In  addition,  enlargement  countries 
were only partially covered by interviews. DG NEAR provided additional information. 

4.  METHOD  

An external evaluation carried out by a team of consultants from October 2017 to May 
2018  provided  information  for  the  current  midterm  evaluation.  The  methodological 
approach chosen for the external study is described in detail in Annex 4. In this SWD, the 
Commission has built on the findings and conclusions of the external evaluation, unless 
highlighted  otherwise  in  the  text.  In  addition,  to  prepare  the  midterm  evaluation,  the 

83 In  line  with  the  2011  Council  conclusions,  both  approaches  stimulate  the  implementation  of  inclusion  policies. 
Therefore, the remaining analysis does not differentiate between countries with strategies and those with integrated 
sets of policy measures. Both are referred to as NRIS. 

84 DK, DE, LU. 
85 DE, FR. 
86 CY. 
87 European Commission, ‘Communication from the Commission to the European Parliament and the Council on the 

Midterm review of the EU framework for national Roma integration strategies’  COM(2017) 458 final. 

88 BG, SI, SK. 
89 ICF, chapter 2.4.2. 

13 

 
 
 
 
 
 
 
 
 
 
                                                           
Commission  put  together  a  set  of  draft  evaluation  questions  relating  to  the  five  main 
criteria of effectiveness, relevance, coherence, efficiency and EU added value, as per the 
evaluation design and road map90. The evaluation framework developed by the external 
evaluator  under  the  guidance  of  an  inter-service  steering  group  (ISSG)  built  on  and 
further  elaborated  these  questions,  also  covering  data  collection  and  analysis  methods. 
The evaluation also addresses the criteria of coordination, sustainability and equity as the 
ISSG deemed them relevant for this evaluation91.  

Data collection and analysis 

Evaluation findings are based on the following range of data sources in particular: 

literature review (175 sources92) 

 
  phone  and  face-to-face  interviews  of  key  stakeholders  in  27  Member  States  (191 

interviews) 

  phone and face-to-face interviews of 10 key stakeholders in three enlargement countries 
  8 phone and face-to-face interviews at EU level (Commission staff, MEP, EU-level NGO) 
 
country analysis studies (11 Member States with a total share of 90.2 % of the EU Roma 
population) pulling together quantitative and qualitative country-specific information 
an  open  public  consultation  (OPC),  which  generated  240  responses  and  28  position 
papers 
a targeted NGO survey, which generated 65 responses 

 
  data provided by Member States in the online reporting tool for annual reporting to the 

 

Commission (years covered: 2015 and 2016) 

  ESIF  programming  documents 93 ,  including  through  the  use  of  dedicated    portals 

provided by the Commission (SFC94 2014) 

  2011 FRA Roma Survey and 2016 FRA EU-MIDIS II survey 
  2011 UNDP/ World Bank/ EC regional Roma survey and 2017 UNDP/World Bank/EC 

survey on the Western Balkans 

  discussions at the evaluation workshop on preliminary findings of this evaluation, at an 
event  discussing  the  evaluation  at  the  EU  Roma  Week,  at  the  2017  European Platform 
for  Roma  Inclusion  (focusing  on  the  transition  from  education  and  employment)  and 
other stakeholder and consultation events. 

Method strengths, limitations and mitigation actions 

The  methodological  approach  chosen  for  this  evaluation  (see  Annex  4)  ensured  the 
gathering  and  triangulation  of  a  substantial  amount  of  qualitative  data,  in  particular 
through the stakeholder interviews, country analyses, the OPC and the NGO survey. This 
was  important  given  the  various  limitations  which  made  it  impossible  to  present  a 
quantified  counter-factual  scenario,  i.e.  the  consequences  of  not  having  an  EU 
framework. 

a) General lack of Roma-specific quantitative data  

The  lack  of  solid  Roma  specific  performance  indicators  and  quantitative  data  affected 
this  evaluation  in  several  ways,  in  particular  the  ability  to  assess  the  effectiveness  and 

90 europ.eu/info/law/better-regulation/initiatives/ares-2017-1716166_en. 
91 See Annex 3 for all evaluation questions. 
92 Bibliography is provided in the external evaluation study; in addition, around 200 national sources were included in 

the country analysis studies. 

93 Operational  Programmes;  synthesis  reports  of  annual  implementation  reports  such  as  for  the  ESF  2017: 

https://ec.europa.eu/esf/transnationality/content/synthesis-report-esf-2017-annual-implementation-reports. 

94 System for fund management in the European Union. 

14 

 
 
 
                                                           
efficiency  criteria.  Other  limiting  factors  included  the  relative  unreliability  of  the  data 
from  the online reporting tool (used by Member States to  report to  the Commission on 
NRIS implementation) and reduced availability of data on quantifiable costs and benefits 
(see Annex 4 for a detailed description and the mitigation actions taken). 

b)  Limitations  regarding  attribution  of  outputs,  outcomes  and  impacts  to  the  EU 
framework 

A  number  of  social  and  economic  factors  (economic  developments,  political  priorities 
etc.)  unrelated  to  the  EU  framework,  directly  or  indirectly  impact  efforts  for  Roma 
integration.  In  addition,  given  that  the  policy  areas  covered  by  the  EU  framework  fall 
largely  under  the  competence  of  Member  States,  the  aim  of  the  EU  framework  was  to 
provide  guidance  for the NRISs and their implementation.  It  was expected that the EU 
framework  would  enable  exchange  and  encourage  cooperation  of  relevant  stakeholders 
as  well  as  mobilise  other  EU  level  policy,  legal  and  funding  instruments  for  Roma 
integration. In turn, it was for the NRISs to mobilise and align the use of national legal, 
policy  and  funding  instruments,  and  provide  guidance  for  local-level  action  plans  and 
strategies, which would interact with local-level policies (see Annex 895). Therefore, the 
results  and  impacts  of  the  EU  framework  on  the  situation  of  Roma  can  thus  only  be 
considered  to  be  indirect  and  as  being  influenced  by  the  instruments  and  actors  at 
national and local levels. 

Consequently,  a  prudent  approach  was  chosen  for  this  evaluation,  particularly  with 
regard  to  efficiency  criterion.  The  evaluation  focuses  primarily  on  costs  and  benefits 
attributable to the EU framework that have so far been identified. On the other hand, it 
does  not  attempt  to  assess  the  extent  to  which  EU  and  national  funding  in  the  area  of 
Roma  integration  is  directly  justified  and  proportionate  to  the  long-term  benefits  from 
Roma  integration  in  inclusive,  quality  mainstream  education,  employment,  healthcare 
and  housing 96 .Such  an  assessment  would  have  needed  more  solid  Roma-specific 
indicators as well as a clear relationship between cause and effect of interventions. 

Validity and reliability of the findings 

Despite the lack of a quantified counter-factual scenario and the limitations set out above, 
the  evaluation  findings  are  valid  and  reliable.  Mitigation  measures  were  taken  and  the 
evaluation  was  underpinned  by  a  large  body  of  qualitative  evidence  which  provides  a 
solid  basis  for  drawing  conclusions.  In  particular  the  workshop  disseminating  and 
discussing  preliminary  findings  helped  to  further  verify  the  correctness  of  the  findings 
and fine-tune the findings. 

95 This mapping was included in the evaluation roadmap. 
96 This was one of the original evaluation questions; the external study did not take this approach. 

15 

 
 
 
 
 
 
 
                                                           
5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS 

5.1 EFFECTIVENESS 

The analysis of effectiveness considers the extent to which the EU framework has made 
progress  towards  its  objectives  at  the  midterm  stage,  using  appropriate  points  of 
comparison  and  assessing  the  role  of  the  EU  framework  in  delivering  the  observed 
changes. The  effectiveness  of  the  EU  framework  so  far  has  been  assessed  as  being 
largely  limited  regarding  progress  towards  the  Roma  integration  goals.  The 
assessment  is  more  positive  with  respect  to  the  setting  up  and  strengthening  of 
instruments  and  structures  and  increased  cooperation  among  and  capacity  of 
stakeholders. 

The assessment has been informed by 14 evaluation questions that looked into the extent 
to which: 

  progress  has  been  made  so  far  on  the  expected  outcomes,  in  particular  towards 
reaching  the  Roma  integration  goals  in  the  areas  of  access  to  education, 
employment, housing and health, as well as the reasons for this progress; 

  changes  in  the  situation  of  Roma  in  the  four  areas  can  be  attributed  to  the  EU 

 

 
 

framework and NRIS; 
the EU framework contributed to the setting up of instruments and structures for 
Roma  integration  at  EU  and  national  level,  increasing  cooperation  among 
stakeholders and capacity of national and local actors; 
the EU framework has indeed served as a framework for NRIS;  
the  EU  framework  and  NRIS  contributed  to  an  increase  in  the  number  of  and 
funding  for  Roma  integration  measures  as  well  as  an  increase  in  the  number  of 
Roma beneficiaries. 

5.1.1. Progress towards the Roma integration goals, attribution and outlook 

Attributing progress towards the attainment of the Roma integration goals is difficult in 
light of: (i) numerous external factors such as the economic and financial crisis (and the 
resulting economic and labour market challenges in various countries), the 2015 refugee 
crisis (which in some Member States led to lowering the priority of Roma integration in 
the national policy agenda)97 or the rise of nationalist parties and spread of discriminatory 
rhetoric; and (ii) the distribution of competences between the EU and the Member States. 
The  policy  areas  covered  by  the  EU  framework  fall  largely  under  the  competences  of 
for  NRIS 
Member  States.  The  EU 
implementation,  enabled  exchange  and  cooperation  between  relevant  stakeholders  and 
mobilised  other  policy,  legal  and  funding  instruments  for  Roma  integration.  These 
aspects are elaborated in detail below and in the coherence section. In turn, it was for the 
NRIS to mobilise and align the use of national legal, policy and funding instruments, and 
provide  guidance  for  local-level  action  plans  and  strategies,  which  would  interact  with 
local-level  policies.  The  effects  of  the  EU  framework  on  the  situation  of  Roma  can 
therefore only be indirect, and mitigated by the instruments and actors at play at national 
and local levels. 

therefore  provided  guidance 

framework 

97 In the 2016 Council conclusions on accelerating the process of Roma integration Member States recognise the new 

challenges posed by the refugee crisis; see recital (11);. 
16 

 
 
 
 
                                                           
The area of education observed the most pronounced progress98. Participation of Roma 
in early childhood education and care (ECEC) has improved since 2011 in most  countries99, 
from  47 %  in  2011  to  53 %  in  2016.  School  attendance  at  compulsory  schooling  age  has 
improved,  from  86 %  to  90 %  and  more  specifically  in  four  out  of  the  nine  countries 
surveyed  (CZ, EL, HR,  PT);  early leaving from  education and training has decreased from 
87 % to 68 % and more specifically in seven out of the nine countries (BG, CZ, ES, HU, PT, 
RO, SK). However, substantial gaps remain both between Roma and the general population 
and between countries (ranging from a 28 % participation rate of Roma children in Greece to 
94 % in Spain). Attendance of education when in compulsory schooling age is not far from 
the  general  population  (90 %  Roma  compared  to  96 %  non-Roma).  The  share  of  Roma 
students attending classes where ‘all classmates are Roma’ has increased (from 10 % to 15 %) 
and one to two thirds of Roma children attend schools where most or all children are Roma. 
For the Western Balkan countries, the situation shows improvement between 2011 and 2017 
especially as regards enrolment in compulsory education (ages 7-15). However,, pre-primary 
school  enrolment  rates  slightly  decreased  in  3  out  of  6  countries  (Albania,  Bosnia  and 
Herzegovina,  the  former  Yugoslav  Republic  of  Macedonia).  Substantial  improvement  was 
registered in Montenegro only and appears to be directly linked with the implementation of a 
UNICEF  project.  Importantly,  in  most  of  the  cases,  there  was  no  reduction  of  the  gap  in 
education between e Roma and non-Roma100. 

FRA survey data shows a lack of observable improvement in access to employment 
across the analysed period. The proportion of Roma who are in paid work as their main 
activity  —  including  self-employment  and  occasional  work  —  remains  similar  (around 
25 % of Roma aged 16+). The share of young Roma aged 16 to 24 whose current main 
activity  is  cited  as  being  neither  in  employment,  education  or  training  (NEET)  has 
increased from 56 % to 63 %. The gap with the general population remains wide. Until 
now, the improvements observed in education have barely resulted in higher employment 
rates, and the transition from education to employment remains a challenge. In respect to 
the  Western  Balkan  countries,  there  has  been  has  been  a  general  worsening  of  the 
situation  since  2011;  only  the  former  Yugoslav  Republic  of  Macedonia  has  shown  an 
improvement related to reduction of unemployment and informal employment of Roma. 
Most  likely  reasons  for  this  lack  of  improvement  in  employment  rates  include:  (i)  the 
consequences of the economic crisis; (ii) existing prejudices and discrimination; (iii) the 
absence  of  adapted  support  measures;  (iv)  insufficient  qualifications 101 .  Country 
analysis studies and interviews102 mention that  low educational status, the lack of basic 
skills  and  discrimination  are  not  tackled  effectively.  Also,  mainstream  measures  for 
active  employment  typically  lack  explicit  references  to  Roma  and  do  not  provide 
individualised  support.  Therefore,  coupled  with  adverse  economic  conditions,  the 
employment  measures  have  failed  to  effectively  reduce  the  employment  gap  between 
Roma  and  non-Roma.  However,  given  the  duration  of  the  educational  cycle  and 

98 These statements are based on FRA survey data for 9 EU MS, for which data were collected in a comparable way 
for  both  2011  and  2016.  These  countries  account  for  over  80 %  of  the  overall  estimated  Roma  and  travellers 
population in the EU (as per the Council of Europe). 

99 6 out of the 9 countries surveyed (BG, EL, ES, HR, HU, SK). 
100 UNDP, Regional Roma Survey 2017: Country fact sheets 
101 Gatti, Roberta et al. 2016. Being Fair, Faring Better — Promoting Equality of Opportunity for Marginalised Roma. 
(World Bank). Washington D.C.; O’Higgins, Niall. 2012. Roma and Non-Roma in the Labour Market in Central 
and South Eastern Europe. edited by A. Ivanov and J. Kling. (ILO) Bratislava; ILO. 2016. Promoting Decent Work 
Opportunities  for  Roma  Youth 
in  Central  and  Eastern  Europe:  An  ILO  Resource  Guide.  Geneva. 
http://www.ilo.org/budapest/what-we-do/publications/WCMS_535448/lang--en/index.htm. 

102 RO-07, CS-RO, CS-AT, EL-01, BE-05, El-06, CZ-12, El-16, ES-09. 

17 

 
 
 
                                                           
considering its long-term qualitative effects, improvement in the employment situation is 
likely in the medium-to long-term.    

Health self-perception  has improved but  medical health  insurance coverage is  still 
limited103. The reported average self-perceived health status of Roma is equal to that of 
the general population. As compared with previous years, more Roma consider that their 
health is good or very good (from 55 % to 68 %). Nevertheless, healthcare coverage has 
not  increased  (around  75 %  of  Roma  aged  16+),  with  differences  between  countries 
ranging  significantly  from  45 %  in  Bulgaria  to  98 %  in  Spain104.  Access  to  health 
insurance and health services, including health promotion and disease prevention, poses 
special  problems  for  many  Roma  and  results  in  difficulties  particularly  in  the  area  of 
sexual  and  reproductive  health  services  and  vaccination 105 .  In  the  Western  Balkan 
countries, general improvements in health insurance coverage and reduced inequality106 
were observed. However, Montenegro had poorer outcomes for both of these indicators 
and  health  insurance  coverage  reduced  in  Albania.  Over  70 %  of  OPC  respondents 
claimed  that  their  country  had  not  made  progress  on  the  implementation  of  Roma 
integration measures on healthcare. Most likely reasons for this limited progress over the 
short time frame are a range of barriers such as language and literacy, a lack of trust, a 
lack of identification documents and physical distance to healthcare facilities107. 

There has been very little change in Roma housing conditions. The housing situation 
for many Roma families continues to be difficult. In 2016, the average number of rooms 
per person was 0.7 (0.6 in 2011), 30 % of households did not have tap water inside the 
dwelling  (29 %  in  2011)  and  the  share  of  people  living  in  households  having  neither  a 
toilet,  shower,  nor  bathroom  inside  the  dwelling  was  reportedly  38 %  (36 %  in  2011). 
There is still a gap between Roma and the general population in these areas108. However, 
most households reportedly have an electricity supply (92 % to 96 %).  

Most  likely  reasons  identified  for  the  lack  of  progress  are  discrimination  and  lack  of 
funding. The data does not indicate any decrease in discrimination in access to housing. 
On  the  contrary,  in  some  countries  a  notable  increase  is  observed109.  Many  housing 
conditions  are  substandard  and  a  gap  remains  in  access  to  public  utilities.  Many 
respondents to the NGO survey (31 out of 65110) believe that since 2011, the NRIS of 
their Member State did not contribute to ensuring national funding for Roma integration 
in  the  area  of  housing.  Forced  evictions  of  Roma  from  their  homes  continue  to  be 

103 FRA (2018) A persisting concern: anti-Gypsyism as a barrier to Roma inclusion 
104 SWD (2017) 458,  FRA  Roma integration scoreboard and FRA (2018)  A persisting concern: anti-Gypsyism as a 

barrier to Roma inclusion 

105 Sándor, János et al. 2017. ‘The Decade of Roma Inclusion: Did It Make a Difference to Health and Use of Health 
Care Services?’ International Journal of Public Health 62(7); European Public Health Alliance 2017:  Health and 
Early  Childhood  Development  in  Roma  Communities  —  A  document  analysis  of  European  Union  and  national 
policy  commitments  in  the  national  Roma  integration  strategies;  European  Public  Health  Alliance  2016.  ‘The 
European Semester and Roma Health.’; European Commission, Roma Health Report — Health status of the Roma 
population.  Data collection in the Member States of the European Union, 2014,; Fésüs, Gabriella, Piroska Östlin, 
Martin  McKee,  and  Róza  Ádány.  2012.  ‘Policies  to  Improve  the  Health  and  Well-Being  of  Roma  People:  The 
European Experience.’ Health Policy 105(1):25-32. 

106 UNDP Survey. Overview at a Glance. 
107 Matrix, 2014. Roma Health Report: Health status of the Roma population: Data collection in the Member States of 

the European Union. DG SANCO, http://eige.europa.eu/resources/2014_roma_health_report_en.pdf 

108 FRA (2018) A persisting concern: antigypsyism as a barrier to Roma inclusion 
109 SWD (2017) 458, FRA Roma integration scoreboard and FRA (2018) A persisting concern: antigypsyism as a 

barrier to Roma inclusion 

110 S-NGO. Of the remaining respondents, 19 indicated ‘Contribution to some extent’, 6 a ‘Significant contribution’, 1 

a ‘Very significant contribution’, and 8 stated ‘Not applicable’. 

18 

 
 
 
                                                           
reported  across  many  Member  States  and  enlargement  countries 111 .  Regarding  the 
Western Balkan enlargement region, the housing situation of Roma varies considerably 
between  countries,  with  improvements  in  Serbia  and  Bosnia  and  Herzegovina,  and  a 
deterioration in Albania and Montenegro112. 

The  following  table  presents  a  shortened  and  simplified  'nutshell  assessment'  of 
progress towards the specific objectives of the EU framework: 

Table 4: Progress towards the Roma integration goals 
Objective 
Access  to  education:  Ensure  that  all 
Roma 
least 
children 
primary  school 
Access 
the 
employment  gap  between  Roma  and 
the rest of the population 

to  employment:  Cut 

complete  at 

attendance 
in 

of 
compulsory 

Progress status 
Some 
progress: 
education  when 
schooling age has increased113  
No  progress:  the  share  of    Roma  in 
paid  work 114  has  not  increased  in 
comparison to the employment rates 
of the general population. 

Access to healthcare: Reduce the gap in 
health  status  between  the  Roma  and 
the rest of the population 

Access  to  housing:  Close  the  gap 
between the share of Roma with access 
to housing and to public utilities (water, 
electricity,  gas)  and  that  of  the  rest  of 
the population 

progress: 

self-reported 
Limited 
health  status  ‘very  good’  or  ‘good’ 
has reached the same level as that of 
the general population However, this 
is  self-reported  and  no  progress  in 
access to medical insurance coverage 
could be observed115. 
No  progress:  The  housing  situation 
for  Roma  remains  difficult.  The  gap 
the  general 
between  Roma  and 
population  remains  and 
is  even 
widening. 

2011 
86 % 

2016 
90 % 

gap  of 
44 pps 

gap  of 
46 pps 

Gap  of 
13 
pps116 

Gap  of 
0 
pps117 

gap  of 
29 
pps118 

gap  of 
33 pps 

gap  of 
0.66 
rooms 
per 
person 

gap  of 
0.79 
rooms 
per 
person 

Source:  Roma:  Based  data  quoted  in  FRA  (2018)  —  A  persisting  concern:  anti-Gypsyism  as  a  barrier  to  Roma 
inclusion; General Population (Eurostat); Data: Percentage point difference between Roma and General Population 
indicators for nine Member States covered by FRA 2011 Roma / 2016 EU-MIDIS II surveys. 

Modest  progress  has  been  observed  for  the  general  objective  of  fighting  poverty. 
According  to  the  2011  FRA  Roma  survey,  on  average  86 %  of  the  Roma  population 
surveyed were at risk of poverty. The situation had improved somewhat by 2016, when 
80 % of Roma surveyed were found to be at risk119. 

Table 5: FRA Roma survey results on poverty 

111 UN-OHCHR 2017: Lessons Learned: Views in the Context of Mid-Term Review of Implementation of the EU 

Framework for National Roma Integration Strategies 2012-2020, Brussels;. 

112 Regional overview at First Glance of the UNDP/WB/EC survey. WB3; WB6; WB8. 
113 Shown value (participation rate) is for share of children attending education when in compulsory schooling age, 5-

17 (depending on the country). No comparable data for primary school completion are available.. 

114 Used as a proxy for the employment rates of the general population. 
115 The gap between Roma and non-Roma has even increased from 18 % in 2011 to 22 % in 2016. 
116 Self-reported. 
117 General population: Own calculations (weighted average) based on Eurostat,EU-SILC 2016, General population; 
EU-SILC 2011 ([hlth_silc_01] (download 06/07/2018);  **OECD Health Database; Health at Glance: Europe 
reports 2010, 2016; (download 06/07/2018); also FRA (2018) A persisting concern: anti-Gypsyism as a barrier to 
Roma inclusion; p. 13 states that self-reported health status is on average similar to the general population. 

118 Gap in people living without toilet, shower or bathroom inside the dwelling. 
119 At-risk-of-poverty (national threshold (€): 60 % of median equivalent income after social transfers), FRA (2018) A 

persisting concern: antigypsyism as a barrier to Roma inclusion 

19 

 
 
 
 
 
 
 
 
 
                                                           
2011 

Roma  

General 
populatio
n  

2016 

Roma  

General 
population 

Gap 

2011 

2016 

People at risk of poverty  

86 % 

19 % 

80 % 

20 % 

67pps 

60pps 

Source: Roma: for 2011, 2016 — EU Roma and EU-MIDIS II survey data quoted in FRA (2018); General population: 

Own calculations (weighted average) based on Eurostat (t2020_52) EU-SILC 2014,; Eurostat, EU-SILC 
2011 

Discrimination  continues  to  be  of  high  concern  in  European  societies,  despite 
evidence of some improvement in a few areas. The enforcement of anti-discrimination 
law  regarding  Roma  in  the  period  2011-2016  shows  a  scattered,  mixed  picture  among 
Member  States.  However,  in  most  countries  the  situation  has  not  changed  in  this  five-
year period120. According to surveys the overall share of Roma who felt discriminated 
against  when  in  contact  with  schools  (as  a  parent  or  student)  has  not  decreased 
significantly  (17 %  to  14 %).  There  has,  however,  been  a  significant  decrease  when 
accessing health services (20 % to 9 % in the case of men and 19 % to 7 % in the case of 
women)121.  In  the  past  5  years,  the  share  of  Roma  who  felt  they  were  discriminated 
against  when  looking  for  housing  has  dropped  slightly  (from  45 %  to  41 %)  but  the 
extent  of  discrimination  felt  by  Roma  ‘when  looking  for  a  job’  and  ‘at  work’  remains 
high  (40 %  and  17 %  respectively),  even  though  for  the  former  there  has  been  a 
considerable  drop  of  10 %.  In  2016,  almost  every  third  Roma  (30 %)  had  experienced 
some  form  of  harassment  in  the  past  12  months 122 that  they  felt  was  due  to  their 
ethnicity123.  For  76 %  of  the  respondents,  harassment  due  to  ethnicity  is  a  recurring 
experience. Instances of antigypsyism in the form of hate crime against Roma diminishes 
Roma  people’s  trust  in  their  public  institutions,  in  particular  in  law  enforcement  and 
justice,  thus  undermining  social  inclusion  efforts124.  Antigypsyism  is  reported  to  have 
increased,  fuelled  by  the  stigmatisation  of  the  Roma  community  by  some  mainstream 
political parties125. The majority of the OPC respondents identify rising discrimination 
and antigypsyism as main challenges to Roma inclusion at both EU and national level126.  

The  OPC  respondents  confirmed  that  the  most  significant  progress  was  achieved  in 
education. In health, employment and housing, most respondents see no change. Moreover, 
while  on  health  more  people  see  improvement  than  deterioration,  opinions  are  divided  on 
employment,  whereas  on  housing  those  who  consider  the  situation  to  be  worsening 

120 European network of legal experts in gender equality and non-discrimination, Roma and the enforcement of anti-

discrimination law (2017) 

121 Second European Union minorities and discrimination survey (EU-MIDIS II). 
122 Offensive  or  threatening  comments  in  person;  threats  of  violence  in  person;  offensive  gestures;  offensive  or 

threatening e-mails or text messages; offensive comments made online. 
123 FRA(2018), A persisting concern: antigypsyism as a barrier to Roma inclusion 
124 idem. 
125 Antigypsyism as an underlying cause for discrimination has been consistently mentioned in the consultations with 
stakeholders for this evaluation, such as in the 15 March workshop. It has been highlighted by submissions to the 
open  consultation  as  well  as  in  specific  contributions,  see:  Carrera,  Sergio,  Iulius  Rostas,  and  Lina  Vosyliūtė. 
2017. Combating Institutional Antigypsyism Responses and Promising Practices in the EU and Selected Member 
States.  Brussels  Alliance  against  Antigypsyism.  2017.  Antigypsyism  —  a  Reference  Paper.  Berlin  /  Budapest. 
http://antigypsyism.eu/;. 

126 When  asked  whether  rising  discrimination  and  antigypsyism  was  one  of  the  main  challenges  (in  the  2011-2016 
period)  in  respect  of  Roma  inclusion  at  the  European  level  (Q11.1),  86.3 %  of  the  respondents  rather  agreed, 
10.3 %  rather  disagreed,  and  3.4 %  did  not  have  an  opinion.  When  asked  about  the  perceived  increase  in 
antigypsyism since 2011 in the EU-15 and EU-13 (Q16.2), 72.6 % of EU-13 and 62.2 % of EU-15 respondents 
agreed that antigypsyism was more severe today than it was in 2011, while 19.8 % of EU-13 and 23.3 % of EU-
15 respondents disagreed. 

20 

 
 
 
 
 
 
                                                           
outnumber  those  that  see  improvement.  On  discrimination,  the  biggest  response  is  ‘no 
change’, and many more people refer to a worsening situation than to improvements. 

Figure 1: OPC— Perceived changes in the situation of Roma since 2011

Overall,  6  years  is  too  short  a  period  to  make  significant  improvements.  Structural 
changes require time and the real impact of progress (including a change in employment 
trends,  resulting  from  positive  steps  in  education)  may  only  be  seen  in  a  generation. 
However, the evaluation indicates that further progress can still be made in the remaining 
period  provided  that  efforts  to  combat  discriminatory  attitudes  and  tackle  antigypsyism 
increase. Favourable economic developments, falling unemployment rates and increasing 
labour  shortages  in  the  EU  could  provide  some  conditions  for  progress  in  terms  of 
improving the employment situation of Roma. 

5.1.2 Alignment between the EU framework and NRIS 

All  Member  States127 and  enlargement  countries  have  a  NRIS  as  well  as  a  National 
Roma  Contact  Point  with 
the  development  and 
implementation of NRIS in place. A total of 71 % of OPC respondents stated that one of 
the  main  achievements  of  the  EU  framework  had  been  that  Member  States  developed 
NRISs.  

to  coordinate 

the  authority 

Member  States  make  a  differentiated  use  of  these  instruments,  depending  on  country 
specificities. In total, 15 Member States128 have adopted strategies, while the remaining 
12  Member  States 129  have  adopted  integrated  sets  of  policy  measures  within  their 
broader social inclusion policies130.  The human and financial resources, as well as the 
mandate  and  tasks  of  NRCPs  vary.  While  with  few  exceptions  they  are  in  charge  of 

127 except MT which does not have a Roma community 
128, BG, CZ, IE, EL, ES, FR, HR, IT, HU, PL, PT, RO, SI, SK, FI, SE. 
129 BE, DK, DE, EE, FR, CY, LV, LT, LU, NL, AT, UK. 
130 In line with the 2011 Council conclusions. Both approaches can stimulate the implementation of inclusion policies. 
Therefore,  the  remaining  analysis  does  not  differentiate  between  countries  with  strategies  and  those  with 
integrated sets of policy measures. 

21 

 
 
 
 
 
 
 
                                                           
coordination  and  monitoring  of  NRIS  and  almost  all  of  them  also  facilitate  civil 
involvement, fewer are involved in development of other policies and the planning of the 
use  of  EU  funds,  and  less  than  half  have  regular  contacts  with  equality  bodies.  The 
placement  of  NRCPs  within  the  national  government  structures  also  vary:  Some  are 
placed in institutions with a wider social inclusion agenda with relatively strong mandate 
and  influence  on  mainstream  policies,  while  others  are  in  well-established  institutions 
responsible for a specific narrower policy agenda (such as housing). Some NRCPs rather 
have  a  weaker  technical  or  policy  communication  role  with  other  institutions  in  place 
coordinating social inclusion. 131 

The EU framework called on Member States  to set achievable national targets for Roma 
integration — addressing the four EU Roma integration goals as a minimum — to bridge 
the  gap  with  the  general  population.  The  below  analysis  shows  that  in  particular  those 
countries with the highest share of Roma fully addressed the Roma integration goals in 
their NRIS (Bulgaria, Spain, Hungary, Romania, Slovakia), complemented by Finland. 

Table 6: Inclusion of national goals explicitly addressing the Roma integration goals 

Roma integration goal  No 

Yes 

that 

Ensure 
all 
children  complete  at 
least 
primary 
education 

CZ,  DE,  EE,FR  IE,  IT,  CY  LV, 
LT, LU, NL, PL, PT, SI, SE, UK 

Cut  the  employment 
gap  between  Roma 
and  the  rest  of  the 
population 

DK,  DE,  EE,  IE,  EL,  FR,  IT, 
CY,  LV,  LT,  LU,  NL,  AT,  PL, 
PT, SI, SE, UK 

in 
Reduce  the  gap 
health  status  between 
the Roma and the rest 
of the population 

BE,  CZ,  DK,  DE,  EE,  IE,  EL, 
FR,  HR,  CY,  LV,  LT,  LU,  NL, 
AT, PL, PT, SI, SE, UK 

Close the gap between 
the  share  of  Roma 
to 
with 
housing  and  that  of 
the 
the 
rest 
population 

access 

of 

BE,  DK,  DE,  EE,  EL,  FR,  HR, 
CY,  LV,  LT,  LU,  NL,  AT,  PL, 
PT, SI, UK 

BE,  BG,  DK,  EL,  ES, 
FR,  HR,  HU,  RO,  SK, 
FI 

Yugoslav 
former 
Republic 
of 
Macedonia,  Kosovo, 
Serbia, Turkey 

BE,  BG,  ES,  HR,  HU, 
RO, SK, FI 

Yugoslav 
former 
Republic 
of 
Macedonia,  Kosovo, 
Serbia, Turkey  

BG,  ES,  IT,  HU,  RO, 
SK, FI 

Yugoslav 
former 
Republic 
of 
Macedonia,  Kosovo, 
Serbia, Turkey 

BG,  CZ,  ES,  HU,  RO, 
SK, FI, SE 

Yugoslav 
former 
Republic 
of 
Macedonia,  Kosovo, 
Serbia, Turkey 

Partial 

AT 

Albania, Bosnia 
and 
Herzegovina, 
Montenegro 

CZ 

Albania, Bosnia 
and 
Herzegovina, 
Montenegro 

Albania, Bosnia 
and 
Herzegovina, 
Montenegro 

IE, IT 

Albania, Bosnia 
and 
Herzegovina, 
Montenegro 

Source: ICF Research based on NRIS and NRCP interviews; complemented with a contribution from DG NEAR 

The  cases  of  incomplete  alignment  with  the  Roma  integration  goals  above  can  be 
explained primarily by the fact that  the Council used the following wording to endorse 
the  Roma  integration  goals:  ‘four  goals  of  ensuring  equal  access  in  education, 

131 Roma integration measures reporting tool, 2017. Roma Civil Monitor 2018. 

22 

 
 
 
 
 
 
 
 
                                                           
employment,  healthcare  and  housing132’.  The  scope  was  therefore  broader,  providing 
more flexibility to Member States in shaping their national goals, taking into account the 
size of their Roma population, different starting points and country-specific features, for 
example in the composition of the Roma population. 

The same reasoning applies to the measures taken by the Member States in the four key 
areas. As shown in the comparison between the 2013 Council Recommendation and the 
EU framework set out in Annex 6 to this SWD, the Recommendation has widened the set 
of possible measures per area but strengthened their voluntary nature by stating that the 
Roma integration goals could be attained by means of measures such as those listed. 

5.1.3.  Setting  up  other  instruments  and  structures  for  Roma  integration  and 
increasing capacity of  those involved at national and local level 

At  EU  level,  this  evaluation  finds  that  that  the  EU  framework  was  effective  in 
introducing  a number of  instruments  (in particular  annual  reporting by  Member States 
and  the  Commission,  recent  reporting  by  civil  society)    and  EU-level  governance 
structures  (for  example  the  NRCP  network  —  see  coordination  section).  Mirroring  the 
Member  States’  practices,  similar  instruments  have  been  introduced  in  the  enlargement 
region.  With  regard  to  instruments,  the  2013  Council  Recommendation  introduced 
comprehensive  reporting  by  Member  States  from  2016.  The  reporting  has  been  put  in 
place but focuses on policy measures taken by Member States, rather than on outcomes. 
Independent  monitoring,  in  particular  through  FRA  surveys,  is  still  hampered  by  the 
relatively low coverage of Member States (although the nine Member States covered in 
both 2011 and 2016 surveys make up  more than 80 % of the EU Roma population). In 
2017  the  Commission  piloted  a  project  to  monitor  the  implementation  of  the  NRIS  by 
civil society in all Member States but Malta133. As for the structures, even if some of 
them  predated  the  EU  framework  (the  Roma  task  force  was  created  in  2010,  while  the 
European  Platform  for  Roma  Inclusion  met  for  the  first  time  in  2009134),  they  were 
strengthened and made permanent following the adoption of the EU framework. Notably, 
66.7 %  of  OPC  respondents  recognised  that  the  development  of  these  structures  at  EU 
level was one of the key achievements of the EU framework. 

With regard to the capacity of national actors to implement Roma integration measures, 
the  evaluation  shows  that  today  there  is  better  awareness  and  more  knowledge  about 
Roma  integration 135 .  During  the  interviews  with  NRCPs,    nine  Member  States  in 
total136 reported  an  increase  in  the  number  of  government  officials  working  on  Roma 
integration  in  response  to  the  EU  framework.  Closer  engagement  with  Roma 
stakeholders  either  at  local  or  central  level,  including  via  formal  Roma  platform 
structures at national level137, has helped increase understanding of Roma issues among 
government officials138. However, some NGO representatives voiced the criticism that 
this  increased  understanding  had  not  sufficiently  been  translated  into  action139.  Roma 

132 Both in the 2011 Council Conclusions and the 2013 Council Recommendation. 
133 European Commission,  Description of the EU pilot project for civil society  capacity building and monitoring of 

the implementation of national Roma integration strategies, 2017,. 

134 European Commission ‘EU Platform for Roma inclusion: Frequently asked questions’ [website], 2011,. 
135 AT-11, EUI-1, EUI-6, HR-3, NL-4, NL-5. 
136 AT; BE; DE; ES; FI; HR; LT; PL; SE. 
137 National Roma platforms exist in 18 Member States. 
138 AT-6, PT-1, UK-5. 
139 FI-10, FI-13, SK-14. 

23 

 
 
 
                                                           
participation has increased at national level and most NRISs have systems of consultation 
with  Roma  civil  society  in  place.  However,  there  remains  a  need  for  more  effective 
Roma engagement at all levels, including from the gender perspective140. The quality of 
civil  society  participation  is  still  considered  inadequate  in  terms  of  stakeholder 
involvement in the monitoring progress. Furthermore, cooperation between Roma NGOs 
and broader social NGOs rarely occurs in practice, and Roma issues are barely present in 
the mainstream social inclusion agenda141. 

5.1.4.  Increase  in  number  of  and  funding  for  Roma  integration  measures  and  in 
Roma beneficiaries 

Finally,  the  evaluation  finds  that  there  is  insufficient  quantitative  data  available  at  EU 
and national levels to assess whether the EU framework has contributed to an increase in 
the  number  of  Roma  integration  measures,  the  funding  attributed  and  the  number  of 
Roma beneficiaries. 

As described in the EU added value and coherence sections, the EU framework and EU 
structural funds have been successfully aligned for the 2014-2020 programming period. 
In  total,  12  Member  States  (AT,  BE,  BG,  CZ,  ES,  FR,  EL,  HU,  IT,  PL,  RO  and  SK) 
selected  investment  priority  (IP)  9.2  on  socio-economic  integration  of  marginalised 
communities  such  as  the  Roma,  allocating    it  a  total  of  EUR 1.5  billion,  or  3.6 %  of 
European Social Fund (ESF) funding. However: 

  the  Investment  Priority  covers  marginalised  communities  in  general  (and  not 
exclusively  Roma)  thus  making  it  more  difficult  to  quantify  concrete  amounts 
spent on Roma integration; 

  the  ESF  output  indicator  for  participants  ‘migrants,  participants  with  a  foreign 
background, minorities (including marginalised communities such as the Roma)’ 
is  not  specific  enough  to  provide  a  reliable  assessment  of  the  number  of  Roma 
beneficiaries142; and particularly 

  no comparison can be made with the 2007-2013 programming period, which took 

a mainstream approach with no such IP and indicator. 

The evaluation could nevertheless provide some qualitative insights. For example: 

  regarding  EU  funding  in  10  Member  States,  the  consensus  among  interviewed 

stakeholders was that it  has increased143, in particular from the ESF; 

  in  the  enlargement  countries,  all  respondents  expressed  a  similar  view  on  the 

impact of IPA funding144; 

140 Roma Civil Monitor 2018: Synthesis Report on the Implementation of the EU Framework for National Roma 

Integration Strategies in Bulgaria, Czech Republic, Hungary, Romania and Slovakia. Brussels / Budapest. 
cps.ceu.edu/roma-civil-monitor. 

141 An exception seems to be the work of the European Public Health Alliance (EPHA), which integrates the Roma 

health issues within a wider agenda of public health and universal health care. See EPHA 2017: Health and Early 
Childhood Development in Roma Communities — A document analysis of European Union and national policy 
commitments in the national Roma integration strategies, Brussels. 

142 In countries without significant migrant inflows and larger Roma populations one could in principle assume that a 
large majority of the participants are actually Roma, which would in such cases be confirmed through the content 
of the specific investment priority in an operational programme. However, the low level of implementation of the 
operational programmes so far is an obstacle to this approach. 

143 AT, BG, EE, EL, ES, HR, IT, PT, RO, SK. 

24 

 
 
 
 
 
 
 
 
                                                           
  in seven Member States145, there was consensus that there had been no increase, 
while in another eight Member States146 the government officials interviewed did 
not know if and how funding had changed; 

  the majority of OPC respondents (53.5 %) also agreed that the EU framework had 
contributed  to  an  (increased)  allocation  of  EU  funding  under  ESIF  (2014-2020) 
for Roma integration; 

  regarding  national  funding,  the  consensus  from  the  in-depth  interviews  in  13 

Member States was that there had been no increase147; 

  OPC respondents held similar views: 52 % stated that the EU framework had not 
contributed to (increased) allocation of national funding for Roma integration; 
  in  the  enlargement  region,  respondents  argued  that  prior  to  the  financial  crisis, 
budgets  for  Roma  integration  were  generally  on  the  rise,  except  in  Turkey  and 
Kosovo; and 

  the  crisis  provoked  a  drop  in  the  budgets  in  most  countries  in  the  region  that 

neither the Roma Decade nor the EU framework reversed148. 

144 In the enlargement region, all countries have used their national IPA envelops to finance Roma integration 

projects. They used different approaches: Roma integration-specific projects, projects related to vulnerable parts 
of population benefiting also Roma, mainstream projects including specific Roma integration targets. 
Furthermore, multi-country IPA projects have been used to deal with questions relevant to all the enlargement 
region: set up a Roma integration support team in the Regional Cooperation Council (RCC), support Roma civil 
society and actions in the field of education, boost Roma integration at local level, organise the Roma survey and 
the returnee study (information provided by DG NEAR). 

145 CZ, FI, IE, LT, LU, LV, PL. 
146 BE, CY, DE, DK, NL, SE, SI, UK. 
147 AT, CY, CZ, EE, EL, ES, IT, LU, NL, PL, RO, SI, SK, and enlargement countries. 
148 WB-3, WB-8. 

25 

 
 
 
 
                                                                                                                                                                            
5.2. COORDINATION 

The analysis of coordination considers to  what  extent actions carried out  under the EU 
framework have been coordinated to maximise effects. It looks at coordination at EU and 
national  level  and  at  cooperation  with  stakeholders.  Coordination  is  closely  linked  to 
effectiveness  but  treated  separately  as  a  key  precondition  for  delivering  the  EU 
framework. Coordination of the EU framework has been assessed as being positive 
overall at EU level and limited at national level. 

The  assessment  has  been  informed  by  five  evaluation  questions  that  looked  into  the 
extent to which: 

  coordination  mechanisms  at  EU  and  national  levels  have  been  established  in 

response to the EU framework and whether they are effective; and to which 

  stakeholders  —  in  particular  civil  society  —  both  at  EU  and  national  level,  are 
involved  in  implementing,  planning  and  monitoring  the  EU  framework  and  the 
NRIS. 

5.2.1. Coordination mechanisms 

At  the  EU  level,  several  coordination  mechanisms  supporting  the  EU  framework  have 
been set up at EU level. 

Box 1: Coordination mechanisms at EU level and in the enlargement region 

European  Commission  Roma  Team:  the  specific  team  within  the  ‘Non-discrimination 
and  Roma  coordination unit’  (in  addition  to  the non-discrimination  and  the  legal  teams), 
coordinates the implementation of the EU Roma framework. 

In  addition,  since  2014,  a  Special  Adviser  is  in  place  to  coordinate  Roma  integration 
policies  in  the  enlargement  and  neighbourhood  regions.  In  the  enlargement  region,  the 
Special  Adviser  is  supported  by  a  Roma  network  composed  of  representatives  from  the 
Commission geographic teams and the Roma contact point from each EU delegation. The 
EU Anti-trafficking Coordinator has also been working closely on the risk of trafficking in 
Roma,  in  particular  of  women  and  children.  The  coordinator  is  responsible  for:  (i) 
improving  coordination  between  EU  institutions,  EU  agencies,  Member  States  and  those 
involved at international level; and (ii) developing existing and new EU policies to address 
trafficking in human beings. 

European  Platform  for  Roma  Inclusion:  the  platform  brings  together  national 
governments, the EU, international organisations, representatives of civil society and local 
authorities  on  an  annual  basis.  It  aims  to  encourage  cooperation  and  the  exchange  of 
experiences on successful Roma inclusion149. 

Network  of  NRCPs:  created  in  2012  as  a  mechanism  of  coordination  among  Member 
States  and  between  Member  States  and  the  Commission,  this  network  meets  twice  per 
year. Its key role is to facilitate coordination between the Commission and the NRCPs. In 
2016, the network was extended to the enlargement region. 

Task  force  of  the  ‘Roma  Integration  2020’  initiative  in  the  enlargement  region:  the 

149 https://ec.europa.eu/info/policies/justice-and-fundamental-rights/combatting-discrimination/roma-and-

eu/european-platform-roma-inclusion_en 

26 

 
 
 
                                                           
task  force  organises  annual  task  force  meetings  and  regional  thematic  workshops 
involving, among others, NRCPs and Roma NGOs from enlargement countries150. 

European  network  on  social  inclusion  and  Roma  under  the  structural  funds  (EU 
Roma):  This  is  a  learning  network  which  includes  NRCPs  and  managing  authorities, 
focusing on the use of ESI funds for the inclusion of Roma151. 

Roma task force: Commission internal group involving several Directorates-General and 
the FRA152. 

Consultation  meetings  with  civil  society  and  international  organisations: 
the 
Commission regularly organises meetings bringing together local authorities, civil society, 
international institutions and organisations (Council of Europe, UN agencies, OSCE).  

These  mechanisms  provide  a  permanent  system  of  cooperation.  During  the  workshop, 
several  NGOs  highlighted  the  stronger  cooperation  between  all  relevant  stakeholders 
resulting from the EU Roma framework, and acknowledged the role of the Commission 
in  achieving  it.  The  network  of  NRCPs  is  considered  by  the  NRCPs  interviewed  as  a 
good  opportunity  for  establishing  contacts  with  colleagues  in  other  Member  States 
dealing with similar situations and for exchanging practices153. 

At  the  national  level,  NRCPs  are  the  main  coordinators,  having  the  mandate  to 
coordinate the development  and implementation of the  NRISs.  In addition, the national 
Roma platforms established in most Member States and all enlargement countries aim to 
ensure inclusive involvement of and coordination with all relevant stakeholders (such as 
civil society, public authorities, Roma and non-Roma communities and business). In the 
enlargement region, since 2011 in all countries except  Turkey, the  government and the 
Commission organise a national Roma integration seminar every 2 years. The objective 
is for all partners (central and local authorities, civil society, international organisations, 
donors)  to  monitor  progress  and  agree  on  the  Roma  integration  priorities  to  be 
implemented in the next 2 years.  

Many of the NRCPs interviewed in the EU and enlargement countries (11 out of the 19 
that provided an opinion) say they have sufficient administrative capacity to effectively 
coordinate NRIS implementation154.  

The  NRCPs  have  improved  cooperation  among  public  administrations  and  between 
public administrations and other stakeholders. However, given their mandate and power, 
they have a limited influence on the design and implementation  of mainstream  policies 

150 For more information, see: http://www.rcc.int/romaintegration2020/home. 
151 euromanet.eu. 
152 (JUST, EMPL, EAC, SANTE, NEAR, HOME, REGIO, AGRI)  Other DGs — e.g. SG, BUDG, RTD, ECFIN, 

COMM — are part of the ‘extended taskforce’, which meets less frequently. The Roma taskforce was created in 
2010 (i.e. before the adoption of the EU Roma framework) with the goal to assess to use of EU funds for Roma 
inclusion, and its conclusions led to the setting up of the EU framework.. 

153 AT1, BE5, DE3, LU1, LV1, NL6. 

154 Interviews conducted with 26 NRCPs in the EU and enlargement countries, question ‘Is the administrative 

capacity of the NRCP sufficient to organise effective coordination of the implementation of the NRIS? What 
about in ministries or institutions responsible for the implementation of specific inclusion policies?’ In relation to 
the capacity of NRCPs: out of 26, 11 replied yes, 7 did not reply, and 8 provided comments pointing to the need 
to reinforce capacity. In relation to other ministries and institutions: in 7 cases the NRCPs explicitly replied ‘yes’. 

27 

 
 
 
 
                                                           
and,  more  generally,  on  policies  implemented  by  other  institutions  at  the  national  and 
local  level155.  Coordination  is  not  yet  well  integrated  into  the  policy  cycle  of  planning, 
budgeting,  implementation,  monitoring  and  evaluation.  Moreover,  NRISs  are  not  well 
known,  even  in  some  cases  by  key  administrative  departments  in  the  government. 
NRCPs  have  limited  power  in  influencing  decision-making  processes  across  policies, 
which  is  a  major  obstacle  for  their  NRIS  coordination  capacity.  In  other  cases, 
insufficient  coordination  seems  to  stem  from  a  lack  of  commitment  by  ministries,  the 
political context or the lack of participation at local and regional levels156. 

5.2.2. Stakeholder involvement 

At the EU level, stakeholder cooperation includes the Council of Europe, the Economic 
and  Social  Committee,  EU  agencies  such  as  the  FRA,  UN  agencies,  the  OSCE, 
EEA/Norway  Grants,  the  World  Bank,  civil  society  associations  and  foundations, 
representatives  of  municipalities,  thematic  umbrella  organisations  and  others.  In 
enlargement  countries,  partners  include  a  number  of  bilateral  and  multilateral  donors. 
Stakeholder cooperation and dialogue has been developed through regular events such as 
the EU Roma Summit, Roma platforms, and consultation meetings with civil society and 
international  organisations.  Synergies  were  achieved  through  the  cooperation  with  the 
Council  of  Europe  in  different  joint  programmes 157 .  With  regard  to  ensuring  their 
participation  in  decision-making  at  EU-level,  Roma  organisations  think  that 158 :  (i) 
dialogue  with  the  Roma  team  and  other  Commission  departments  could  be  more 
permanent;  (ii)  information  and  transparency  could  be  improved;  and  (iii)  the 
Commission should help facilitate dialogue with the NRCPs. 

At  national  level,  stakeholder  participation  —  including  of  Roma  representatives  and 
civil  society  —  has  been  boosted  through  the  EU  framework.  New  mechanisms 
introduced  are  diverse  and  include:  (i)  national  Roma  platforms;    (ii)  annual  national 
progress  reports  that  are  shared  with  stakeholders;  (iii)  dedicated  advisory  committees; 
(iv) projects to further strengthen cooperation with Roma NGOs; (v) regular consultation 
meetings with Roma representatives and other stakeholders (e.g. municipalities); and (vi) 
expert groups including both government and NGOs representatives159.  

Despite these efforts, weaknesses have been identified with regard to the involvement of 
civil  society  in  the  practical  implementation  of  the  NRIS  and  in  the  monitoring  and 
evaluation  processes.  As  regards  planning  and  implementation  of  the  ESIF,  it  is 
suggested that the degree and quality of stakeholder participation varies across countries, 
and  ‘in  many  cases  the  mechanisms  and  processes  for  involvement  are  weak  (e.g. 
informal)’ or limited to specific phases of the policy cycle160. 

The following chart161 illustrates this mixed assessment. 

155 Open Society European Policy Institute, 2017, Revisiting the EU Roma framework: Assessing the European 

dimension for the post-2020 future; European Parliament, 2017, Report on fundamental rights aspects in Roma 
integration in the EU: fighting antigypsyism; CEPS, 2017, Combating institutional antigypsyism responses and 
promising practices in the EU and selected Member States. 

156 Sources for these statements include BE4, DE1, DE2, DE3, HR2; CS-RO, CS-HU, CS-SI, CS-ES, CS-BG. 
157 https://www.coe.int/en/web/portal/roma. 
158 Views expressed at the evaluation workshop 
159 CY1 and LV1;DE3; EE2; NL6 and PT3; WB2 (Serbia). 
160 EURoma, 2016, Promoting the use of ESI Funds for Roma Inclusion. 
161 Source: NGO survey. 

28 

 
 
                                                           
Figure  2:  Reply  to:  ‘Please  indicate  the  extent  to  which  you  agree  with  the  following 
statement: The existing mechanisms for coordination and implementation of the NRIS of 
my  Member  State  allow  for  effective  cooperation  and/or  consultation  with  all  key 
stakeholders.’  

No opinion/ Do not 
know 
9% 

Strongly agree 
8% 

Strongly 
disagree 
23% 

Partially agree 
25% 

Partially 
disagree 
17% 

Neither agree, nor 
disagree 
19% 

29 

 
 
 
 
 
 
 
 
5.3. RELEVANCE 

The analysis of relevance looks at whether the EU framework objectives set in 2011 were 
appropriate  to  the  needs  at  the  time  and  whether  they  continue  to  be  relevant  today. 
Relevance of the EU framework has been assessed162 as positive overall, with some 
limitations as set out below. 

The analysis has been informed by seven evaluation questions that looked into the extent 
to which: 

 

 

the EU framework’s original objectives were appropriate in view of the original 
needs; and 
the  EU  framework’s  original  objectives  are  appropriate  in  view  of  the  current 
needs. 

Annex 6 provides an overview of the objectives of the EU framework and the proposed 
measures for Member States, together with an overview of  how these have been adapted 
and  extended  by  the  2013  Council  Recommendation.  In  general,  as  explained  above, 
while respecting Member State competences,  the EU framework provides flexibility163 
so that its objectives can be adapted to specific national contexts and changes that take 
place over time. This flexibility implies that the relevance of the EU framework hinges 
closely on the appropriateness and ambition of the concrete objectives and measures set 
out  in  the  individual  NRISs.  The  flexibility  and  the  non-binding  nature  of  the  EU 
framework have allowed Member States to follow a tailored approach taking account of 
specific national contexts when selecting measures. This evaluation therefore assesses the 
relevance of the EU framework’s general and specific objectives but not the relevance of 
the individual measures proposed. 

5.3.1. Relevance in light of the original needs 

The evaluation finds that the original objectives set in 2011 were appropriate in view of 
the Roma integration needs at the time, both in the Member States and in the enlargement 
countries.  Overall,  taking  into  account  the  baseline  situation  described  above  and  the 
persistent socio-economic exclusion of and discrimination against the Roma, the general 
objectives on poverty and discrimination and the specific objectives on employment, 
education, health and housing have been confirmed as relevant for Roma inclusion. 
This  view  was  clearly  shared  by  the  workshop  participants  when  discussing  the 
preliminary  findings  of  this  evaluation.  During  the  OPC,  an  overwhelming  majority  of 
respondents (between 86 and 95 % depending on the thematic area) agreed that targeted 
public interventions are needed in these areas. 

The evaluation finds, however, that for education, housing and non-discrimination, 
relevance of the EU framework could have been stronger. 

162 Using the following criteria: very positive/positive/limited/negative/very negative. 
163 See Section 2, description of the initiative; in particular: designing NRIS taking into account the size of the Roma 
population and different starting points; also non-binding approach to measures to be implemented by Member 
States. 

30 

 
 
 
 
 
 
 
 
                                                           
Education:  while  relevant,  the  Roma  integration  goal  in  the  area  of  education  was 
considered  to  be  insufficiently  ambitious  by  several  civil  society  organisations  and 
interviewees164. Indeed, primary school completion rates of Roma children were already 
quite high at the time the EU framework was introduced. The relevance of the EU Roma 
framework  could  have  been  increased  by  other  objectives  such  as:  (i)  the  transition  of 
Roma  from  lower  secondary  to  upper  secondary  education;  (ii)  reducing  the  share  of 
early leavers from education and training; and (iii) focusing on  reducing  gaps between 
Roma and the rest of the population in the other areas. 
Housing:  the  Roma  integration  goal  to  close  the  housing  and  essential  services  gap 
between Roma and the rest of the population is a key factor for improving the integration 
of Roma into society. While acknowledging that the EU framework’s specific objective 
was  appropriate  in  view  of  the  original  needs,  the  civil  society  stakeholders  consulted 
165 expressed  regret  that  there  were  no  tailored  housing  objectives  across  the  Member 
States  to  address  country-specific  needs.  The  2011  FRA  Roma  survey  and  the  data 
collected for the country analysis reports indicate that deprived living conditions, lack of 
basic  infrastructure  and  segregation  among  Roma,  was  worst  in  Eastern  European 
countries  —  particularly  so  in  Hungary,  Romania  and  Slovakia.  In  countries  such  as 
Austria and Finland, the problems faced by Roma mainly related to access to housing due 
to discrimination, rather than to housing conditions or housing segregation166. 
Non-discrimination:  a  key  conclusion  of  the  workshop  was  that  the  EU  framework’s 
relevance was reduced because non-discrimination received insufficient attention. Indeed, 
the  EU  framework  did  not  set  a  specific  non-discrimination  goal  but  referred  to  the 
existing legal obligations of Member States. 

The evaluation also finds that the following factors reduced the relevance of the EU 
framework in light of the original needs. 

a) The scope of the Roma integration goals: Stakeholders  consider that some issues that 
were highly relevant to the needs of the Roma at the time the EU framework was adopted 
merited  more  attention.  These  include  gender  equality,  Roma  empowerment  and  active 
participation,  including  civil  and  political  participation 167 .  The  2013  Council 
Recommendation  reinforced  these  objectives  by  inviting  Member  States  to  pay  special 
attention  to  the  gender  dimension  and  by  introducing  the  empowerment  of  Roma  as  a 
horizontal policy measure. 

b)  The  EU  framework  gives  a  broad  definition168 of  its  target  population  but  does  not  
highlight  specific  groups  to  be  addressed,  for  example  travellers  or  other  subgroups. 
Despite the political context at the time of adoption of the EU framework (see chapter 2), 
questions concerning the freedom of movement of Roma EU citizens were not included 
in the EU framework. Similarly, with regard to non-EU Roma nationals legally residing 
in the EU, only a short reference to EU policies promoting integration of this group was 

164 Workshop; EUI1, and B1. 
165 Central Council of German Sinti and Roma, ERGO Network and ENAR. 
166 CS-AT; CS-FI; CS-HU; CS-RO; CS-SK. European Union Agency for Fundamental Rights, Poverty and 

employment: the situation of Roma in 11 EU Member States. Roma survey — Data in focus, Publications Office 
of the European Union, 2014. 

167 Workshop participants. EUI-1. European Parliament, Study on the Empowerment of Roma Women within the 

European Framework of National Roma Inclusion Strategies, 2013, p. 10. 

168 The term ‘Roma’ is used, in line with the terminology of European institutions and international organisations, to 
refer to a number of different groups (such as Roma, Sinti, Kale, Gypsies, Romanichels, Boyash, Ashkali, 
Egyptians, Yenish, Dom, Lom, Rom, Abdal) and also includes travellers, without denying the unique features 
and varieties of lifestyles and situations of these groups. 
31 

 
 
 
 
                                                           
included in the EU framework. For the enlargement region, access to civil documentation 
was only identified at a later stage as an area requiring full attention169. 

5.3.2. Relevance in light of the current needs 

The  evaluation  finds  that  the  original  general  and  specific  objectives  continue  to  be 
relevant  today, both  in Member States and in  enlargement countries.  As set  out in  the 
effectiveness section, although there is visible progress on education, progress in  other 
areas is slower. Despite better economic conditions, Roma continue to have much higher 
rates  of  being  at  risk  of  poverty,  having  lower  employment  and  higher  numbers  of 
NEETs, substandard housing conditions and unequal access to healthcare services.  The 
survey  of  national  NGOs  also  shows  that  the  objectives  remain  largely  relevant  to  the 
current needs170. 

a)  With  regard  to  the  four  Roma  integration  goals  (see  effectiveness  section  for 
more detail on the 2016 situation of Roma in these areas): 

Education:  this  goal  would  be  more  relevant  to  current  needs  if  it  did  not  focus 
exclusively  on  primary  education  but  was  extended  to  aspects  such  as  transition  from 
lower  to  upper  secondary  school  attendance,  early  school  leaving  (ESL)  and  transition 
from education to employment. These are areas were the gaps between Roma and non-
Roma remain high. 
Health: despite some improvements, the analysis finds that the Roma integration goal in 
the area of health is still appropriate today.  Although the gap between Roma and non-
Roma  largely  disappeared  for  self-reported  health  status,  it  remained  practically 
unchanged  for  medical  insurance  coverage  since  2011  (see  effectiveness  section).  The 
actual (not self-reported) health status of Roma remains unknown, but EU-wide studies 
reveal  ‘indirect  evidence’  such  as  shorter  life  expectancy  versus  the  population  as  a 
whole,  that  clearly  indicates  a  health  disparity 171 .  The  Roma  population  is  also 
disproportionately  affected  by  communicable  diseases 172 .  Persistently  poor  living 
conditions  often  result  in  a  higher  probability  of  serious  illnesses  and  chronic  diseases, 
even  when  access  to  healthcare  is  provided173.  Furthermore,  little  progress  has  been 
made  on  preventative  healthcare,  as  Roma  continue  to  have  consistently  lower  child 
vaccination rates compared to non-Roma174.  
Employment:    the  original  objective  still  corresponds  to  the  current  needs,  as  the 
employment  gap  between  the  Roma  and  the  general  population  remains  very  high. 
Furthermore, nearly half of the respondents to the NGO survey considered that the needs 
of  Roma  youth  and  Roma  women  were  not  addressed  in  employment  measures  and 
programmes175.  

169 This issue was first explicitly addressed as a problem for Roma by the 2013 enlargement strategy, COM(2013) 

final. 

170 The share of respondents perceiving the objectives as relevant accounted for 60 % (strongly — 20 % or partially 

— 40 %) while 20 % disagreed (partially — 14 % or strongly — 6 %). 

171 European Commission (2014) Roma Health Report, 

http://eige.europa.eu/resources/2014_roma_health_report_en.pdf. 

172 Ibid. 
173 CS-EL and CS-AT. 
174 European  Commission,  Roma  Health  Report  —  Health  status  of  the  Roma  population.    Data  Collection  in  the 

Member States of the European Union, 2014,. 

175 NGO survey. 

32 

 
 
 
 
 
 
                                                           
Housing:  this  Roma  integration  goal  also  continues  to  be  relevant  to  the  needs  of  the 
Roma.  About  half  of  the  country  analysis  studies  identify  housing  as  one  of  the  most 
important  areas  for  further  work176 —  a  view  that  is  shared  by  several  stakeholders 
consulted in the Member States and enlargement countries. The OPC confirms the lack 
of  progress  between  2011  and  2016,  with  more  than  half  (57.3 %)  of  respondents 
indicating  the  housing  situation  has  worsened  due  to  higher  levels  of  housing 
discrimination. 

b)  In  light  of  the  current  needs,  the  evaluation  finds  that  the  general  objective  of 
non-discrimination needs updating.  

Already  in  2016  the  Commission177 highlighted  the  importance  of  more  systematic 
approaches  and  greater  political  will  on  the  part  of  Member  States  in  fighting 
antigypsyism.  Levels  of  discrimination  remain  high  (see  section  on  effectiveness)  and 
antigypsyism is  a critical  issue as  recently  highlighted by the European  Parliament  and 
the  Fundamental  Rights  Agency178.  Stakeholders,  including  workshop  participants179, 
found that compared to the general objective of combating poverty and the four specific 
objectives,  giving more weight to the anti-discrimination objective is critical to address 
antigypsyism180 and  its  manifestations  in  hate  speech  and  crime181.  The  OPC  results 
show that, in  addition  to  noting a  rise in  discrimination,  respondents  find that little has 
been  done  to  actively  fight  discrimination  within  the  four  main  policy  areas  over  the 
course of the current EU framework. 

Figure 3: Perception of progress/implemented measures to fight discrimination in the four main 
policy areas 2011-2016 

176 CS-BG; CS-EL; CS-FI; CS-FR; CS-IT; CS-RO. BE3; BE4; DE1; IE1; IE2; LT1; LT2: LT4; NL3; PT2; PT3; SE3; 

SE4; UK4; WB3 (Regional Cooperation Council). 

177 COM(2016) 424. 
178 European Parliament Report on fundamental rights aspects in Roma integration in the EU: fighting anti-Gypsyism, 
11 October 2017/2038(INI). Alliance against Antigypsyism. 2017. Antigypsyism — a Reference Paper. Berlin / 
Budapest.  http://antigypsyism.eu/;  FRA  2018:  A  persisting  concern:  anti-Gypsyism  as  a  barrier  to  Roma 
inclusion, Vienna. 

179 Expressed  by  the  Central  Council  of  German  Sinti  and  Roma,  ERGO  Network,  Open  Society  Foundations  and 

ENAR. 

180 Centre  for  European  Policy  Studies,  Research  Report:  Combating  Institutional  Anti-Gypsyism.  Responses  and 
promising  practices  in  the  EU  and  selected  Member  States,  No  2017/08,  2017,  p.  26,  European  Parliament, 
Resolution of 9 March 2011 on the EU strategy on Roma inclusion, P7_TA(2011)0092,. 

181 Open Society Foundations, European Roma and Travellers Forum, Roma Women’s Network; EU-13. 

33 

 
 
 
 
 
 
 
                                                           
Source: OPC — Public consultation on the Evaluation of the EU framework for national Roma integration strategies 
up to 2020, 2017. 

In order to make more progress towards the Roma integration goals, more efforts are thus 
needed  to  fight  discrimination  and  anti-gypsyism  if  equal  access  to  education, 
employment, health and housing is to be ensured.  

c)  The  EU  framework’s  objectives  remain  relevant  also  from  an  economic 
perspective 

The  effects  of  Roma  exclusion  on  welfare  dependency,  the  labour  market  and  tax 
revenues182 were already clear at the time the EU framework was adopted. Since then, 
for the five Member States with the highest share of Roma (BG, CZ, HU, RO and SK) 
the  country  reports  prepared  in  the  context  of  the  European  Semester  have  repeatedly 
underlined  the  risks  of  adverse  demographic  trends.  Low  fertility  rates  are  expected  to 
cause  the  working-age  population  to  shrink,  leading  to  difficulties  with  labour  supply, 
shortages  of  skilled  and  unskilled  workers.  These  factors  are  projected  to  hamper 
potential growth in the next years183.  

In  this  climate,  the  economic  rationale  for  providing  Roma  with  skills  and  integrating 
them  in  the  open  labour  market  is  clear  in  ageing  Central  Eastern  European  societies. 
Recent research carried out for the European Parliament illustrates the economic impacts 
of discrimination  on grounds of race and  ethnicity, ranging from  GDP  and tax revenue 
loss to direct costs in mental health provision184. Roma integration is also relevant for 
attaining the Sustainable Development Goals185 of the United Nations 2030 Agenda for 
Sustainable Development, as many countries cannot do this without making progress on 
Roma inclusion186. 

182 InGrid  —  Integrating  expertise  in  inclusive  growth;  Methodological  and  data  infrastructure  report  on  Roma 
population  in  Europe;  August  2016;  World  Bank  (2010)  —  Roma  Inclusion:  An  economic  opportunity  for 
Bulgaria, Czech Republic, Romania and Serbia. 

183 See for example the European Semester 2017 country reports 
184 European Parliamentary Research Service, March 2018, cost of non-Europe report, Equality and the Fight against 

Racism and Xenophobia; based on conceptual framework for the impacts of discrimination, page 28. 

185 Among the sustainable development goals the following are of  highest relevance  for Roma inclusion: 1.3 social 
protection floors, 2.1 hunger and food security, 3.8 universal health coverage, 4.1-6 access to quality education, 
6.1  access  to  safe  drinking  water,  11.1  affordable  housing.  Gender  equality  is  transversal  to  all  goals  (and 
indicators)  and  specifically  mentioned  in  goal  5.  Equal  treatment  is  covered  in  a  cross-cutting  manner  and 
specifically  in  target  10.3.  Also  relevant  are:  16.1  protection  from  violence  and  16.7  political  participation. 
Target 17.18 calls for ethnically disaggregated data collection. 

186 FRA 2018; A persisting concern: antigypsyism as a barrier to Roma inclusion 

34 

 
 
 
 
 
 
 
                                                           
5.4. EQUITY 

For this evaluation, the analysis of equity considers whether the EU framework has the 
capacity  to  address    the  needs  of  subgroups  among  Roma.  Equity  is  closely  related  to 
effectiveness and relevance (hence its inclusion here directly after the effectiveness and 
relevance  chapters),  but  looks  more  deeply  into  the  needs  of  Roma  subgroups.  This 
analysis  is  important  as  ‘the  Roma’  targeted  in  the  EU  framework  may  experience 
different  forms  of  disadvantage  which  need  to  be  addressed.  The  analysis  is  therefore 
presented  separately.  Equity  of  the  EU  framework  has  been  assessed  as  limited 
overall. 

The assessment was informed by four evaluation questions that looked into the extent to 
which the EU framework addresses the needs of subgroups among Roma. The evaluation 
concludes that the EU framework, as adopted in 2011, does not sufficiently address the 
needs  of  specific  subgroups.  The  country  studies  and  the  targeted  stakeholder 
consultations  reveal  that  whenever  special  attention  is  given  to  vulnerable  Roma 
subgroups  at  national  level,  Roma  children,  youth  and  women  are  most  likely  to  be 
included187,  and  Roma  with  disabilities  to  a  much  lesser  extent188.  Various  sources, 
including  contributions  during  the  workshop,  the  country  studies  and  interviews  with 
national stakeholders, indicate that several other subgroups currently receive insufficient 
attention, if any at all, in the EU framework and at national level. These include Roma 
migrants189, LGBTI Roma190 and Muslim Roma191. Insufficient attention is also given 
to the distinction between urban and rural Roma192. 

With  regard  to  the  three  subgroups,  the  EU  framework  only  very  briefly  refers  to  the 
situation of Roma  women, pointing out their lower employment rates and the need for 
better access to quality healthcare. This was partly rectified by the new orientation given 
in the 2013 Council Recommendation, which also speaks about the need to fight violence, 
including  domestic  violence,  violence  against  women  and  girls,  trafficking  in  human 
beings,  underage  and  forced  marriages,  and  begging  involving  children.  However,  the 
absence of an explicit focus on gender equality in the EU framework has meant that the 
NRISs  lack  indicators  and  targets  to  tackle  Roma  women’s  specific  situation,  and  also 
lack  the  political  commitment  at  national  level,  to  take  a  more  proactive  approach  to 
addressing  gender  needs193.  Many  NRIS  either  take  a  gender-neutral  approach194 or 
they  lack  explicit  gender  measures  in  nearly  all  areas  except  for  health 195 .  Some 
enlargement countries provide a positive example, such as the former Yugoslav Republic 
of  Macedonia,  where  a  national  plan  for  strengthening  the  position  of  Roma  women 

187 CS-BG; CS-EL; CS-FI; CS-HU; CS-RO; CS-SK; HR2; SE2. 
188 CS-BG; CS-FI; HR2. 
189 Two NRCP participants in the workshop; CS-FR; CS-ES; CS-FI; WB6 (former Yugoslav Republic of Macedonia). 
190 CS-SK. 
191 IE5; SE2. The first Finnish national policy on Roma did include religion (CS-FI). 
192 NRCP participant in the workshop; CS-FR; LT3; PT4. 
193 European Roma and Travellers Forum, Report: National Roma Integration Strategies. Evaluating Gender, 2015, p. 
7; European Parliament, Evaluation of the EU Framework for National Roma Integration Strategies, 2015. 
194 European Parliament, Study on the Empowerment of Roma Women within the European Framework of National 

Roma Inclusion Strategies, 2013, p. 10 and 41. 

195 Per the country analysis studies, FR, ES, RO and SK do put a focus on women in the area of health. 

35 

 
 
 
                                                           
includes  the  goal  of  Roma  women  and  girls’  enjoyment  of  their  human  rights  without 
discrimination196. 

Among  the  four  areas  of  the  EU  framework,  it  is  in  the  employment  area  that  the  gap 
between  Roma  men  and  Roma  women  has  increased  between  2011  and  2016,  both  in 
terms of employment rates and NEET rates. 

Table 7: Difference between male and female experiences in employment (2011-2016) 

Difference 
b/w  male  / 
female  
2011 in pp 

 Difference 
/ 
b/w  male 
female    2016 
in pp 

Roma 
Male 

Roma 
Female 

(2016) 

(2016) 

Share of people with the  self-declared main activity 
‘paid  work’  (including  full-time,  part-time,  ad  hoc 
jobs,  self-employment),  household  members,  16+ 
(%) 

Share  of  young  persons,  16-24  years  old  whose 
current  main  activity  is  neither  in  employment, 
education or training, household members (%) 

11 

11 

18 

17 

34 

16 

55 

72 

Source:  Unpublished  information  from  FRA  Roma  Survey  (2011)  and  EU-MIDIS  II  survey  presented  by  FRA  to 
Commission services in February 2018 

As set out in the relevance section, stakeholders consider that gender equality was highly 
relevant to the needs of the Roma at the time the EU framework was adopted and thus 
merited  more  attention197.  The  above  findings  in  the  area  of  employment  confirm  this 
view. 

The  EU  framework  mentions  that  young  Roma  should  be  strongly  encouraged  to 
participate  in  secondary  and  tertiary  education  but  provides  no  further  guidance  or 
specific objective. Today however, the rising levels of young Roma aged 16 to 24 who 
are not in employment, education or training (NEET) — an increase from 56 % in 2011 
to 63 % in 2016 — is a challenge that needs to be addressed. Research at national level 
indicates that only a small number of countries include references to the needs or special 
situation of young Roma in their NRIS198. One factor that could explain the absence of 
focus on Roma youth at national level is the lack of statistical data on Roma youth. This 
was identified in several country studies as a factor  hindering insight into the situation of 
young  Roma199 that  could  help  those  who  advocate  for  Roma  youth  measures200 or 
analyse  the  extent  to  which  young  Roma  benefit  from  mainstream  measures201.  The 
results  of  the  NGO  survey  confirmed  the  view  that  key  policy  measures  for  Roma 
integration  at  national  level  only  address  the  needs  of  Roma  youth  to  a  limited  extent. 
The respondents were divided on whether Roma young people’s needs were addressed in 
the area of education, while they largely agreed that they were not sufficiently addressed 
on healthcare, employment, housing and equality and anti-discrimination. As is the case 
for Roma women, it was not the EU framework but the 2013 Council Recommendation 
that  put  a  focus  on  the  needs  of  Roma  children.  In  2015,  the  European  Parliament 
stressed the need to  prioritise Roma children in  the promotion  of the Roma’s access  to 

196 WB11 (former Yugoslav Republic of Macedonia). 
197 Workshop  participants.  EUI-1.  European  Parliament,  Study  on  the  Empowerment  of  Roma  women  within  the 

European framework of national Roma inclusion strategies, 2013, p. 10. 

198 CS-EL; CS-HU; SI1; SE3; BE4; HU4. 
199 CS-IT. 
200 CS-ES. 
201 CS-BG. 

36 

 
 
 
 
                                                           
education,  healthcare  and  housing  when  implementing  the  EU  framework202.  In  2016, 
the  Commission  called  the  situation  of  Roma  children  ‘particularly  worrying 203 ’. 
Country  analysis  studies  and  targeted  stakeholder  interviews  only  provided  limited 
information on specific measures for  Roma children in the Member States. According to 
OPC  respondents,  education  is  an  area  in  which  at  least  some  of  the  needs  of  Roma 
children were addressed. However, 71% of the OPC respondents disagreed that measures 
were  taken  in  terms  of  promoting  Roma  girls’  school  participation  and  65%  of  OPC 
respondents  disagreed  that  measures  against  misdiagnosis  and  misplacement  of  Roma 
children were taken. 

Both  Roma  EU-mobile  citizens  and  Roma  third-country  nationals  are  barely 
considered  in  the  EU  framework  and  NRIS  of  reception  countries 204 .  We  must 
remember that restrictions on the free movement of workers from Bulgaria and Romania 
after EU accession were lifted in 2014, and while this was not specific to Roma, it also 
affected  them.  Since  that  date,  a  significant  number  of  people  from  these  countries 
moved  to  EU-15  Member  States 205 .  The  2013  Council  Recommendation  already 
highlighted  the  need  to  ‘respect  the  right  to  free  movement’  of  EU  citizens  and  ‘the 
conditions  for  its  exercise,  including  the  possession  of  sufficient  resources  and  of  a 
comprehensive sickness insurance cover, in accordance with Directive 2004/38/EC of the 
European  Parliament  and  of  the  Council,  while  also  seeking  to  improve  the  living 
conditions  of  Roma  and  pursuing  measures  to  promote  their  economic  and  social 
integration  in  their  Member  States  of  origin  as  well  as  their  Member  States  of 
residence.’206  With regard to Roma who are third-country nationals staying legally in a 
Member State,  the 2013 Council Recommendation states that they  can also  be put  in  a 
vulnerable  position,  particularly  when  they  share  the  same  poor  living  conditions  as 
many  Roma  who  are  citizens  of  the  EU,  while  also  facing  the  challenges  of  many 
migrants  coming  from  outside  the  EU.  Country  studies  confirm  the  high  level  of 
discrimination  faced  by  Roma  migrants  and  EU-mobile  Roma  for  being  Roma  and 
migrants  at  the  same  time,  their  precarious  living  conditions  and  issues  related  to  their 
residence  status 207 .  Several  interviews  with  stakeholders  have  highlighted  concerns 
about  Roma migrants  and the need to  recognise this group as a specific area for action 
under the EU Roma framework208. 

202 European Parliament, Resolution of 15 April 2015 on the occasion of International Roma Day — antigypsyism in 
Europe and EU: recognition of the memorial day of the Roma genocide during World War II, P8_TA(2015)009, 
http://www.europarl.europa.eu/sides/getDoc.do?type=TA&language=EN&reference=P8-TA-2015-0095. 

203 COM(2016) 424, p. 9 and 17, available at http://ec.europa.eu/justice/discrimination/files/roma-report-2016_en.pdf. 
204 EUROCITIES (2017), Roma Inclusion in cities — Mapping of the situation of Roma in cities in Europe). 
205  http://ec.europa.eu/social/BlobServlet?docId=14000&langId=en.  Open  Migration,  2015,  From  Morocco  to 
Romania:  how  immigration  to  Italy  has  changed  over  10  years,  http://openmigration.org/en/analyses/from-
morocco-to-romania-marocco-how-immigration-to-italy-changed-in-10-years/;  CBS,  2017,  More  Eastern 
Europeans  working 
the  Netherlands,  https://www.cbs.nl/en-gb/news/2017/05/more-eastern-europeans-
working-in-the-netherlands; The Guardian, 2017, Number of Romanians and Bulgarians in UK rises to 413 000, 
https://www.theguardian.com/uk-news/2017/oct/11/number-of-romanians-and-bulgarians-in-uk-rises-413000 

in 

206 Recital (12). 
207 CS-AT and CS-FR. 
208 EUI7, EUI4, UK 4, IE1. 

37 

 
 
 
 
 
                                                           
5.5. COHERENCE 

The analysis of coherence focuses on how well the EU framework works together with 
other  EU  and  national  instruments.  The  coherence  of  the  EU  framework  has  been 
assessed as positive at EU level and as limited at national level. 

The  assessment  has  been  informed  by  nine  evaluation  questions  that  looked  into  the 
extent to which209: 

the EU framework is consistent with other EU policies; 

 
  policy, legal and financial instruments at EU level were mobilised and aligned to 

accomplish the objectives of the EU framework; and 

  policy,  legal  and  financial  instruments  at  national  level  were  mobilised  and 

aligned to meet the objectives of the EU framework. 

5.5.1. EU level: internal coherence210 

The EU framework was the first comprehensive EU approach to Roma integration across 
the  four  policy  areas.  However,  the  formulation  of  the  Roma  integration  goals  was  not 
consistent or in line with the SMART criteria (specific, measurable, achievable, relevant 
and  time-bound 211 ).  While  the  goal  for  education  was  not  particularly  ambitious, 
considering  the  situation  in  2011212 the  goal  for  housing213 was  very  ambitious.  The 
goals  were not easy to measure214 given issues of comparability of data for Roma and 
for  the  general  population,  nor  were  they  adequately  specific.  The  fact  that  three  goals 
aim  to  bridge  gaps  between  Roma  and  non-Roma  while  the  education  goal  does  not 
relate to gaps, is another inconsistency. 

The  evaluation  also  finds  it  less  coherent  that  in  contrast  to  having  Roma  integration 
goals in the areas of employment, education, housing and health, the EU framework did 
not include a specific non-discrimination goal215. Non-discrimination was rather set as a 
cross-cutting theme216, which was identified as a significant shortcoming of the existing 
EU framework by the European Court of Auditors217 and others. 

However, consistency between the EU framework and the work of the FRA is found to 
be strong. The FRA has generated valuable evidence and data on living  conditions  and 
discrimination  of  Roma,  in  particular  through  its  2011  Roma  pilot  survey,  the  2016 
second  EU  minorities  and  discrimination  survey  (EU-MIDIS  II)  and  the  Roma 

209 The external evaluation study did not look into internal coherence. 
210 The findings on internal coherence are mainly based on Commission analysis. 
211 Better Regulation principles. 
212 ‘ensure that all Roma children complete at least primary school’ — much more ambitious would have been a goal 

on completion of secondary education for example. 

213  ‘close the gap between the share of Roma with access to housing and to public utilities and that of the rest of the 

population’. 

214 ‘reduce the gap in health status’ and ‘cut the employment gap’. 
215 European Roma Information Office, position paper on the EU framework for national Roma integration strategies 
and a post-2020 strategy, 2017; UN-OHCHR 2017: Lessons Learned: Views in the context of midterm review of 
implementation of the EU framework for national Roma integration strategies 2012-2020 

216 Which was more explicit under education, employment and housing than under health. 
217 European  Court  of  Auditors,  Special  report:  EU  policy  initiatives  and  financial  support  for  Roma  integration: 
significant  progress  made  over  the  last  decade,  but  additional  efforts  needed  on  the  ground,  2016, 
https://www.eca.europa.eu/Lists/ECADocuments/SR16_14/SR_ROMA_EN.pdf. 

38 

 
 
 
 
 
 
 
                                                           
integration  indicators  scoreboard  (2011-2016) 218 .  In  2012,  the  FRA,  in  close 
cooperation  with  the  European  Commission,  established  a  working  party  on  Roma 
integration  indicators  to  help  EU  Member  States  develop  indicators  to  monitor  the 
progress of the NRIS. 18 Member States participated in the process, represented by their 
NRCPs,  which  consulted  with  a  variety  of  national  stakeholders  including  competent 
ministries,  statistical  agencies, structural  funds managing authorities, regional  and local 
authorities, Roma civil society and communities, and other institutions that have a role in 
implementing  the  NRIS 219.  This  process  resulted  in  the  development  of  the  online 
reporting  system  used  by 
information  on 
implementation of the NRISs. 

to  collect  annual 

the  Commission 

5.5.2 EU level: external coherence 

Intervention  logic  Annex  7a  illustrates  that  progress  on  Roma  integration  depends  on 
consistency between the EU framework’s objectives and those of other EU and national 
policy, legal and financial instruments. 

Table 8: Coherence at EU level (partly based on ICF, Chapter 3.3) 

EU Policies 

Coherent 

Partial 

Not coherent 

Comment 

  X 

  X 

Mainstreaming  Roma 
integration 
EU 
policies 

in 

Fighting 
discrimination and 
racism legislation 

EU funding 

  X 

is 

consistency 
There 
between 
EU 
the 
framework  and  other  key 
EU  policies  (for  example 
European  Semester)  but 
some gaps were detected. 

framework 

is 
The  EU 
overall 
externally 
consistent 
with 
instruments  in  this  area 
but the lack of measurable 
anti-discrimination 
target 
leads to partial coherence 

The  introduction  of    the 
ESIF ex ante conditionality 
and  IP  9.2  has  increased 
consistency. 

Mainstreaming into policies 
Overall, progress has been made in mainstreaming Roma integration in EU policies. On 
employment,  education  and  poverty,  3  out  of  5  Europe  2020  headline  targets  are 
directly  linked  to  the  EU  framework  targets  for  Roma  integration:  (i)  the  fight  against 
poverty  and  social  exclusion;  (ii)  raising  employment  levels;  and  (iii)  reducing  school 
dropout rates while increasing attendance in tertiary education.  

Making sufficient progress towards these three Europe 2020 targets requires addressing 
the  situation  of  Roma220.  The  2013  Council  Recommendation  on  Roma  integration 
therefore called on Member States to ‘… depending on the size and social and economic 
situation  of  their  Roma  populations,  consider  making  Roma  integration  an  important 

218 See the FRA website dedicated to Roma issues here fra.europa.eu/en/theme/roma/ 
219 FRA paper ‘Measuring Roma inclusion strategies — a fundamental rights based approach to indicators’, June 

2016. 
220 COM(2012) 226 final. 

39 

 
 
 
 
 
 
 
 
 
 
 
                                                           
issue  within  their  national  reform  programmes  or  their  national  social  reports  in  the 
context of the Europe 2020 strategy.’  

Country-specific  recommendations  (CSRs)  have  been  addressed  to  Bulgaria,  the  Czech 
Republic, Hungary, Romania and Slovakia in the context of the European Semester since 
2012. In addition, the ESIF regulations established a link between the European Semester 
and  EU  funding  in  the  2014-2020  ESIF  programming  period.  Member  States  which  at 
the  time  of  programming  had  received  CSRs  on  Roma  integration  were  to  include  the 
above-mentioned  IP  9.2  in  their  operational  programmes  (OP)  and  use  EU  funds  for 
thematic  ex  ante 
necessary 
conditionality221 and its fulfilment criteria applicable, closely linked to the objectives of 
the EU framework. 

the  Roma-related 

reforms.  This 

turn  made 

in 

Other policies also include an explicit link to the EU framework. 

  Roma  integration  has  been  fully  mainstreamed  into  European  enlargement 
policy and explicitly identified as one of the key priority areas. Progress on Roma 
integration is now also fully included in the negotiations of enlargement Chapter 
23  ‘Judiciary  and  Fundamental  Rights’.  Furthermore,  the  Commission  analyses 
the state of play by country in the context of the annual enlargement package.  
  Roma  integration  has  been  reflected  in  the  EU  urban  agenda,  allowing  for  a 

better mainstreaming of Roma integration into urban policies222.  

  The  framework  is  coherent  with  EU  anti-trafficking  policies  which  often 
address  the  situation  of  Roma  and  their  particular  risks  and  vulnerability  to 
trafficking in human beings223.  

  In the area of health, the European Commission’s recent proposal for a recast of 

the Drinking Water Directive224 refers directly to the 2011 EU framework. 

There  are  also  policy  initiatives  with  no  explicit  link  to  the  objectives  of  the  EU 
framework. Roma integration is not explicitly considered in social policy and education 
initiatives  such  as  the  Youth  Guarantee 225 ,  the  Skills  Agenda,  the  Alliance  for 
Apprenticeships  and  the  Pillar  of  Social  Rights.  These  initiatives  take  a  mainstream 
approach  to  vulnerable  and  disadvantaged  groups,  mentioning  Roma  specifically  either 
fleetingly or not at all. The non-explicit reference to Roma in initiatives such as the ones 
mentioned  may  be  the  reason  why  a  large  share  of  the  OPC  respondents  (88.9 %) 
identified the lack of effective mainstreaming of Roma integration in policy as one of the 
main challenges at EU level in the 2011-2016 period. 

Legislation on fighting discrimination and racism 

221 Ex ante conditionalities are preconditions that Member States must fulfil when submitting their programming 

documents. 

222 Eurocities. 2017: Roma in Cities in Europe. Brussels. 
223 Directive 2011/36/EU and the related EU policy framework take an integrated and holistic, victim-centred, gender-
specific, child sensitive, human rights-based approach. Article 5 of the Charter on Fundamental Rights addresses 
human trafficking. Study on high-risk groups for trafficking in human beings (2015); Study on comprehensive 
policy  review  of  anti-trafficking  projects  funded  by  the  European  Commission  (2016);  first  progress  report 
(COM(2016)  267);  2012-2016  EU  strategy  towards  the  eradication  of  trafficking  in  human  beings 
(COM(2012) 286);  European  Commission  Communication  on  Reporting  on  the  follow-up  to  the  EU  strategy 
towards the eradication of trafficking in human beings and identifying further concrete actions, COM(2017) 728 
final. 

224 COM(2017) 753 final. 
225 Compare ERGO network (2016).  Youth Guarantee  — Opportunities for  young Roma. Findings of a small scale 
in  six  EU  countries.  http://ergonetwork.org/wp-content/uploads/2017/05/Youth-Guarantee-

field  research 
%E2 %80 %93-Synthesis-note_ERGO-Network.pdf. 

40 

 
 
 
                                                           
The  Racial  Equality  Directive,  together  with  the  Council  Framework  Decision  on 
combating  racism  and  xenophobia  of  2008  and  other  instruments,  has  provided  a 
normative  ground  for  monitoring  the  fundamental  rights  situation  of  Roma 226.  The 
Directive’s implementation has been monitored by the Commission227 And a review of 
the  enforcement  of  anti-discrimination  law  in  relation  to  Roma  found  that  they  do  not 
benefit equally from the non-discrimination legal instruments, and a lack of enforcement 
is  particularly  evident  in  the  areas  of  education  and  housing.  Consequently,  the 
Commission  initiated  infringement  procedures  over  the  school  segregation  of  Roma 
children against three countries228 and put in place several policy tools to help national 
to  combat 
authorities 
antigypsyism229 among other issues. 

implement  anti-hate  speech  and  hate  crime 

legislation 

The  EU  framework’s  socio-economic  inclusion  standards  and  the  provisions  of  the  EU 
legislation  complement  each  other  significantly  in  terms  of  aims  and  scope.  The 
Commission’s  enforcement  of  the  EU  legislation  can  strengthen  the  EU  framework’s 
policy  objectives,  while  the  monitoring  and  implementation  of  the  EU  framework 
informs  the  Commission  on  the  state  of  play  of  the  EU  legislation  and  supports  its 
practical  application.  About  83 %  of  the  OPC  respondents  considered  the  measures 
relating to the monitoring and enforcement of EU anti-discrimination laws as very useful 
for increasing political commitment to ambitious public policies for Roma inclusion230. 
However, as set out under the ‘internal coherence’ section, the lack of specific objectives 
on non-discrimination is seen as a missed opportunity for full coherence. 

EU funding 

In  the  2007-2013  programming  period,  EU  funding  regulations  targeted  disadvantaged 
groups in general, but not explicitly Roma. 

For the 2014-2020 period, however, substantial changes were made. The Rights, Equality 
and Citizenship Programme (REC) as well as the instrument for pre-accession assistance 
(IPA), made  funding for Roma integration a priority.  Going further than  this, the ESIF 
regulations231 established an explicit link between the EU framework and EU funding. 

  IP 9.2  of 

the  ESF  entitled  ‘Socio-economic 

integration  of  marginalised 

communities such as the Roma’ was introduced. 

226 Chopin, Isabelle, Catharina Germaine, and Judit Tanczos 2017: Roma and the Enforcement of Anti-Discrimination 
Law,  Brussels.  http://ec.europa.eu/newsroom/just/document.cfm?action=display&doc_id=47560;  FRA  and  EC. 
2018.  Handbook  on  European  Non-Discrimination  Law  2018  Edition.  Vienna  /  Brussels:  European  Union 
Agency 
http://fra.europa.eu/en/publication/2018/handbook-european-law-non-
discrimination;  EC.  2016.  European  Handbook  on  Equality  Data  —  2016  Revision.  Brussels  (European 
Commission — DG Justice) http://ec.europa.eu/newsroom/just/item-detail.cfm?item_id=54849 

for  Fundamental  Rights. 

227 EC 2014: Joint Report on the application of Racial Equality Directive and of the Employment Equality Directive 
https://eur-lex.europa.eu/legal-

Com(2014)2, 

Brussels 
— 
content/EN/TXT/PDF/?uri=CELEX:52014DC0002&from=EN 

228 HU, CZ and SK. 
229 The Code of Conduct on countering illegal hate speech online and the High Level Group on combating racism and 

xenophobia and other forms of intolerance. 

230 OPC, p. 18. 
231 Council Regulation (EU) No 1303/2013 and Council Regulation (EU) No 1304/2013 of 17 December 2013. 

41 

 
 
 
 
 
                                                           
  The  thematic  ex  ante  conditionality 232  ‘a  national  Roma  inclusion  strategic 
policy  framework  is  in  place’  was  linked  to  the  above  IP  and  three  European 
Regional  Development  Fund  (ERDF)  IPs  dealing  with  investment  in  health  and 
social  infrastructure,  support  for  regeneration  of  deprived  communities  and 
investment in education and training infrastructure233. The fulfilment criteria234 
for  this  thematic  ex  ante  conditionality  were  fully  aligned  with  the  EU 
framework’s objectives. 

  In 2011 and 2015, the Commission issued thematic guidance on the use of ESIF 
in tackling educational and spatial segregation235. For its internal organisation, it 
then  prepared  in  2014,  thematic  guidance  for  ESF  desk  officers  on  how  to 
operationalise the objectives of the EU framework for the ESI funds236. 

These  measures  have  been  considered  to  be  an  'element  of  progress'237,  'a  promising 
practice' 238,  as  'having  provided  additional  opportunities  to  align  Roma  needs  with 
mainstream  policies'239 and  as  being  'instrumental  in  mobilising  public  administrations 
to  work  together  and  connecting  Roma  integration  goals  to  mainstream  policies'240.  In 
total, 12 Member States selected this IP (AT, BE, BG, CZ, ES, FR, EL, HU, IT, PL, RO 
and SK) allocating a total of EUR 1.5 billion, or 3.6 % of ESF funding. Across all ESIF, 
six  Member  States  reported  action  plans  related  to  the  Roma-specific  ex  ante 
conditionality (BG, EL, HU, LT, RO and SK) in their partnership agreements. Hungary 
and  Romania  succeeded  in  meeting  the  conditions  between  the  submission  of  the 
partnership agreement and the OP. Examples of actions related to the Roma strategy are: 
(i)  developing  strategies  (e.g.  RO)  or  action  plans  (e.g.  LT);  (ii)  adopting  revised 
strategies  (e.g.  HU);  and  (iii)  developing  an  adequate  monitoring  system  (e.g.  EL,  CZ 
and BG241). 

232 In total there are 36 ex ante conditionalities; their overarching aim is to help achieve an effective use of EU funds. 

Ex ante conditionalities are preconditions that Member States must fulfil when submitting their programming 
documents. If a Member State did not fulfil an ex ante conditionality, it had to present an action plan and a 
timetable for its implementation to the Commission. 
233 Regulation (EU) No 1303/2013 of 17 December 2013. 
234 These fulfilment criteria are: ‘It sets achievable national goals for Roma integration to bridge the gap with the 

general population. These targets should address the four EU Roma integration goals relating to access to 
education, employment, healthcare and housing. It identifies disadvantaged micro-regions or segregated 
neighbourhoods where communities are most deprived. It includes strong monitoring methods to evaluate the 
impact of Roma integration actions and a review mechanism for the adaptation of the strategy. It is designed, 
implemented and monitored in close cooperation and continuous dialogue with Roma civil society, regional and 
local authorities.’. 

235 EC  2015:  Guidance  for  Member  States  on  the  use  of  European  Structural  and  Investment  Funds  in  tackling 
educational  and  spatial  segregation,  Brussels  ;  EC  2011:  Guidance  note  on  the  implementation  of  integrated 
housing interventions in favour of marginalised communities under the ERDF, Brussels;. 

236  This guidance also stressed the partnership principle in ESIF and that representatives of marginalised groups 
should be involved in the partnership organised by Member States around partnership agreements and 
operational programmes. 

237 EURoma 2014. Promoting the Use of Structural Funds for Roma Inclusion in the European Union — A Glance at 

EURoma’s 8 years of work and how Roma inclusion is considered in the 2014-2020 programming period, 2014. 
238 Centre  for  European  Policy  Studies,  Research  Report:  Combating  Institutional  Antigypsyism.  Responses  and 

promising practices in the EU and selected Member States, No 2017/08, 2017,. 

239 EURoma 2014. Promoting the Use of Structural Funds for Roma Inclusion in the European Union — A Glance at 
EURoma’s 8 years of work and how Roma inclusion Is considered in the 2014-2020 programming period, 2014. 
240 Roma Civil Society Monitoring Initiative (RCM): Summary of findings year 1 country reports; unpublished; April 

2018. 

241 The implementation of the provisions in relation to the ex ante conditionalities during the programming phase of 

the European Structural and Investment (ESI) funds’, DG REGIO, 2016. 

42 

 
 
 
 
                                                           
5.5.3 National level 

At national level, limited progress has been made overall. 

Table 9: Coherence at national level (partly based on ICF, Chapter 3) 

National Policies 

Coherent 

Partial 

Not coherent 

Comment 

Mainstreaming 

       X 

The  EU 
framework  and 
NRIS  have  contributed  to 
mainstreaming 
Roma 
integration into policies to a 
limited extent.  

  X 

  X 

National funding 

Fighting 
discrimination and 
racism legislation 

Mainstreaming 

National  funding  has  been 
the 
allocated 
implementation 
the 
NRIS,  complementing  EU 
funding.  

to 

of 

NRIS  have  contributed  to 
fighting discrimination, hate 
speech  and  hate  crime  in 
some  countries,  while  their 
contribution  was  minor  or 
non-existent in others.  

The  evaluation  finds  that  the  EU  framework  has  contributed  to  incorporating  Roma 
integration into all policies to a limited extent at national level. Evidence of coherence is 
particularly  limited  in  the  health  sector.  In  terms  of  employment,  objectives  were 
generally part of mainstream policies, but often without specific targeted measures or a 
specific reference to Roma inclusion into the labour market. On housing, Roma inclusion 
measures  have  been  short-term  actions  rather  than  longer-term  programmes.  There  is 
some  reluctance  to  include  specific  references  to  Roma  and  their  specific  needs  into 
mainstream  policies  to  avoid  singling  out  a  specific  vulnerable  group 242.  Thematic 
policies often address vulnerable or disadvantaged groups in general, without referring to 
the particular needs of specific groups, including Roma. Indeed, a large share of the OPC 
respondents (88.9 %) identified the lack of effective inclusion of Roma in policies as one 
of the main challenges at national level. 

National funding 

In  the  11  countries 243  analysed  in  depth  for  this  evaluation,  national  funding  was 
allocated  for  implementing  the  NRIS,  in  addition  to  or  topping  up  EU  funding.  In  the 
enlargement  countries,  all  but  Turkey244 have  allocated  specific  funding  (national/EU) 
to implement the strategy. In some countries, a consistent approach has developed over 
time  between  NRIS  and  funding  mechanisms  at  national  level,  either  through  co-
financing  or  an  increase  in  national  budget  allocations  for  Roma  integration  measures. 
Quantification of amounts is difficult despite Member States providing the Commission 
with  information  on  the  implementation  of  their  NRIS,  including  financial  data,  since 

242 CS-ES, CS-AT, CS-BG. 
243 AT, BG, CZ, EL, ES, FI, FR, IT, HU, RO, SK. 
244 Turkey has allocated funds from the annual budgets of the relevant ministries to implement some of the actions 

provided for in the national action plan. 

43 

 
 
 
 
 
 
 
 
 
 
                                                           
2015,  through  the  Roma  integration  measures  reporting  tool.  The  information  is 
incomplete due to the limitations of the tool explained in Annex 4. 

Fighting discrimination and racism 

The  evaluation  finds  that  the  NRISs  have  contributed  to  fighting  discrimination,  hate 
speech  and  hate  crime  against  Roma  in  some  Member  States,  while  in  others  their 
contribution was minor or non-existent. Overall, combating discrimination has not been 
at the core of the NRIS. The NGO survey respondents considered that the NRIS of their 
respective  Member  State  contributed  to  some  extent  to  reducing  discrimination  against 
Roma (40 %), reducing hate speech (50 %), reducing hate crimes (53 %) and improving 
the enforcement of anti-discrimination legislation (44 %) during 2011-2017245. Targets 
and  measures  to  combat  trafficking  in  human  beings  are  included  in  the  NRIS  of  only 
two  of  the  countries  researched  for  this  evaluation  (BG  and  HU).  However,  other 
countries  have  taken  measures  against  trafficking  in  Roma  people  outside  the  scope  of 
the NRIS. 

245 NGO survey report, p. 15. 

44 

 
 
 
 
                                                           
5.6. EFFICIENCY 

The  analysis  of  efficiency  assesses  the  relationship  between  the  resources  used  by  an 
intervention and the changes generated by it. The efficiency of the EU framework has 
been  assessed  as  limited  with  regard  to  monitoring  and  reporting  systems. 
Furthermore, costs and benefits could not be conclusively evaluated. 

Efficiency was analysed through five  evaluation  questions covering the following areas 
in particular: 

  efficiency  of  the  monitoring  and  reporting  system,  including  scope  for 

simplification; and 

  costs and benefits in the context of the EU framework and NRIS. 

5.6.1. Monitoring and reporting systems 

The EU framework called for a robust monitoring system to be set up based on detailed 
and complete data as well as benchmarks246 and announced that the Commission would 
report  annually  to  the  European  Parliament  and  to  the  Council  on  progress 247 .  It 
highlighted  the  importance  of  ‘Member  States  contributions  based  on  their  own 
monitoring  systems  of  Roma  integration’,  assuming  that  ‘the  in-depth  monitoring  by 
Member States  and stakeholders of the implementation  of NRIS is  a sound method for 
enhancing transparency and accountability in order to ensure the most effective impacts 
of Roma integration.’  

The  2013  Council  Recommendation  stressed  that  ‘Member  States  should  choose  their 
own  monitoring  methods,  including  appropriate  methods  for  any  data  collection,  and 
possible indicators.’248 It  recommended that Member States provide their NRCPs with 
‘an adequate mandate and resources appropriate to their role so that they can effectively 
coordinate the cross-sectoral monitoring of Roma integration policies with a view to their 
implementation,  while  respecting  the  division  of  responsibilities  within  each  Member 
State 249 ’.  Importantly,  the  Recommendation  introduced  a  more  explicit  reporting 
obligation  whereby  the  Member  States  should  communicate  to  the  Commission  any 
measures  taken  in  accordance  with  the  Recommendation  by  1 January  2016,  and 
thereafter  ‘any  new  measures  taken  on  an  annual  basis,  at  the  end  of  each  year,  along 
with  information  on  the  progress  achieved  in  implementing  their  NRIS 250 ’.  The 
Recommendation asked the Commission to ensure that the information provided by the 
Member  States  would  serve  as  a  basis  for  the  preparation  of  its  annual  reports  to  the 
European Parliament and to the Council251. 

The evaluation finds that monitoring and reporting systems have  gradually been set up, 
but  they  are  still  weak  and  need  improvement.  In  particular,  countries  have  monitored 
and  reported  on  process  indicators,  rather  than  results  and  outcome  indicators  which 

246 Chapter 8 of COM(2011) 173 final. 
247 Reporting by the Commission has been carried out since 2012 in the form of country-by-country and horizontal 
guidance. For the  enlargement countries, country-specific descriptions of the situation of the Roma were added 
in 2016. 

248 Recital (19) of 2013 Council Recommendation. 
249 Paragraph 3.8 of 2013 Council Recommendation (National Contact Points for Roma integration). 
250 Paragraph 4 of 2013 Council Recommendation (reporting and follow-up). 
251 Idem. 

45 

 
 
 
 
 
 
                                                           
would allow the overall effectiveness of the interventions to be measured. An evaluation 
commissioned  by  the  European  Parliament  in  2015252 noted  that  almost  none  of  the 
Member  States  provided  details  on  their  progress  in  implementing  the  strategy  and, 
wherever  references  were  made,  targets,  baselines  and  indicators  set  by  most  Member 
States were generally poor and the monitoring mechanisms weak. 

To  accompany  the  new  reporting  requirement  for  Member  States,  the  Commission 
introduced  a  new  online  reporting  tool  (based  on  the  work  of  the  FRA-coordinated 
working party on Roma integration indicators) to be used by the NRCPs as of 2016 for 
their annual  reporting253. The data collected in  the  online tool includes a description of 
targeted and mainstream policy measures implemented across 12 thematic areas set out 
in the Council Recommendation. The tool also gathers data related to those involved in 
implementation,  funding  and  beneficiaries.  The  evaluation  came  to  the  following  
conclusions: 

  The tool enables a variety of actors and levels of public administration working 
on Roma integration to collaborate online. This reportedly decreases the amount 
of  time  taken  by  NRCPs  to  collect  and  compile  data  from  those  involved,  and 
reduces the variety in the responses they receive. 

  There  is  insufficient  information  relating  to  the  pre-2016  system  to  allow  for  a 

comparison of efficiency. 

  NRCPs  reported  some  difficulties  including  in  particular:  (i)  unavailability  of 
Roma-specific  data  requested  by  the  tool;  and  (ii)  regional  municipal  and  local 
levels not feeding the tool appropriately. As a result, the information provided has 
limited validity. 

  NRCPs  reportedly attempted to  fill  data gaps with  information  from  small-scale 
evaluations  and  research  projects.  However,  this  was  considered  a  time-
consuming and inefficient way to collect data as it did not guarantee coverage at 
population level and comparability of data across Member States. 

  Some NRCPs also  found that the  reporting tool  relied too  much on quantitative 
information,  whereas  additional  qualitative  and  contextual  information  was  also 
considered to be important. 

In  addition  to  the  EU  framework’s  specific  reporting  requirements,  other  related  EU 
instruments have reporting requirements. Examples include ESIF reporting254, reporting 
in  the  context  of  the  European  Semester  and  implementation  reports  for  Directive 
2000/43/EC255. Member States also have a national reporting requirement linked to the 
implementation  of  the  NRIS.  As  in  some  Member  States  the  NRIS  is  broader  in  scope 
than  the  EU  framework,  the  national  implementation  reports  are  sometimes  more 
comprehensive than the reports submitted to the Commission. The evaluation observes a 
general absence of coordination of reporting times and formats. 

With  regard  to  monitoring  systems,  as  demonstrated  in  the  ‘coherence’  section,  the 
analysis and data collection by FRA is a positive development at EU level.   At national 

252  European  Parliament,  Evaluation  of  the  EU  framework  for  national  Roma  integration  strategies,  2015, 

http://www.lse.ac.uk/business-and-consultancy/consulting/assets/documents/evaluation-of-the-EU-Framework-
for-National-Roma-Integration-Strategies.pdf.  The  evaluation  focused  on  the  following  countries:  Bulgaria, 
France, Hungary, Italy, Romania, Slovakia and Spain. 

253 For the 2015 reporting year. 
254 In SFC. 
255 Council Directive 2000/43/EC of 29 June 2000. 

46 

 
 
 
 
                                                           
level,  the  EU  framework  does  not  provide  any  specific  guidelines  on  the  kind  of 
monitoring system that should be put in place. The national systems serve the purpose of 
monitoring  NRIS,  but  their  use  may  be  limited  in  terms  of  comparability  or  assessing 
progress  on  the  EU  framework  objectives  in  cases  of  limited  alignment  between  NRIS 
and the EU framework256. 

During  the  stakeholder  interviews,  those  involved  in  reporting  to  the  EU  expressed  the 
opinion  that  there  could  be  a  reduction  in  the  administrative  costs257.  Nevertheless, 
substantiating these claims with comprehensive evidence on the actual costs (in terms of 
full-time  equivalents)  has  not  been  possible  as:  (i)  only  nine  NRCPs258 were  able  to 
provide the number of staff and/or dedicate the amount of time required to complete the 
reporting and monitoring tasks; and (ii) the key difficulty in estimating the time involved 
in  the  reporting  process  is  not  assessing  the  time  taken  by  individuals  to  complete  the 
online reporting tool  forms259, but  the time it takes to  collect the  required information 
and the time spent by other public servants not in directly reporting, but in summarising 
and presenting the information to be included in the report. 

According to the NRCPs, the main areas for improvement are: 

 

the  timelines  for  EU  reporting  could  be  better  aligned  with  Member  States’ 
reporting deadlines; 

to  supplement 

  given  that  several  Member  States  do  not  have  Roma-specific  data,  the  efforts 
information  are 

required 
disproportionate, without clear added value to the monitoring process; 
the  frequency  and  level  of  reporting  could  be  linked  to  the  size  of  the  Roma 
population  in  the  Member  State,  meaning  that  Member  States  with  small  Roma 
populations and few policies could report less frequently; 

reporting  with  qualitative 

the 

 

  a  ‘materiality’  threshold  that  eliminates  reporting  on  small  projects  that  do  not 
contribute  significantly  to  the  objectives  of  the  EU  framework  could  be 
introduced, setting  a different format or frequency for small projects whose only 
added value is the ‘exchange of experience’; and 
reducing reporting frequency or replacing annual reporting with a strategic cycle 
reporting  would  reduce  the  administrative  burden  of  NRCPs  and  be  more 
appropriate  considering  that  the  vast  majority  of  NRIS  policies  are  long-term 
ones. For example, a staggered reporting model  could  provide more meaningful 
and comparable data. 

 

5.6.2 Costs and benefits in the context of the EU framework 

It was not possible to conclusively evaluate costs and benefits for a number of reasons. 
Firstly, most known or potential costs and benefits cannot exclusively be attributed to the 
EU  framework  or  they  are  not  quantifiable.  Secondly,  where  costs  are  quantifiable  in 
principle  (for  example  in  the  case  of  ESIF  investment),  information  about  the  share  of 

256 See the chapter on effectiveness, in particular alignment between the EU framework and NRIS. 
257 AT, CY, CZ, DE, EE, EL, FI, FR, HR, IE, LT, NL, PT, SE, SI, UK. Additionally, a West Balkan regional body 

and Bosnia and Herzegovina. 
258 CY, CZ, EL, ES, FI, HR, LT, LV, SI. 
259 This could be assessed fairly straightforwardly by examining the number of reported measures (939 in 2016, and 

937 in 2015) and trying to assess the average time to complete the information on one measure. 

47 

 
 
 
 
 
                                                           
funding spent exclusively on Roma is not sufficiently available. This is particularly due 
to  the  absence  of  Roma-specific  indicators  because  of  the  broad  consensus  that  the 
correct approach is to target funding at Roma explicitly, but not exclusively260 (i.e. not 
excluding  non-Roma  in  a  similar  situation).  Lastly,  output  values  in  terms  of  Roma 
beneficiaries and participants are not systematically available, also because of the lack of 
specific indicators. 

With  regard  to  benefits,  in  the  longer  term,  progress  made  in  education,  employment, 
housing and health can potentially not only further reduce poverty among Roma, but is 
also likely to have fiscal benefits such as contributions to national budgets (increased tax 
payment, social security, indirect taxes, such as VAT or excise tax261) or impact the use 
of public goods and services (reduced take-up of social welfare, unemployment or child 
support  benefits).  The  integration  of  Roma  could  be  positive  for  the  labour  market 
(improved  efficiency  through  a  decrease  in  labour  and  skills  shortages 262 )  and  the 
economy263, in particular GDP (albeit with a significant delay to labour gains stemming 
e.g. from educational progress or desegregating housing policies).  

In the light of the above limitations, this evaluation: 

i) 

ii) 

iii) 

assesses  those  costs  and  benefits  that  are  directly  attributable  to  the  EU 
framework (see below); 
provides information about quantifiable benefits gained so far through having 
a  higher  share  of  Roma  in  paid  employment  and  a  lower  share  of  Roma 
NEETs where applicable (as these changes are not directly attributable, they 
are not presented here but in Annex 5); and 
does  not  assess  the  extent  to  which  EU  and  national  funding  in  the  area  of 
Roma integration is justified and proportionate to the long-term benefits from 
Roma  integration  in  quality  inclusive  mainstream  education,  employment, 
healthcare and housing (original evaluation question264). 

With regard to the costs and benefits identified and attributable to the EU framework, the 
evaluation finds that there are administrative  costs (staff) for both the EU and Member 
States:  

  At  EU  level  the  administrative  costs  relate  in  particular  to  the  Commission’s 
Roma team with its current staffing of six full-time equivalents. Additional input 
and  contributions  are  made  via  the  Commission’s  internal  Roma  task  force 
involving a number of Commission departments.  

  Other costs directly stemming from the implementation of the EU framework at 
EU level are estimated to be around EUR 800 000 per year, notably covering the 

260 In line with the common basic principles on Roma inclusion already agreed in 2009 by the Council: 

http://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/lsa/108377.pdf 

261 Kertesi, Gábor, and Gábor Kézdi. 2006. Expected long-term budgetary benefits to Roma education in Hungary. 

No. BWP-2006/5. Budapest working papers on the labour market, 
http://www.romaeducationfund.org/sites/default/files/publications/kertesi-kezdi-budgetarybenefits.pdf 
262 de Laat, J., Bodewig, C., Arnhold, N., Linden, T., Dulitzky, D., Kosko, S. and Torracchi, T., 2010. Roma 

Inclusion: An Economic Opportunity for Bulgaria, Czech Republic, Romania and Serbia. Policy note, World 
Bank, Human Development Sector Unit, Washington, DC. 

263 Marcinčin, A. and Marcinčinová, L., 2009. The Cost of Non-inclusion. The key to integration is respect for 
diversity. http://childhub.org/sites/default/files/library/attachments/the_cost_of_non_inclusion.pdf 

264 The question was replied to by the external study; this evaluation finds that data limitations are too strong for the 

findings to be reliable. 

48 

 
 
 
 
 
 
 
                                                           
organisation  of    bi-annual  NRCP  meetings  and  annual  EU  Roma  platform 
meetings,  co-financing  national  Roma  platforms  and  development  and 
maintenance of the reporting tool.  

  At national level, administrative costs are estimated at around  EUR 3.3 million 
annually265, covering staff required to implement the EU framework and report 
on the NRIS.  

In  terms  of  benefits,  the  evaluation  found  that  the  EU  framework  provides  EU  added 
value (see section on EU added value) — such as: (i) putting Roma inclusion on the EU 
and  national  political  agendas;  (ii)  raising  awareness  of  Roma  issues  including  in 
countries with smaller Roma populations; and (iii) ensuring a continuous focus on Roma 
integration through its multiannual character. This EU added value brings a benefit to the 
EU as it is being recognised as promoting equality and Roma integration. The evaluation 
also  identifies  benefits  for  EU,  national  and  local  stakeholders  working  on  Roma 
integration. It found that increased cooperation between and capacity of stakeholders, in 
particular  through  the  setting  up  and  strengthening  of  instruments  and  structures  for 
Roma integration, but also through their increased awareness of Roma issues. Increased 
awareness  and  cooperation, including  with  Roma civil  society, can in  turn improve the 
efficiency of policies. 

Other  costs  and  benefits  of  Roma  integration  are  not  directly  or  exclusively 
attributable to the EU framework or not yet measurable because they are long-term 
costs and benefits. These are presented in detail Annex 5. 

265 See information about calculations in Annex 5. 

49 

 
 
 
 
 
 
                                                           
5.7. EU ADDED VALUE 

Analysis of EU added value looks at  changes triggered by the EU framework over and 
above  what  could  reasonably  have  been  expected  from  stakeholders  alone  or  from  no 
action  at  all.  The  evaluation  highlights  the  positive  EU  added  value  of  the 
framework266. 

The assessment of the EU added value of the EU framework has been informed by four 
evaluation questions covering the extent to which: 

  EU  action  is  necessary  to  stimulate,  complement,  leverage  and  create  synergies 

with national action;  

  EU action in areas such as policy guidance and knowledge exchange bring added 

value; and 
there would be consequences of stopping targeted EU action. 

 

This  evaluation  finds  that  EU  action  is  necessary  and  has  provided  added  value  to 
national  Roma  policies  and  their  implementation  at  the  political  level  and  in  terms  of 
governance and financial support. 

At  political  level,  the  EU  framework  has  stimulated  political  action  and  encouraged 
political  will  to  improve  Roma  integration.  In  some  countries,  Roma  integration  has 
found its place on the national political agenda thanks to the steer from the EU political 
agenda267.  The  framework  also  raised  attention  on  Roma  issues  in  countries  with  a 
smaller  Roma  population,  mainly  in  northern  and  western  European  countries,  and  in 
countries with a Roma population including high shares of EU-mobile and third-country 
Roma.  In  this  sense  it  laid  the  foundation  of  an  incipient  European  agenda.  In  those 
countries with larger Roma populations or previous engagement in the Roma Decade268, 
the  EU  framework  strengthened  existing  structures.  The  perspective  of  a  multiannual 
policy  framework  ensures  a  certain  stability  at  national  level,  especially  in  the  face  of 
changing  political  priorities269.  More  than  three  quarters  of  the  242  OPC  respondents 
recognised that thanks to the EU framework Roma inclusion was higher on the EU policy 
agenda and that more funding was available for projects promoting Roma inclusion270. 
The view that the EU framework has been ‘the key driver forcing national governments 
to act’271 is confirmed by civil society. Many NGO participants in the workshop stated 
that  none  of  the  Member  States  would  be  where  they  are  now  without  the  EU 
framework272.  This was also confirmed in a number of  — but not all — national-level 
interviews, pointing to the importance of EU action in fostering political ownership and 
will to act on Roma inclusion273. 

266 In particular for those countries having a sizeable Roma population. 
267 See ICF, Chapter 3.7.1 for country-specific examples. 
268 AL, BiH, BG, CZ, ES, HU, HR, former Yugoslav Republic of Macedonia, ME, RO, RS, SK. 
269 Confirmed for example in CS-SK, ‘added value’ section. 
270 When broken down by respondent type, the response trends (public institutions, NGOs/think tanks, citizens) or and 

country cluster (EU-15, EU-13, enlargement) were overall consistent with the overall response trends. 
271 Open Society European Policy Institute, Report: Revisiting the EU Roma Framework: Assessing the European 

Dimension for the Post-2020 Future 2017, https://www.opensocietyfoundations.org/reports/revisiting-eu-roma-
framework-assessing-european-dimension-post-2020-future. 

272 In  the  workshop  discussion,  97 %  of  all  participants  agreed  that  there  is  EU  added  value  with  regard  to  Roma 

inclusion in the Member States. 

273 BE1, BG6, BG8, DE1, FR2, HR1, IE1, LT1, NL1. 

50 

 
 
 
 
 
 
                                                           
In terms of governance, the EU framework: (i) provided policy guidance: (ii) supported 
greater  accountability  and  transparency;  and  (iii)  created  opportunities  for  mutual 
exchange and coordination, in particular through the meetings of the NRCP network and 
the  EU  Roma  platform.  The  framework  encouraged  a  move  away  from  a 
compartmentalised approach to an integrated approach covering all core policy areas274. 
The  framework’s  more  comprehensive  approach  allowed  Roma  integration  to  be 
addressed from various policy perspectives and for central, regional and local authorities 
to  work  together  on  Roma  issues,  thus  helping  to  spread  ownership  of  the  Roma 
inclusion issue275. The EU framework also encouraged Roma representatives to become 
more  actively  involved  in  the  policy  and  decision-making  structures  directly  affecting 
Roma276. Monitoring and reporting was highlighted as being important to regularly take 
stock  of  progress  and  evaluate  which  measures  are  working  better  than  others277.  The 
monitoring and reporting system also allows easier comparisons with other EU countries, 
giving a certain competitive and comparative aspect to the process278. 

At the financial level, EU added value was created through the close link made between 
the  European  Semester,  ESIF  in  the  2014-2020  programming  period  and  NRIS  (see 
section on coherence). In many countries, much of the funding for Roma integration has 
indeed  been  ESIF  or  IPA  funding,  while  there  seems  to  be  more  reticence  to  invest 
domestic money for Roma integration, at  least  beyond national  co-funding required for 
ESIF.  However,  this  is difficult  to  assess  due  to  the  limitations  set  out  above.  In  many 
cases, ESIF helped scale up existing projects, intensify actions and improve the quality of 
interventions279. 

Consequences of stopping targeted EU action 

The evaluation finds280 that stopping targeted EU action on Roma integration is likely to 
result in reduced political commitment and focus on Roma integration in both the EU and 
enlargement  countries  and  a  significant  decrease  in  available  funding.  Stopping  EU 

274 CS-AT, CS-ES, CS-IT. 
275 CS-ES, CS-IT. 
276 For  example,  in  Spain  the  EU  framework  has  reinforced  the  participation  of  the  Roma  population  through  the 

National Roma Council. 

277 BE5, FR3, LT1, EU-level interviews EU1, EU2. 
278 IE2, LV1, PT3, SE3, WB3. 
279 For example, in Austria EU action has allowed for targeted projects to be implemented at a (monetary) scale that 
far exceeded previous targeted projects, thus allowing for projects to be scaled up and greater numbers of Roma 
to be reached.  In Bulgaria, the scale of the programmes, especially in the areas of education and employment, is 
unlikely to have been the same as the one enabled via the EU funds. In Finland also, the measures could not have 
been  accomplished  by  Finland  alone,  without  the  support  of  EU  structural  funds,  with  the  exception  of  the 
education measures carried out using Finnish State funds. Similarly in Greece, almost all measures in the areas of 
education,  employment,  housing,  health  and  anti-discrimination  (excluding  those  aimed  at  combating  poverty) 
are  EU  co-funded  projects,  the  implementation  of  which  would  be  difficult  if  these  funds  were  not  available, 
especially during a period of economic crisis and  structural and financial adjustments. In Italy, without the EU 
framework there would be no NRIS, and without the NRIS the situation would be much more similar to the one 
seen in 2009 (IT-2, IT-3). Source: country analysis studies, ‘added value’ section. However, these views could 
not be quantitatively confirmed due to an overall lack of robust data on ESIF funding. 

280 ICF, Chapter 3.7.4 underlines that this overall conclusion was confirmed by evidence gathered though the case 

studies, stakeholder workshop discussion, national- and EU-level interviews and the NGO survey. 

51 

 
 
 
 
 
 
                                                           
action is also likely to result in less and looser monitoring and reporting of the situation 
of  Roma.  Less  policy  focus,  funding  and  monitoring  is  likely  to  worsen  the  living  and 
working conditions of Roma, not only stopping but also potentially reversing the current 
trend towards an improvement of the Roma situation. 

For  a  majority  of  the  OPC  respondents  (almost  60 %),  the  EU  still  has  a  major  role  to 
play  in  supporting  national,  regional  and  local  authorities  as  they  cannot  effectively 
improve  the  situation  of  Roma  on  their  own.  This  opinion  is  shared  by  the  public 
institutions, NGOs/think tanks and citizens who expressed their views. The main reasons 
provided  relate  to  the  perceived  role  of  EU  funding  programmes 281  and  to  the 
importance of monitoring the Roma situation as a critical driver for reform and political 
commitment  at  national  level.  National-level  interviews  and  country  analysis  studies 
showed that political focus on Roma inclusion risks being reduced in the absence of an 
EU  requirement  for  reporting  and  monitoring282.  Another  possible  consequence  would 
be  that  Roma  inclusion  could  become  more  dependent  on  national  political  parties  in 
government and their political priorities. Interviews and country analysis studies confirm 
the likely  consequences  of discontinuing  EU  action in  terms  of fewer measures  funded 
and less focus on Roma inclusion283. There is a risk that stopping EU action could result 
in  a  less  comprehensive  and  coordinated  approach  to  Roma  inclusion,  narrowing  the 
scope,  reaching  fewer  beneficiaries  and  moving  backwards  towards  small-scale  and 
short-term projects and initiatives 284. 

5.8 SUSTAINABILITY 

The analysis of sustainability assesses the extent to which effects stemming so far from 
the  EU  framework  are  likely  to  last  after  it  ends.  This  separate  evaluation  criterion  is 
intended to feed into a potential post-2020 initiative. 

The  sustainability  of  the  EU  framework  is  assessed  as  limited  overall.  The 
assessment has been informed by four evaluation questions that looked into the extent to 
which: 

  measures  have  been  taken  at  EU  and  national  level  to  ensure  continuity  and 

sustainability; and 

  outcomes are expected to continue after 2020. 

The evaluation finds that, at present, the effects of the EU framework are unlikely 
to last after 2020. Most of the current national policies and structures created around the 
NRIS  (NRCP,  systems  of  coordination,  monitoring  and  reporting,  systems  of 
consultation  with  civil  society  and  Roma  organisations)  would  stop  or  would  be  less 
operational  and  become  more  symbolic  if  the  EU  framework  did  not  continue  after 
2020285. A longer duration is needed to ensure the sustainability and long-term impact of 
policies  and  to  consolidate  working  structures.  At  national  level,  substantial  and 
continued  political  commitment,  public  action  and  funding  of  NRIS  are  still  needed  to 

281 The important role of EU funding programmes was confirmed by 60 % of the respondents of the NGO survey, 

who highlighted a lack of sufficient funding at national level. 

282 DE3, FR6, IE4, NL1, SI1. 
283 CZ4, AT11, ES2, ES10, EUI1, FI2, HR1, IE4, LT1, PT4, SK8, RO1, UK1. 
284 CZ9, CZ4, AT11, ES2, ES10, EUI1, FI2, HR1, IE4, LT1, PT4, SK8, RO1, UK1. BE4, ES1. 
285 ICF, Chapter 8, as documented in country analysis studies and interviews. 

52 

 
 
 
 
 
                                                           
maintain  the  improvements  in  integration  and  lower  the  risk  of  some  programmes 
disappearing286. 

The  evaluation  was  based  on  three  main  prerequisites  for  sustainability:  (i)  sustainable 
funding;  (ii)  incorporating  Roma  integration  into  policy-making;  and  (iii)  community 
engagement (involvement of Roma, NGOs, regional and local authorities). 

Sustainable funding 

The  EU  framework  stated  that  ‘Member  States  should  allocate  sufficient  funding  from 
national  budgets,  which  can  be  supplemented,  where  appropriate,  with  EU  and 
international funding’287. 

The  evaluation  notes  that  funding  is  not  yet  sustainable.  Many  stakeholders  questioned 
the  extent  to  which  resources  allocated  to  support  the  implementation  of  NRIS  are 
sustainable in the long-term288.  The framework encouraged the use of national funding 
for  implementing  measures  supporting  Roma  integration,  but  Member  States  and 
enlargement countries still largely relied on European funding to implement the NRIS289.  

in 

integration  from  ESIF  would  result 

In  particular,  phasing  out  Roma 
the 
discontinuation  of  many  programmes,  as  many  of  the  new  initiatives  addressing  Roma 
inclusion  have  been  developed  due  to  the  EU  framework  and  its  support  by  ESIF, 
especially  the  ESF.  In  the  countries  where  the  majority  of  Roma  are  living,  many 
programmes and projects are highly dependent on the ESIF fundign, with many receiving 
modest  co-funding  from  national  budgets  (around  20%).  Consequently,  much  of  the 
action  consists  of  transitory,  one-off  projects  rather  than  long-term  and  sustainable 
programmes.  Changes  in  political  leadership,  new  priorities  or  budgetary  constraints 
could put the sustainability of such projects at risk if not supported by an EU framework. 
Indeed,  many  programmes  and  projects  initiated  prior  to  the  EU  framework  were  only 
short-term and relied on annual extensions, with interruptions in some cases290. 

Mainstreaming Roma integration 

To incorporate Roma integration into general education, employment, housing and health 
programmes, Roma need  to  benefit  from general  policies. This  includes removing their 
specific  barriers  in  gaining  access  to  public  services.  A  policy  of  inclusive  mainstream 
services can help ensure that measures continue in the future (even if specific strategies 
on  Roma  inclusion  are  discontinued).  Projects  with  a  mainstream  approach  are  more 
likely to be sustainable than targeted ones. Regular monitoring and periodic evaluations 
are  the  main  means  available  to  ensure  that  sustainability  is  checked  at  policy 
implementation  level.  While  in  principle  Member  States  have  reporting  obligations  on 
the implementation of the EU framework, in practice there are no mechanisms in place to 

286 Open Society European Policy Institute 2017: Revisiting the EU Roma Framework: Assessing the European 

Dimension for the Post-2020 Future, Budapest/Brussels. 

287 COM(2011) 0173 final. 
288 ICF, Chapter 3.8.2; this was stressed by several interviewees, e.g. BE-3; BE-5; DK1; ES1; ES10; SK5 and in 

country analysis studies AT, BG, FI, FR, EL. 

289 Open Society European Policy Institute, Report. Revisiting the EU Roma framework: Assessing the European 

dimension for the post-2020 future, 2017,. 

290 See the successful programming principles outlined in EC 2012: What works for Roma inclusion in the EU — 

Policies and model approaches, Brussels. 

53 

 
 
                                                           
ensure  a  sustainability  check  at  policy  implementation  level  because  of  data  and 
information gaps as detailed in the reporting and monitoring section above. 

The  evaluation  finds  that  insufficient  measures  have  been  taken  so  far  to  ensure  that 
Roma  integration  is  included  in  mainstream  measures  in  the  four  key  areas.  Positive 
outcomes  may  be  long-lasting  in  education,  where  inclusive  reform  of  mainstream 
policies have been initiated291. However, long-term benefits in employment, health and 
housing are more questionable as initiatives are usually short-term, ad hoc and have not 
brought the institutional or cultural changes required to be sustainable292. 

Community engagement 

Effective  community  engagement  can  help  to  identify  funding  priorities293,  empower 
local  communities,  provide  critical  feedback  and  increase  accountability  for  Roma 
inclusion  policies.  However,  to  date,  not  enough  effort  has  been  made  to  provide 
sustainable support for building the capacity of Roma grassroots organisations — a factor 
that in practice is reducing their participation294. 

291 EPRD, 2015 ‘Thematic Evaluation on IPA Support to Roma Communities’:. 
292 Ibid.; ICF, Chapter 3.8 and country analysis studies BG, FR, EL. 
293 IES (2014) The Missing Piece: Empowerment of Roma Grassroots Organisations in EU Roma Integration 

Policies. 

294 ERIO (2017), Position paper on the EU Framework for National Roma Integration Strategies and a post-2020 

strategy as a contribution to the Mid-term review of the European Commission; Open Society European Policy 
Institute (2017), Revisiting the EU Roma Framework: Assessing the European Dimension for the Post-2020 
Future. To be noted that the Commission financially supports the European Roma Grassroot Network. 

54 

 
 
                                                           
6.  CONCLUSIONS  

Effectiveness of the EU framework has been assessed as limited regarding progress 
towards  the  Roma  integration  goals.  While  there  is  some  progress  in  the  area  of 
education,  progress  was  limited  in  the  area  of  health  and  lacking  in  the  areas  of 
employment  and  housing.  Overall,  attributing  the  progress/lack  of  progress  to  the  EU 
framework  is  difficult  in  light  of  external  factors  such  as  the  economic  and  financial 
crisis  or the 2015 refugee crisis  and  given the distribution of competences  between the 
EU and the Member States. Also, structural changes require time and the real impact of 
progress (such as positive initial changes in the area of education) will only be seen later. 
Monitoring progress in Roma integration continuous to be hampered by a lack of reliable 
data  running  over  time.  The  EU  framework's  objectives  could  have  been  designed  in  a 
more  specific  and  measurable  manner  for  them  to  serve  as  both  political  signals  and 
quantifiable policy goals.  

The  EU  framework  has  been  more  effective  in  setting  up  and  strengthening 
instruments  and  structures  for  Roma  integration,  increasing  cooperation  and 
capacity of people and institutions working on Roma integration. This is particularly 
true at EU level, where coordination  has  been assessed as positive overall. At national 
level, an obstacle is the limited influence of the National Roma Contact Points (NRCPs) 
on the design and implementation of mainstream policies and decision-making processes 
across policies. Nevertheless, at both EU and national level the EU framework was found 
to have stepped up stakeholder cooperation. NRISs have systems of consultation in place 
with Roma civil society. However, the need remains for more effective, transparent and 
inclusive Roma engagement and participation at all levels295. 

Relevance  of  the  EU  framework’s  original  objectives  in  view  of  the  original  and 
current  needs  has  been  assessed  as  overall  positive,  however  with  limitations. The 
framework was and remains relevant regarding the needs of Roma, and also in terms of 
social  and  economic  needs  due  to  increasing  labour  shortages  in  countries  with  the 
largest  shares  of  Roma.  However,  some  limitations  were  identified,  namely:  (i)  the 
relatively  low  relevance  and  narrow  focus  of  the  Roma  integration  goal  on  education, 
with  its  emphasis  on  completing  primary  school;  (ii)  the  absence  of  a  specific  non-
discrimination goal alongside the four Roma integration goals and insufficient attention 
to antigypsyism; (iii) the limited attention to the specific disadvantages of Roma women, 
children and  youth and other groups within the Roma population, including EU-mobile 
Roma and Roma who are non-EU nationals (equity).  

Overall,  while  respecting  Member  State  competences,  the  EU  framework  provides  the 
flexibility296 to adapt its objectives to specific national contexts and to changes that take 
place over time. This suggests that the relevance of the EU framework hinges closely on 
the appropriateness and ambition of the concrete objectives and measures set out in the 
NRISs.  The  flexibility  and  the  non-binding  nature  of  the  EU  framework  have  allowed 
Member States to follow a tailored approach taking account of specific national contexts. 
While such an approach has the potential to make actions more relevant, the evaluation 

295 Roma Civil Monitor 2018: Synthesis Report on the Implementation of the EU Framework for National Roma 

Integration Strategies in Bulgaria, Czech Republic, Hungary, Romania and Slovakia. Brussels / Budapest. 
296 See Section 2, description of the initiative; in particular: Designing NRIS taking into account the size of the Roma 

population and different starting points; also non-binding approach to measures to be implemented by Member 
States. 

55 

 
 
 
 
 
 
 
                                                           
also found that it contributed to a generally fragmented implementation and incomplete 
alignment between the Roma integration goals and the NRIS. 

Coherence  has  been  assessed  as  generally  positive  at  EU  level  and  as  limited  at 
national  level.  At  EU  level,  progress  has  been  made  in  aligning  and  mobilising  legal, 
policy and financial instruments for the support of Roma integration. EU funding such as 
ESIF,  the  Europe  2020  strategy  and  the  use  of  CSRs  were  found  to  be  particularly 
instrumental to achieve the EU framework's objectives. Other areas of coherence include 
the European enlargement policy, the EU urban agenda and EU anti-trafficking policies 
or recent proposals such as the recast Drinking Water Directive297. Roma integration is 
not explicitly included in key social policy initiatives such as the Youth Guarantee or the 
European  Pillar  of  Social  Rights.  The  evaluation  found  that  the  EU  framework  has 
contributed to incorporating Roma integration into legal, policy and financial instruments 
at  national  level  to  a  more  limited  extent,  and  this  across  Member  States  and  the 
enlargement countries. 

Efficiency  has  been  assessed  as  limited  with  regard  to  monitoring  and  reporting 
systems  and  inconclusive  on  the  costs  and  benefits,  mainly  due  to  limited  data 
availability.  Improvements  are  necessary  in  aligning  domestic  and  European  reporting 
requirements  as  well  as  coordination  between  the  EU  framework’s  reporting  times  and 
formats  and  related  reporting  such  as  for  ESIF  or  the  European  Semester.  The  online 
reporting  tool  provided  by  the  Commission  for  reporting  from  the  national  to  the 
European  level  is  a  positive  step,  but  several  weaknesses  (data  availability  to  feed  the 
tool;  cooperation  between  the  NRCPs  and  other  stakeholders  in  gathering  data; 
development of more meaningful indicators) need to be addressed. Existing comparable 
data (e.g. from the FRA) could be integrated into the tool to provide outcome and impact 
indicators.  Costs  and  benefits  could  not  be  conclusively  evaluated  for  a  number  of 
reasons, including: (i) most identified or potential costs and benefits cannot exclusively 
be attributed to the EU framework or are not quantifiable; (ii) where costs are in principle 
quantifiable (for example ESIF investment), information about the proportion of funding 
spent exclusively on Roma is not sufficiently available, this being in particular due to the 
absence  of  Roma-specific  indicators;  and  (iii)  output  values  in  terms  of  Roma 
beneficiaries and participants are not systematically available. 

EU added value has been assessed as positive. EU action has provided added value to 
national  Roma  policies  and  their  implementation  at  the  political  level  and  in  terms  of 
governance and financial support. In particular, EU action has: (i) put Roma inclusion on 
the  political  EU  and  national  agendas;  (ii)  raised  attention  to  Roma  issues  in  countries 
with  smaller  Roma  populations;  (iii)  strengthened  existing  structures  for  Roma 
integration and contributed to the creation of new ones; (iv) ensured stability through its 
multiannual  character;  (v)  provided  policy  guidance,  monitoring  and  reporting 
(vi)  provided 
frameworks  supporting  greater  accountability  and 
opportunities  for  mutual  exchange  and  cooperation;  (vii)  enabled  —  through  its 
comprehensive  approach  —  Roma  inclusion  to  be  addressed  from  various  policy 
perspectives as well as collaboration between different political and governmental levels; 
and viii) ensured that ESIF supports implementation of the NRIS. Stopping targeted EU 
action on Roma integration is likely to reduce political commitment and focus on Roma 
integration in both the EU Member States and enlargement countries and lead to a sharp 

transparency; 

297 COM(2017) 753 final. 

56 

 
 
 
 
 
                                                           
decrease  in  available  funding.  Stopping  EU  action  is  also  likely  to  result  in  less  and 
looser monitoring and reporting about the situation of Roma.At present, the effects of the 
EU  framework  are  unlikely  to  last  after  2020  if  there  is  no  further  EU  support 
(sustainability). Most of the current national policies and structures created around the 
NRIS  (NRCPs,  systems  of  coordination,  monitoring  and  reporting,  systems  of 
consultation  with  civil  society  and  Roma  organisations)  would  stop  or  would  be  less 
operational  and  become  more  symbolic,  if  the  EU  framework  did  not  continue  after 
2020 298 .  Increased  efforts  at  national  level  are  needed  as  well  as  more  time  to 
consolidate  working  structures,  to  further  align  and  mobilise  other  policy,  legal  and 
financial instruments and to better monitor the impact of policies. 

A number of the above lessons learnt can be addressed in the longer term but not in 
the  context  of  the  current  EU  framework,  its  objectives  and  its  set  up.  These  lessons 
learnt include in particular:  

  The  need  for  a  revision  of  the  framework’s  objectives  to  make  them  more 
specific, measurable and realistic while ambitious enough to bring about changes;  
  The  importance  of  enabling  Member  States  to  follow  a  more  tailored  approach 
taking  account  specific  national  contexts,  for  example  through  individually 
adaptable Roma integration objectives; 

  The  need  for  complementing  the  four  priority  areas  of  employment,  education, 
health  and  housing  with  a  specific  focus  on  fighting  non-discrimination  and 
antigypsyism;  

  The importance of addressing the limited attention to  the specific disadvantages 
of  Roma  women,  children  and  youth  and  other  groups  within  the  Roma 
population, including EU-mobile Roma and Roma who are non-EU nationals; 
  The need to strengthen the national Roma contact points in terms of mandate and 

capacity for mainstreaming Roma inclusion into all relevant policy areas.  

However, in  the  remaining  implementing  period  until  2020 the following corrective 
measures could be prioritised: 

  Some improvements to the monitoring and reporting systems associated with 
the  EU  framework  and  NRIS  can  be  initiated  swiftly.  This  includes  in 
particular  the  weaknesses  accompanying  the  current  online  Roma  reporting  tool 
provided  by  the  Commission  (data  availability  to  feed  the  tool;  cooperation 
between  the  NRCPs  and  other  stakeholders  in  gathering  data;  development  of 
more  meaningful  indicators)  which  should    be  addressed.  Equally,  suggestions 
made during the evaluation for reducing administrative burden (such as regarding 
frequency and level of reporting) should be further explored. Existing comparable 
data (e.g. from the FRA) could be integrated into the tool to provide outcome and 
impact indicators. Also, on-going efforts of the Member States towards reliable, 
ethnically  disaggregated  data  collection  should  continue,  in  line  with  national 
practices, as the improvements regarding data availability to feed the tool will not 
solve the problem of lack of disaggregated data in some cases. 

  Effective  Roma  participation  in  policy-making  can  be  further  encouraged, 
both at national and EU level. This includes capacity building of Roma and pro-
Roma  civil  society  as  well  as  their  involvement  in  the  monitoring  of  NRIS 

298 ICF, Chapter 8, as documented in country analysis studies and interviews. 

57 

 
 
 
                                                           
implementation.  At  EU-level,  events  similar  to  the  workshop  organised  for  this 
evaluation, bringing together NRCPs and Roma civil society for a discussion, are 
useful to stimulate increased dialogue.  

  Continued efforts  at EU and national level are needed to  halt  and  break  down 
the rising levels of discrimination and antigypsyism. Building on efforts made 
in  the  past  (at  EU  level  for  example  the  prioritisation  of  the  fight  against 
discrimination of Roma, antigypsyism, hate speech and hate crime in calls under 
the  Rights,  Equality  and  Citizenship  programme),  Member  States  should  be 
encouraged 
tackling  discrimination  and 
antigypsyism  (such  as  for  example  awareness  raising  targeting  employers, 
teachers, police). 

implement  more  measures 

to 

  Roma needs could be better reflected in mainstream policies, in particular at 
national  level.  Effective  inclusion  of  Roma  happens  when  mainstream  policies 
are  sensitive  and  responsive  to  their  needs.  In  this  context,  awareness  about  he 
existence  of  NRIS  and  NRCP  could  be  increased.  Also,  based  on  the  generally 
positive evaluation of the link between the EU framework and ESIF, Member 
States should make full use of the ESF investment priority 9.2 ‘Socio-economic 
integration of  marginalised communities such as  the Roma’ as well as of  other 
relevant ESF and ERDF investment priorities. In their programming for the post-
2020 period, Member States should make full use of the opportunities offered by 
the post-2020 ESIF regulations299 and enlargement countries should make use of 
IPA  assistance.  At  EU  level,  consistency  between  the  objectives  of  the  EU 
framework and other key policies such as the European Semester, the European 
Pillar  of  Social  Rights  and  the  UN  2030  Agenda  on  Sustainable  Development 
should be ensured. 

  While  consolidating  its  achievements  in  terms  of  EU  added  value,  the 
Commission could start to prepare a programme of EU-level  good  practice 
exchanges  and  mutual  learning,  taking  into  account  transferability  of  good 
practices in light of differences between the countries.  

  Overall, Member States could be encouraged to already take  steps  to  make  the 
positive results more sustainable. Key conditions for sustainability presented in 
the  evaluation  are:  (i)  a  consistent  EU  and  national  funding  mix;  (ii)  the 
development  of  inclusive  public  services  that  recognise  the  needs  of  the  most 
vulnerable,  including  Roma;  and  (iii)  involving  Roma  in  a  structured  policy-
making process. 

299 https://ec.europa.eu/commission/publications/investing-people_en for ESF+ proposal adopted by the Commission 
on 30 May 2018; https://ec.europa.eu/commission/publications/regional-development-and-cohesion_en for 
Common Provisions Regulation proposal adopted by the Commission on 29 May 2018. 

58 

 
 
 
 
 
 
 
 
 
 
 
                                                           
ANNEX 1: PROCEDURAL INFORMATION 

Lead DG, Decide planning/CWP references 

 DG Justice and Consumers (DG JUST) 

 PLAN/2017/830 

Organisation and timing 

This  evaluation  has  been  steered  by  DG  Justice  and  Consumers  since  May  2017  under 
the  scrutiny  of  an  inter-service  group  (ISSG)  comprising  representatives  of  DGs  EAC, 
EMPL,  HOME,  REGIO,  AGRI,  JRC,  NEAR,  SG  and  SANTE,  and  the  Fundamental 
Rights  Agency.  The  ISG  was  consulted  at  each  stage  of  the  evaluation  process  and 
reviewed  each  deliverable  produced  by  the  contractor  as  well  as  this  staff  working 
document (SWD). ISSG meetings took place on: 

 5 May 2017 (consultation strategy) 
 14 June 2017 (terms of reference evaluation study) 
 29 November 2017 (inception report) 
 24 January 2018 (interim report) 
 25 April 2018 (final report). 

The  ISG  was  consulted  on  the  draft  SWD  on  28 June  2018  and  informed  about  the 
Regulatory Scrutiny Board’s opinion on 05 October 2018. 

Exceptions to the better regulation guidelines 

None 

Consultation of the Regulatory Scrutiny Board (RSB) (if applicable) 

The RSB was consulted on 29 August 2018 and met on 26 September 2018. The Board 
gave a positive opinion300, with comments to improve the document.  

The  following  changes  were  made  to  this  SWD,  in  response  to  the  Board’s  main 
considerations: 

Regulatory Scrutiny Board's 
recommendation 

Changes made to the Staff Working 
Document 

The  report  does  not  present  a  convinving 
picture  how  monitoring  and  reporting 
systems  could  be  improved  between  now 
and 2020. 
The  report  does  not  explain  why  non-
discrimination  is  left  out  of  the  EU’s 
framework. 

to 

improve 

The conclusions now more clearly list what 
the 
taken 
steps  can  be 
monitoring  and  reporting  systems  until 
2020 (chapter 6) 
Further  explanations  were  added  to  the 
chapter  presenting  the  background  to  the 
EU framework (chapter 2). 

300 ARES(2018)4995826  

59 

 
 
 
 
 
                                                           
The report does not do enough to examine 
how  various  combinations  of  adverse 
factors posed challenges and contributed to 
poor outcomes in different jurisdictions. 

The report does not sufficiently distinguish 
issues  that  can  be  addressed  in  the  short 
term from longer-term issues. Some of the 
conclusions are more positive that what the 
underlying analysis would justify. 

In  addition,  references  to  the  enforcement 
of  EU  non-discrimination  law  were  added 
(chapter 5.1.1. effectiveness). 
The  SWD  now  addresses  differences 
across countries more clearly: 

regarding 
the  framework  and  NRIS 

- References to different shares of Roma in 
the population and different starting points 
were introduced (chapter 2)  
alignment 
-  Observations 
between 
in 
countries with higher shares of Roma were 
added (chapter 5.1.2.)  
- The chapter on effectiveness is now more 
upfront  about  attribution  difficulties.  Also, 
more  specific  references  to  countries  were 
added  to  parts  of  the  effectiveness  section 
(chapter 5.1.1.) 

The  mid-term  evaluation  however  did  not 
evaluate  the  effectiveness  of  NRIS  and 
therefore does not assesss policy responses 
and progress at national level. 
The  conclusions  now  distinguish  more 
clearly  issues  that  can  be  addressed  in  the 
short  term  until  2020  from  more  longer-
term  issues,  likely  to  require  a  new  EU 
policy proposal (chapter 6). 

that 

stating 

relevance, 

The  conclusions  are  also  clearer  now 
the 
regarding 
assessment is positive with limitations. The 
three  criteria  of  coordination,  equity  and 
sustainability are now better integrated into 
the  assessment  of  the  standard  criteria 
(chapter 6) 

Evidence, sources and quality 

This SWD has been drawn up according to a roadmap published in March 2017301, to 
which no feedback was received. 

Sources of evidence 

The  following  box  provides  an  overview  of  the  data  sources  from  which  the  evidence 
was drawn. A detailed description of the individual methods is provided in Annex 2. 

301 https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2017-1716166_en 

60 

 
 
 
 
 
 
 
                                                           
  Literature review (175 sources302) 
  Key  stakeholder  phone  and  face-to-face  interviews  in  27  Member  States  (191 

interviews) 

  10  key  stakeholder  phone  and  face-to-face  interviews  in  three  enlargement 

countries 

  8  phone  and  face-to-face  interviews  at  EU  level  (Commission  staff,  MEP,  EU 

level NGOs) 

  Country  analysis  studies  (11  Member  States  with  a  total  share  of  90.2 %  of  the 
EU  Roma  population)  pulling  together  quantitative  and  qualitative  country-
specific information 

  Open public consultation (240 responses; 28 position papers submitted) 
  Targeted NGO survey (65 responses) 
  Data  provided  by  Member  States  in  the  online  reporting  tool  provided  by  the 

Commission for annual reporting (years covered: 2015 and 2016) 

  ESIF  programming  documents,  303  including  through  the  use  of  dedicated 

portals provided by the Commission (SFC304 2014) 
  2011 FRA Roma Survey and 2016 FRA MIDIS II survey 
  2011  UNDP/World  Bank/EC  regional  Roma  survey  and  2017  UNDP/World 

Bank/EC survey on the Western Balkans 

  Evaluation  workshop  on  preliminary  findings,  EU  Roma  Week  event  on 
evaluation,  2017  European  Platform  for  Roma  Inclusion  (focusing  on  the 
transition  from  education  and  employment)  and  other  stakeholder  and 
consultation events. 

Expert advice 

A workshop was organised on European Commission premises in Brussels on 15 March 
2018. The workshop brought together 88 stakeholders who discussed the initial findings 
of the evaluation. Participants of the workshop represented a wide range of stakeholders 
and  experts.  Present  were  29  national  Roma  contact  points  (across  EU  Member  States 
and enlargement countries), 28 representatives of NGO or civil society organisations, 14 
representatives  of  the  European  Commission,  4  representatives  of  international 
organisations,  2  representatives  of  the  European  Parliament,  1  representative  of  the 
Fundamental Rights Agency and 1 representative of Equinet. 

External consultant 

The external evaluator contracted for this assignment (ICF/Milieu) has carried out since 
October 2017 all tasks as required, under the scrutiny of an inter-service group (ISSG) set 
up  for  this  evaluation  and  the  steer  of  DG  Justice  and  Consumers.  Primary  data  were 
mainly  collected  from  January  to  March  2017.  During  the  inception  phase,  the  ISSG 
clarified the thematic scope of the evaluation compared to the wording in the roadmap, 
which  was  found  to  be  insufficiently  clear.  It  was  clarified  that  the  evaluation  would 
focus  on  assessing  the  EU  framework  but  abstain  from  evaluating  any  other  financial, 
legal  or  policy  initiative  mobilised  for  Roma  integration  or  the  NRIS  itself.  This 

302 Bibliography is provided in the external evaluation study; in addition around 200 national sources were included in 

the country analysis studies. 

303 Operational programmes; synthesis reports of annual implementation reports such as for the ESF 2017 
304 System for fund management in the European Union. 

61 

 
 
 
                                                           
clarification was important in light of the request made by the Council to evaluate the EU 
framework itself. The external evaluation study was finalised in July 2018305. Its overall 
quality was assessed as satisfactory by the ISSG. 

305 Add link once published. 

62 

 
 
                                                           
ANNEX 2: STAKEHOLDER CONSULTATION 

1. Introduction 

This annex presents an overview of all activity conducted with stakeholders as part of the 
‘Midterm  evaluation  of  the  EU  framework  for  national  Roma  integration  strategies 
(NRIS) up to 2020’. The general objectives of the stakeholder consultations, as outlined 
in  the  consultation  strategy  prepared  for  the  evaluation,  were  to  collect  stakeholders’ 
views on: 

 

 

results achieved and challenges faced during the 5 years of implementation of the 
EU framework in the key areas of education, employment, healthcare and housing 
as well as on discrimination; 
the alignment, relevance and effectiveness of European and national policy, legal 
and funding instruments. 

The specific objectives of the stakeholder consultations were to obtain targeted feedback 
on: 

 
 

 

the use and results of the EU framework; 
the use, impact and alignment of European policy, legal and funding instruments 
put in place in support of Roma integration; 
the  impact  on  Roma  of  the  implementation  of  the  NRISs  and  of  mainstream 
policies. 

The  consultation  strategy  specified  that  stakeholder  views  should  be  ensured  by 
facilitating  targeted  stakeholder  consultations,  and  through  an  open  public  consultation 
(OPC).  The  stakeholder  consultations  that  were  ultimately  pursued  as  part  of  the 
evaluation  covered  both  of  these  activities.  The  final  types  of  stakeholder  consultation 
that took place are as follows: 

  an OPC, which featured a set of questions for a range of stakeholder groups; 
 

targeted  stakeholder  consultations,  which  took  the  form  of  interviews  with  a 
variety  of  stakeholder  groups  from  across  Member  States  and  enlargement 
countries; 

  an online survey specifically targeted  at  NGOs;  the survey enabled the views of 
NGOs involved in Roma integration across Member States to be incorporated, as 
their  insight  might  not  have  been  adequately  captured  through  the  other 
consultation methods planned; 

  a  workshop,  which  brought  together  a  wide  variety  of  expert  stakeholders  to 
address different dimensions of the preliminary findings of the external evaluation 
study. 

The  following  table  summarises  the  range  of  stakeholders  consulted  as  part  of  the 
evaluation, in line with the consultation strategy. 

63 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
Table 10: Stakeholder type and data collection method 

Stakeholder type 
National Roma contact points 

representing  national, 

Stakeholders 
local, 
regional  and  municipal  authorities,  and  other 
public  or  mixed  entities  such  as  social  services, 
housing, health, education service providers 
Representatives  of  non-governmental  and  civil 
society organisations (EU umbrella organisations 
and  organisations  active  in  Member  States  on 
national/ regional/ local levels) 
Representatives  of  international  organisations 
and  institutions  active  in  the  area  of  Roma 
integration  in  EU  countries  and/or  enlargement 
countries 
Representatives  of  research  and  academic 
institutions 

Data collection method 
OPC 
Targeted stakeholder consultations 
Workshop 

OPC 
Targeted stakeholder consultations 

OPC 
Targeted stakeholder consultations 
Workshop 
NGO survey 
OPC 
Targeted stakeholder consultations 
Workshop 

OPC 
Targeted stakeholder consultations 
Workshop 

European-level  experts  with  a  stated  interest  in 
Roma integration issues 

Targeted stakeholder consultations 
Workshop 

EU    and  non-EU  citizens,  including  members  of 
Roma communities 

OPC 

Representatives  of  organisations  representing 
churches and religious communities 

OPC 

Representatives  of  business  and  professional 
associations 

OPC 

2. Overview of consultation activities 

While all the stakeholder types included in Table 10 were approached, representatives of 
organisations representing business and professional associations did not respond to the 
OPC. The other stakeholder groups were effectively reached through the data collection 
methods outlined above306. 

2.1. Open public consultation 

The  OPC  carried  out  by  the  Commission  aimed  to  compile  the  opinions  of  these 
stakeholders on the achievement and challenges of the EU framework between 2011 and 
2016,  in  order  to  identify  specific  areas  which  would  need  prioritising  during  the 

306 It cannot be ascertained whether representatives of organisations representing churches and religious communities 
were reached, as the identity categories of the OPC were: non-governmental organisation, public, administration, 
business, employer organisation, trade union, association, academia/research/think tank and other. 

64 

 
 
 
 
 
 
                                                           
remaining  implementation  period.  Additionally,  the  OPC  took  stock  of  the  various 
European and national policies, and legal and funding instruments which had so far been 
mobilised for Roma integration. The OPC questionnaire307 consisted of 16 questions308 
that covered: 

introductory questions on the background of the respondent; 

 
  general  questions  on  social  exclusion  and  discrimination  and  expectations  for 
future priority areas at the European and national level. This set of questions did 
not  require    specific  knowledge  of  European  or  national  instruments  used  to 
further Roma inclusion; 

  specific  questions  on:  (i)  European  and  national  efforts  at  Roma  inclusion;  (ii) 
relevant policy developments; (iii) achievements and challenges pertaining to the 
EU framework and NRISs; and (iv) specific measures taken across the four main 
policy areas of education, employment, health and housing. 

The  online  OPC  ran  between  19  July  and  25 October  2017  on  the  website  of  the 
European Commission. A total of 240 responses were received to the survey309. 165 of 
these  came  from  organisations,  while  75  were  from  individual  citizens.  Of  those  165 
organisations,  106 indicated that they represented a NGO or think tank,  44 represented 
public administration, and 15 answered on behalf of other organisations (such as equality 
bodies). Of the 240 respondents, 202 specified their ethnicity: 91 identified themselves as 
Roma and 111 as non-Roma. 

Additionally, 28 position papers were received as part of the OPC. The majority of these 
were from NGOs, although UN agencies, universities and the World Health Organization 
also provided submissions. Some of the submissions were tailored responses to the OPC, 
while others were research or advocacy papers going back as far as 2010. 

2.2 Targeted stakeholder consultations 

Interviews with a range of relevant stakeholders across Member States and enlargement 
countries were carried out. Interview consultations that were undertaken were categorised 
into three groups: 

  138  interviews  with  stakeholders  (NRCPs,  officials  in  employment,  housing, 
education  and  health  ministries,  NGOs,  equality  bodies,  regional  authorities, 
experts) across 11 Member States selected for country analysis studies310; 

  53  interviews  with  stakeholders  (NRCPs,  officials  in  employment,  housing, 
education and health ministries, NGOs) across 16 Member States not selected for 
country analysis studies311; 

  10  interviews  with  stakeholders  in  three  enlargement  countries  (Serbia,  the 

Former Yugoslav Republic of Macedonia, and Bosnia and Herzegovina). 

307 The online consultation form was published on the EU survey page: 

https://ec.europa.eu/eusurvey/runner/EvaluationEUFrameworkforNRIS. 

308 All questions were optional except those on self-identification. 
309 The results of the consultation are published on the European Commission website and are available here: 

https://ec.europa.eu/info/consultations/public-consultation-evaluation-eu-framework-national-roma-integration-
strategies-2020_en. 

310 AT, BG, CZ, ES, EL, FR, FI, IT, HU, SK and RO. 
311 BE, CY, DE, DK, EE, IE, HR, LV, LT, LU, NL, PL, PT, SE, SI and UK. 

65 

 
 
 
 
 
 
 
 
                                                           
The  final  number  of  stakeholders  interviewed  was  less  than  the  number  originally 
intended, as it was not always possible to arrange an interview with all stakeholders312. 
The main reasons for this included: (i) some of those contacted were unable to take part 
in the interview, but also unable to suggest another possible contact to replace them; (ii) 
some ceased responding; and (iii) others did not reply to the initial invitation to interview. 
In  addition  to  the  above  interviews,  eight  interviews  were  conducted  with  experts 
operating at the European or international level313. 

2.3. NGO survey 

A targeted online survey was opened on 16 January 2018 and ran until 23 February, with 
the  aim  of  giving  NGOs  an  additional  opportunity  to  provide  comments.  This  was 
considered important as not in all countries were NGOs part of the targeted stakeholder 
consultation  described  below.  The  survey  drew  65  full  responses.  Respondents  to  the 
survey represented 19 Member States, with the largest share of respondents representing 
Greece, Slovenia  and Sweden (each Member State individually  accounting for 13 % of 
all  respondents).  A  substantial  proportion  of  organisations  (47 %)  reported  that  they 
operate  at  the  national  level,  while  16 %  of  respondents  represented  a  regional-level 
organisation, 17 % a community- or local-level organisation, while represented 14 % an 
international organisation. 

The  survey  questions  comprised  24  multiple  choice  questions,  organised  in  accordance 
with  the  different  evaluation  criteria  explored  in  the  study:  relevance,  coherence, 
effectiveness,  equity,  coordination,  efficiency, sustainability and EU added value. They 
consisted  of  statements  for  which  the  respondents  had  to  state  whether  they 
agreed/disagreed/neither agreed nor disagreed/had no opinion/did not know.  

2.4. Workshop 

A workshop was organised on European Commission premises in Brussels on 15 March 
2018.  The  workshop  brought  together  88  stakeholders  who  discussed  the  preliminary 
findings  of  the  evaluation.  The  workshop  gave  participants  the  opportunity  to  respond 
specifically  to  the  findings  on  the  effectiveness,  EU  added  value,  relevance  and 
coherence of the EU framework. Participants at the workshop represented a wide range 
of stakeholders. Present were 29 national Roma contact points (from across EU Member 
States  and  enlargement  countries),  28  representatives  of  NGO  or  civil  society 
organisations,  14  representatives  of  the  European  Commission,  4  representatives  of 
international  organisations,  2 
the  European  Parliament,  1 
representatives  of 
representative  of  the  Fundamental  Rights  Agency,  1  representative  of  Equinet  and  9 
members of the ICF/Milieu evaluation team. 

3. Methodology 

3.1 Open public consultation 

312 It was intended that 64 telephone interviews would be conducted with stakeholders in the 16 Member States not 

covered by a country analysis study and in enlargement countries, while up to 20 interviews would be conducted 
in each of the 11 Member States covered by the country analysis studies. 

313 It was intended that 10 European- or international-level stakeholders would be interviewed; despite repeated 

efforts, it was only possible to interview eight. 

66 

 
 
 
 
 
 
 
                                                           
The  survey  data  was  downloaded  by  the  Commission  in  excel  format  and  processed 
using the following excel functions: tables, bar charts, filters and cross tabulations. The 
analytical work involved the breaking down of the results by: 

respondent type (citizens vs organisations); 

 
  organisation type (public administrations, NGOs/think tanks, other); 
  Roma  vs  non-Roma  background  (i.e.  respondents  identifying  as  Roma  vs 

respondents identifying as non-Roma); 
the following country clusters: EU-15, EU-13, enlargement counties. 

 

A separate analysis of the survey results was done for the five countries with relatively 
sizeable  Roma  communities:  Bulgaria,  the  Czech  Republic,  Hungary,  Romania  and 
Slovakia.  The  results  were  broken  down  based  on  the  above  characteristics  to  identify 
any  differences  in  perception  or  opinion  among  certain  respondent  categories  or  in 
certain (groups of) countries. A report analysing the results of the OPC was submitted by 
the external contractor to the Commission as a separate deliverable. 

3.2 Targeted stakeholder consultations 

Stakeholders were selected depending on the type of organisation or institution that they 
represented,  following  provisional  sampling  agreed  at  inception  report  level.  For  each 
country covered, it was imperative to consult with a representative of the national Roma 
contact point314. Other categories of stakeholders interviewed included equality bodies, 
representatives of government entities at national or regional level (dealing with health, 
employment,  education,  housing  and  trafficking),  and  also  NGOs.  For  each  of  the  11 
country  analysis  studies,  local  country  correspondents  carried  out  an  initial  stakeholder 
analysis to also identify local government representatives responsible for the four policy 
areas. The full list of stakeholders to be consulted per country was agreed with the ISSG. 

All  responses  from  the  targeted  consultations  were  processed  using  NVivo  qualitative 
data analysis computer software315. The interview guidelines that were used to support 
the  individual  consultations  had  a  specific  structure,  which  grouped  certain  questions 
together in correspondence with the evaluation criteria being assessed. This meant that as 
a  whole,  the  interview  responses  could  be  effectively  analysed  per  evaluation  criteria, 
using NVivo software to isolate those responses relevant to the evaluation criteria. 

3.3 NGO survey 

A list of 135 national NGOs from across the EU was developed and the survey was sent 
on  15  January  to  these  NGOs.  National  Roma  contact  points  were  also  invited  to 
distribute  the  survey  further.  The  NGO  survey  was  comprised  of  a  series  of  multiple 
choice  questions.  The  data  received  showed  how  many  people  responded  to  each 
question and the  percentage share of respondents that answered a certain  question. The 
content of the response fed into each evaluation criterion. 

3.4. Workshop 

314 Contacts provided by the Commission. 
315 https://en.wikipedia.org/wiki/NVivo. 

67 

 
 
 
 
                                                           
The workshop presented the preliminary evaluation findings and enabled participants to 
provide  feedback  in  particular  to  the  specific  evaluation  criteria  questions  on 
effectiveness,  EU  added  value,  relevance  and  coherence.  This  feedback  was  given 
through  an  open  discussion,  facilitated  by  members  of  the  evaluation  team.  Similarly, 
participants  were  invited  to  share  recommendations  openly  at  the  workshop  or  to  send 
them in writing later. 

4.  Results  of  stakeholder  consultations  per  activity  and  how  they  fed  into  the 
evaluation 

The  results  of  the  consultation  activities  were  used  according  to  the  evaluation 
framework  agreed  at  inception  report  level.  For  each  of  the  evaluation  questions,  the 
evaluation framework clarified which of the consultation activities would be relevant for 
data collection. 

4.1 Open public consultation (OPC) 

A  full  summary  of  results  stemming  from  the  OPC  was  published  on  EUROPA316. 
Findings include: 

  An overwhelming majority of the respondents (between 86 and 95 % depending 
on the thematic area) agreed that targeted public interventions are needed in the 
fields of discrimination, employment, education, housing and healthcare317. 
  For a majority of the respondents (almost 60 %), the EU has a major role to play 
in  supporting  national,  regional  and  local  authorities  because  alone  they  cannot 
effectively improve the situation of Roma318. 

  Respondents consistently stated that both EU institutions and national authorities 
should work together to develop measures to improve Roma inclusion. They see a 
stronger  role  for  the  EU  than  for  national  authorities  in:  (i)  monitoring  and 
enforcing  European  non-discrimination  and  anti-racism  legislation;  and  (ii) 
making access to funding conditional on developing and implementing ambitious 
Roma policies319. 

  National  authorities  are  expected  to  play  a  bigger  role  in  measures  such  as:  (i)  
community building between Roma and non-Roma; (ii) non-discrimination; (iii) 
training for public officials on how to achieve Roma inclusion; (iv) making Roma 
history and culture part of school curricula; and (v) providing policy guidance to 
authorities320. 

  Key  challenges  identified  by  the  respondents  include:  (i)  the  insufficient 
incorporation  of  Roma  inclusion  into  other  policies  and  instruments  at  both 
European  and  national  level;  (ii)  rising  discrimination  and  antigypsyism, 
especially  at  European  level;  and  (iii)  insufficient  funding  allocated  to  Roma 
inclusion at the national level321. 

  With  regard  to  suggested  priorities  at  European  and  national  level,  respondents 
confirmed  that  successful  Roma  inclusion  strategies  need  to  be  comprehensive. 
Access to education came out as a clear priority (67 % at European level and 76 % 

316 OPC results available here: https://ec.europa.eu/info/consultations/public-consultation-evaluation-eu-framework-

national-roma-integration-strategies-2020_en. 

317 EU added value. 
318 EU added value. 
319 Coordination; EU added value. 
320 Coordination. 
321 Effectiveness. 

68 

 
 
 
                                                           
at  national  level), while  access to  employment, healthcare and housing, fighting 
discrimination and addressing antigypsyism were also selected as a priority by at 
least one third of respondents at both European and national levels322. 

4.2. Targeted stakeholder consultations 

Targeted  stakeholder  consultations  consisted  of  201323 interviews  as  explained  above. 
The  replies  to  the  interviews  were  so  numerous  and  varied,  and  covered  such  a  wide 
geographical  scope,  that  summarising  their  results  as  a  whole  for  the  purpose  of  this 
report is not realistic. However, the outcomes of these targeted stakeholder interviews are 
fully  reflected  in  the  external  evaluation  study324 and  this  SWD,  using  the  following 
referencing of sources: 

Table 11: Referencing of sources 

Type of source 
Interviews 

Open public consultation 
Country analysis studies 

Survey of NGOs 
Stakeholder workshop 

4.4. NGO survey 

Referencing code 

the  particular 

Member  State  code,  followed  by  a  number 
indicating 
interview  being 
referenced.  The  full  interview  code  list  is 
available in a separate document. E.g. UK1 
EU-level  interviews  are  abbreviated  as  EU-1, 
EU-2, etc. 
Enlargement  country  interviews  are  coded  as 
WB1, WB2, etc. 
OPC 
CS-[country  code]  —  e.g.  CS-SK  for  the  SK 
Country Analysis Study  
NGO survey 
Workshop 

The  results  of  the  NGO  survey  are  published  in  the  external  evaluation  study325.  Key 
findings include: 

  With regard to contributions made by NRISs to effective changes on the ground, 
survey  participants  considered  that  the  NRIS  of  their  respective  Member  State 
had contributed to some extent to reducing discrimination against Roma (40 %), 
reducing  hate  speech  against  Roma  (50 %),  reducing  hate  crimes  against  Roma 
(53 %)  and  improving  the  enforcement  of  anti-discrimination  legislation  (44 %) 
during 2011-2017326. 

  Nearly half of the respondents considered that the Roma’s access to employment 
has  not  changed  since  2011.  43 %  of  the  respondents  felt  that  Roma  children’s 
access to and integration into education systems has improved since 2011, while 
20 % believed this has worsened over the years327. 

  Around half of the NGOs (53.8 %) are involved in mechanisms for coordination, 
implementation  or monitoring of the NRIS. However, most of them support  the 

322 Relevance. 
323 Comprising 138 stakeholder consultations across 11 Member States covered by a country analysis study; 53 

stakeholder consultations across 16 Member States not covered by a country analysis study; and 10 stakeholder 
consultations with stakeholders from enlargement countries. 

324 Link to external evaluation study once published. 
325 Add footnote once published. 
326 Effectiveness. 
327 idem. 

69 

 
 
 
                                                           
opinion that the existing mechanisms for coordination and implementation of the 
NRIS  do  not  allow  for  effective  cooperation  and/or  consultation  with  all  key 
stakeholders328. 

  Asked  whether  both  EU  and  national  funding  per  Member  State  was  sufficient 
and  proportionate  to  meet  the  needs  of  Roma  across  the  four  policy  areas,  over 
half of all respondents thought that this was not the case. 63 % of respondents did 
not  think  that  EU  or  national  funding  for  Roma  inclusion  reaches  the  Roma 
beneficiaries  it  was  intended  for,  and  did  not  think  that  it  provides  them  with 
long-term  benefits329.  60 %  of  NGO  respondents  believe  that  EU  funding  has 
provided added value in terms of addressing the national funding gap330. 

4.5. Workshop 

Consultation with workshop participants focused on the evaluation criteria of relevance, 
coherence, effectiveness and EU added value. 

With regard to relevance, the participants agreed that the original objectives of the EU 
framework  remained  relevant  today  but  that  the  ambition  in  some  areas  should  be 
increased.  Given  the  deep-rooted  nature  of  discrimination  against  Roma,  several 
participants  noted  that  explaining  and  addressing  institutionalised  racism  was  of  great 
importance, as was generally increasing the focus on antigypsyism. 

For the coherence criterion, several workshop participants highlighted the need for better 
policy  mainstreaming.  On  budget  allocation  and  funding  for  Roma  inclusion,  it  was 
recommended  that  specific  Roma  indicators  be  identified  and  developed  to  effectively 
implement NRISs. Participants noted problems due to the NRCPs feeling isolated within 
their  governments  and  that  the  lack  of  financial  capacity  can  prevent  the  effective 
implementation of NRISs. 

On effectiveness, several participants noted the important role played by the Commission 
in  prioritising  Roma  issues  and  in  particular  by  adopting  the  EU  framework  in  2011. 
Other  participants  highlighted  how  the  availability  of  EU  funding  for  Roma  inclusion 
helped Member States commit to the cause. The rise of populism and far-right political 
parties  was  also  cited  by  participants  as  a  worrying  aspect  that  would  affect  the 
objectives for Roma inclusion. 

With  regard  to  EU  added  value,  following  a  question  asked  through  an  online  tool 
(SLIDO),  97 %  of  the  participants  agreed  that  the  EU  had  provided  added  value  for 
Roma inclusion in the Member States. NGO participants added that none of the Member 
States would be where they were without the EU framework. On the other hand, several 
participants  stated  that  while  certain  tools  are  in  place,  a  clear  connection  between  EU 
funding and the indicators in the NRIS should be made and monitored. 

5. 

Overall results from the consultations 

Across  the  results  of  all  the  consultations,  a  number  of  common  messages  can  be 
identified. In particular, there was a broad consensus that since 2011: 

328 Coordination. 
329 Effectiveness. 
330 EU added value. 

70 

 
 
 
 
 
 
 
                                                           
  The EU framework has been necessary to help realise positive changes related to 
Roma  inclusion  across  the  policy  areas  of  education,  employment,  health  and 
housing at national level. 

  Without the framework and EU direction and support, it is unlikely that Member 

States would be able to effectively improve the situation of the Roma. 

More specifically, multiple stakeholders raised the aspects set out below. 

 

Improving  access  to  education  must  remain  an  absolute  priority  for  all  actors 
involved in the implementation of NRIS. 

  Roma inclusion has become a higher priority on the EU policy agenda. 
  Mainstreaming  of  Roma  inclusion  in  policies  remains  to  be  effectively 

implemented in particular at national levels. 

  Political  commitment  at  national  level  to  policies  that  ensure  Roma  inclusion 

must be increased. 

  Measures at national level to tackle antigypsyism were insufficient. 
  National  funds  are  often  deemed  insufficient  to  implement  Roma  inclusion 

measures. 

The  results  of  the  stakeholder  consultations  generally  demonstrate  a  range  of  common 
aspects and shared areas of concern. While there were differences in the opinions of the 
multiple  stakeholders  consulted,  these  differences  are  normal  given  the  backgrounds  of 
the  interviewees,  for  example  when  discussing  the  functioning  and  influence  of  the 
NRCPs  with  NGOs  vs  with  NRCPs  themselves;  or  when  discussing  mainstreaming  at 
national  level  with  ministries  vs  NGO  experts.  Such  differences  were  reflected  in  the 
analysis and do not challenge the above overall results of the consultations. 

71 

 
 
 
 
ANNEX 3: EVALUATION CRITERIA AND QUESTIONS 

Relevance (relationship between needs and objectives): 

To  what  extent  have  the  (original)  objectives  proven  to  be  appropriate  in  view  of  the 
(original) needs? 
To what extent were the original objectives of the EU framework appropriate in view of 
the needs? 
To  what  extent  was  the  combination  of  social  inclusion  and  anti-discrimination 
objectives sufficiently balanced to address the needs? 

To what extent is the EU framework still relevant? 
How well do the original objectives still correspond to the current needs? How relevant 
is the EU framework considering the current levels of discrimination and disadvantages 
faced by Roma? 
How relevant are the goals of the EU framework for Roma and for European societies? 
Do the objectives need to be updated, when and how? 

Coherence 
mobilised to promote Roma inclusion): 

(relationship  between  different  European/national 

instruments 

To  what  extent  did  the  EU  framework  contribute  to  mainstreaming  Roma  integration 
into European and national policy? 
To what extent did it contribute to mainstreaming Roma integration into policies?   
To what extent did the EU framework contribute to linking Roma integration priorities 
with EU funding? 
To what extent did the EU framework contribute to fighting discrimination against Roma 
under  European  legislation  (such  as  the  Racial  Equality  Directive,  the  Council 
Framework Decision on Racism and Xenophobia and Directive 2011/36/EU of 5 April 
2011  on  preventing  and  combating  trafficking  in  human  beings  and  protecting  its 
victims)? 
To what extent did the monitoring and guidance provided by the European Commission 
(under  the  EU  framework,  the  European  Semester,  European  legislation  such  as  the 
Racial Equality Directive) identify relevant points for improving effectiveness of Roma 
integration efforts? 
To what extent is the EU framework coherent with other EU policies? 

To  what  extent  did  National  Roma  Integration  Strategies  (NRIS)  or  integrated  sets  of 
policy measures contribute to mainstreaming Roma integration under public  policies in 
the fields of education, employment, healthcare and housing? 
To  what  extent  did  they  contribute  to  linking  Roma  integration  priorities  with  EU  and 
national funding? 
To what extent did they contribute to fighting discrimination, hate speech and hate crime 
against  Roma,  enforcing  EU  anti-discrimination  and  anti-racism  legislation  at  national 
and local levels and addressing prevention and awareness-raising regarding trafficking in 
human beings? 

72 

 
 
 
 
 
 
Effectiveness (degree of progress towards achieving objectives and role of the EU 
action in observed changes): 

To what extent have the objectives set out in the EU framework as defined in 2011 and 
extended by the 2013 Council Recommendation been achieved from 2011-2016? 
What  have  been  the  changes  in  discrimination  patterns  in  education,  employment, 
housing and health? 
What  have  been  the  quantitative  and  qualitative  effects  of  Roma  integration  measures 
implemented in Member States in education? 
What  have  been  the  quantitative  and  qualitative  effects  of  Roma  integration  measures 
implemented in Member States in employment? 
What  have  been  the  quantitative  and  qualitative  effects  of  Roma  integration  measures 
implemented in Member States in healthcare? 
What  have  been  the  quantitative  and  qualitative  effects  of  Roma  integration  measures 
implemented in Member States in housing? 
To  what  extent  can  changes  in  discrimination  patterns  and  quantitative  and  qualitative 
effects of Roma integration measures in employment, education, housing and healthcare 
be credited to the EU framework? What other factors have influenced the achievements 
observed? 
Why were certain objectives not reached? 

To  what  extent  did  the  EU  framework  contribute  to  setting  up  and  mobilising  the 
necessary  instruments  (such  as  the  NRIS  and  annual  reports  by  Member  States,  civil 
society and the Commission) and governance structures at European and national levels 
(National  Roma  Contact  Points  and  its  network,  National  Roma  Platforms,  European 
Platform for Roma Inclusion, civil society consultation meetings, EU Roma Summits)? 
Has the EU framework served as a framework for NRIS? 

To what extent did National Roma Integration Strategies (NRIS) lead to effective Roma 
integration measures and inclusive reform of mainstream policies? 
To what extent did they contribute to increased European and national funding for Roma 
integration and inclusive reforms? 
To what extent did the number of Roma beneficiaries of integration measures increase? 

3.4.4  To  what  extent  did  national/local/civil  capacity  to  implement  Roma  integration 
measures improve? 

Coordination (synergies between interventions): 

Have coordination mechanisms at EU/national level been effective in coordinating the 
policy making, funding, implementation, monitoring and evaluation)? 
To what extent effective methods of coordination of the implementation engage the 
relevant stakeholders? The extent of participation of civil society, in planning and 
monitoring inclusion policies and programmes 
What is the level of dialogue and cooperation with Roma representatives? 
Existence and clarity of the communication measures of the NRIS? 
Is there sufficient administrative capacity at EU, national, sub- national and local level to 
organise effective coordination of the implementation of the EU framework and the 
National Roma Integration Policies? 

73 

 
 
 
 
 
 
 
Efficiency (relationship between resources invested and benefits): 

What factors influenced efficiency of EU and national interventions in the context of the 
EU framework, in particular regarding the instruments (such as NRIS and annual reports 
by  MS,  civil  society  and  Commission)  and  governance  structures  built  at  EU  and 
national  levels  (NRCPs  and  its  network,  National  Roma  Platforms,  European  Platform 
for Roma Inclusion, civil society consultation meetings, EU Roma Summit)? 
To what extent are the reporting and monitoring systems in place adequate and what are 
the respective strengths and points for improvement on EU and national levels? 
Is there a need to simplify or reduce administrative burdens on NRCPs and MS public 
officials involved in the reporting and information gathering process? 

What  have  been  the  costs  and  benefits  of  Roma  inclusion  in  the  context  of  the  EU 
framework and NRIS? 
To what extent are EU and national funding in the area of Roma inclusion justified and 
proportionate  to  the  long-term  benefits  from  Roma  inclusion  in  quality  inclusive 
mainstream education, employment, healthcare and housing? 

EU added value (changes due to EU intervention): 

To  what  extent  is  EU  action  necessary  to  stimulate,  complement,  leverage  and  create 
synergies with national action to promote Roma integration? 
To what extent do the EU actions in areas such as monitoring, reporting, policy guidance, 
funding and knowledge exchange bring added value? 
To what extent does Roma inclusion continue to require EU level action? 
What would be the consequences of stopping targeted EU action? 

Equity (have results been achieved in a proportional and fair manner with respect 
to vulnerable subgroups groups): 

Has there been sufficient level of awareness-raising efforts for fighting discrimination 
and addressing antigypsyism331? 
To what extent do the Framework and the National Roma Integration Strategies address 
the risk of double discrimination among Roma? 
To what extent the Framework results address the specific needs of Roma women? 
To what extent the Framework results address the specific needs of Roma youth? 
To what extent the Framework results address the specific needs of Roma children? 
To what extent are mainstream and (non Roma-specific) targeting programmes available 
to Roma332? 

Sustainability (extent to which effects are likely to last after the intervention ends): 

What  measures  were  adopted  by  the  European  Commission  and  the  Member  State  to 
ensure the sustainability of the results of the policy actions implemented within the scope 
of the NRIS / EU framework (both at planning and implementation stage)? 
What measures were adopted to ensure the continuity of the activities carried out thanks 
to the EU funding? 

331 Results as presented in the external study not relevant for this criterion. 
332 Results as presented in the external study not relevant for this criterion. 

74 

 
 
 
 
 
 
 
 
                                                           
Are  there  any  mechanisms  in  place  to  ensure  a  sustainability  check  at  policy 
implementation level? 
To what extent are the outcomes/benefits of the policies implemented via the NRIS and 
EU framework expected to continue post 2020? 

75 

 
 
ANNEX 4: METHODS AND ANALYTICAL MODELS 

This annex describes the following aspects in detail: 

1.  Methodological approach of the external evaluation study 

2.  Reasons for and consequences of the scarcity of ethnic data 
3.  Description of key data sources, their limitations and mitigation measures 

1. Methodological approach of the external evaluation study 

The methodological approach taken for the external evaluation study can be summarised 
as follows333: 

Phase 1 included more specifically: 

  An initial desk review of the NRISs and mapping of existing secondary literature 

of interest to the evaluation; 

  A review of: (i) the 2011 FRA Roma survey and 2016 FRA MIDIS II survey; (ii) 
the  Roma  integration  scoreboard;  and  (iii)  the  2011  UNDP/World  Bank/EC 
regional Roma survey; 

  A  review  of  data  available  in  the  online  Roma  integration  measures  reporting 

tool; 

  Eight EU level scoping telephone and face-to-face interviews; 
  Development  of  the  evaluation  framework,  including  the  approach  to  the 

evaluation questions and corresponding judgement criteria.  

333 ICF, inception report. 

76 

 
 
 
 
 
 
                                                           
  Selection of 11 country analysis studies.  
  Development of data collection instruments: (i) interview guidelines and template 
for  country  analysis  studies;  and  (ii)  interview  guidelines  for  other  stakeholder 
consultations.  

  Development  of  the  analytical  approaches  for  assessing  costs  of  Roma 

exclusion/benefits of inclusion. 

Phase 2  included more specifically: 

  A comprehensive literature review; 
  Analysis  of  the  use  of  EU  funds  for  Roma  inclusion,  using  in  particular  ESIF 
programming  documents334,  including  through  the  use  of  dedicated    portals 
provided by the Commission (SFC 2014); 

  Targeted  stakeholder  consultations:  a  large-scale  programme  of  in-depth  semi-
structured  stakeholder  interviews  and  surveys  was  conducted  (see  Annex  2  for 
details). The interviews fell into three groups: 

o  138 interviews with relevant stakeholders across 11 Member States, which 

fed directly into corresponding country analysis studies; 

o  53 interviews with relevant stakeholders across 16 Member States, which 

did not feed into country analysis studies; 

o  8 interviews with relevant stakeholders in 3 enlargement countries. 

  NGO  targeted  survey:  the  survey  specifically  targeted  NGO  representatives  that 
had  not  taken  part  in  the  OPC.  In  total,  the  survey  was  fully  completed  by  65 
respondents (see Annex 2 for detail). 

  Preparation of 11 country analysis studies (AT, BG, CZ, ES, EL, FR, FI, HU, IT, 
RO, SK). In each country between 15 and 20 face-to-face and phone interviews 
were  conducted  (see  Annex  2  for  details).  Secondary  national  literature  was 
reviewed,  as  were  NRIS  implementation  reports.  The  country  studies  were 
undertaken mostly by local experts with detailed knowledge of the local situation 
and prior experience with Roma issues. The 11 countries  were selected  so  as to 
cover  a  maximum  of  Roma  living  in  the  EU  (90.2 %)  while  ensuring  a  mix  of 
countries  with  different  approaches  to  Roma  integration.  The  selection  was 
agreed  with  the  ISSG.  The  purpose  was  not  to  assess  the  NRISs,  but  rather  to 
collect  sufficient  evidence  to  draw  conclusions  for  the  evaluation  of  the  EU 
framework. 

Phase 3  included more specifically: 

  Analysis  of  replies  to  the  OPC:  analysis  of  the  OPC  carried  out  by  the 
Commission (see Annex 2) was carried out following the principles of the Better 
Regulation  Guidelines.  Closed-ended  questions,  open-ended  questions  and 
written responses submitted outside of the framework of the questionnaire were 
analysed using qualitative analytical techniques. The quantitative analysis of the 
OPC involved descriptive analysis of OPC respondents and a univariate analysis 
of other numerical data included in the OPC questionnaire. 

  Triangulation and synthesis: this task involved the systematic organisation of all 
quantitative  and  qualitative  evidence  collected  from  various  secondary  and 
primary sources. This involved coding and collating data collected with the aid of 

334 Operational Programmes; synthesis reports of annual implementation reports. 

77 

 
 
                                                           
NVivo335,  performing  a  sense  check  to  ensure  its  reliability,  and  transforming 
data into usable formats. 

  Cost-benefit analysis: a simplified form of multi-criteria analysis was carried out. 
Applying  a  full  CBA  to  an  EU  level  intervention  such  as  the  EU  framework  is 
impossible.  Measures  deliver  a  range  of  indirect  and  direct  benefits,  not  all  of 
which can be monetised. Cost effectiveness analysis can be used as an alternative 
a  CBA  to  compare  different  interventions  when  the  benefits  (outcomes  and 
impacts) of an intervention cannot be credibly monetised. It is best used when all 
the  expected  effects  have  been  defined  and  are  homogeneous  and/or  can  be 
measured  in  terms  of  a  key  outcome  indicator.  For  this  evaluation,  a  partial 
monetisation or quantification of  costs and benefits  combined with  a qualitative 
assessment  of  costs  and  benefits  that  cannot  be  monetised  or  quantified  was 
carried out. This corresponds to a simplified form of multi-criteria analysis. 

  Sensitivity analysis for financial information. 
  Standard  cost  model  approach  for  administrative  costs:  using  the  ‘standard  cost 
model  approach’,  the  interviews  carried  out  by  the  external  consultant  with 
NRCPs and analysis of information in the online reporting tool were intended to 
provide clarity about the time required to carry out all activities related to the EU 
framework (for example reporting to the EU level) and implementing the NRIS in 
each Member State. 

  A  stakeholders  workshop  was  organised  on  15 March  2018  to  discuss  the 

preliminary findings of the evaluation. 

2. Reasons for and consequences of the scarcity of ethnic data 

Following  the  adoption  and  transposition  of  the  Racial  Equality  Directive,  racial  and 
ethnic minorities are now protected by anti-discrimination legislation across the EU. The 
focus  has  turned  to  implementation  and  monitoring  and  in  turn  the  need  has  arisen  for 
data on (in)equalities based on racial and ethnic origin. Such data are essential to measure 
the  level  of  implementation  and  monitor  the  impact  of  policies,  but  there  are  serious 
shortcomings with regard to the availability of data on the situation of racial and ethnic 
minorities336. 

Many  of  the  conclusions  in  the  present  report  are  based  on  data  from  surveys,  in 
particular FRA’s 2011 Roma survey and 2016 EU-MIDIS and the WB/UNDP 2011 and 
2017  surveys  on  the  Western  Balkans.  This  is  necessary  because  official  government 
statistics on Roma are generally not collected due to a number of ethical, political and/or 
legal considerations in Member States. If data are collected at Member State level, they 
are not comparable across Member States. In all Member States the collection of ethnic 
data takes place in accordance with the EU Data Protection Directive, which prohibits the 
processing  of  personal  data  in  relation  to  certain  special  categories,  including  data 
concerning ethnic origin. This prohibition is, however, subject to exceptions. In particular, 
it does not prevent the gathering of such data, either with the data subject’s consent, or if 
it is rendered anonymous. Most Member States have chosen to follow the wording of the 

335 https://en.wikipedia.org/wiki/NVivo. 
336 European Commission (2017); Network of legal experts ‘Data collection in the field of ethnicity’; Lila Farkas; 

European Commission (2017) Network of legal experts ‘Legal framework and practice in the EU Member 
States’. 

78 

 
 
 
 
                                                           
Directive and interpretation includes racial as well as ethnic origin  among the sensitive 
data337.. 

At EU level, with the exception of the above surveys, there has been a general reluctance 
to  collect  ethnically  disaggregated  data.  Eurostat’s  labour  force  survey  and  the  EU 
statistics on income and living conditions (EU-SILC338) do not include Roma ethnicity 
as a marker. 

The  lack  of  ethnically  disaggregated  data  leaves  the  following  issues  without  a  clear 
answer for most of the EU Member States: 

 
 

 
 

the baseline situation of the level of discrimination and social exclusion339; 
the  number  of  Roma  beneficiaries  reached  through  policy  measures,  especially 
mainstream measures; 
the funds allocated specifically to measures for Roma inclusion; 
the effects of policy measures in countries not covered by the 2016 FRA survey 
and overall uncertainty about attributing the effects to the policy measures. 

Illustrations of limitations in the quantification of funding and benefits 

  There is a lack of data availability on the proportions of EU and national funding 
specifically  spent  on  Roma  integration.  While  amounts  of  funding  can  be 
identified  if  measures  are  targeting  Roma  only,  funding  cannot  be  identified  if 
measures  are designed for several  groups or disadvantaged  groups  in  general  or 
are mainstream measures, i.e. designed for the general population, without Roma-
specific  indicators.  The  evaluation  found  that  there  was  a  general  absence  of 
Roma-specific indicators in national and EU funding programmes. 

  There is scarce reporting on effects. This is primarily due to a lack of information 
about  the  precise  number  of  Roma  beneficiaries  that  benefit  from  a  particular 
intervention (for example the number of Roma participating in a training session). 
This  is  due  in  turn  to  a  lack  of  data  collection  and  relevant  Roma-specific 
indicators. Where contextualising a non Roma-specific indicator would have been 
possible  in  principle,  such  as  for  the  ESF  2014-2020,  the  low  level  of 
implementation  of  the  operational  programmes  so  far  has  been  an  obstacle  for 
quantification340. 

  While  there  is  research  that  demonstrates  economic  returns  on  general  social 
investment in the long term, no conclusive argument can be made about the Roma 
population due to a lack of ethnically disaggregated data. 

  Where  quantification  of  benefits  was  possible  (for  example  on  higher 
employment  rates),  it  was  not  possible  to  demonstrate  convincingly  that  this 
could  be  attributed  to  the  EU  framework.  It  was  therefore  also  not  possible  to 
establish whether benefits would have been smaller without the EU framework. 

337 European Commission (2017); Network of legal experts ‘Data collection in the field of ethnicity’; Lila Farkas. 
338 Known exceptions on the national level for SILC are BG and HU, which collect ethnically disaggregated statistics 

in their national rounds of SILC. 

339 Except for those covered by the 2011 FRA survey (BG, CZ, EL, ES, HR, HU, PT, RO, SK). 
340 For  the  ESF  2014-2020  for  example,  the  common  output  indicator  is  ‘migrants,  participants  with  a  foreign 
background, minorities (including marginalised communities such as the Roma)’. In countries without significant 
migrant  inflows  one  can  assume  that  a  large  majority  of  the  participants  are  actually  Roma,  to  be  confirmed 
through the content of the specific investment priority in an operational programme. 

79 

 
 
 
 
 
 
                                                           
3. Description of key data sources, their limitations and mitigation measures 

FRA surveys341 

The  Roma  integration  indicators  scoreboard  (2011-2016)  accompanying  the  midterm 
review  of  the  EU  framework  presents  changes  in  the  situation  of  Roma  in  nine  EU 
Member  States,  as  recorded  by  two  FRA  surveys  in  2011  and  in  2016.  In  2016,  the 
second  European  Union  minorities  and  discrimination  survey  (EU-MIDIS  II 342 ) 
collected  information  on  the  situation  of  Roma  in  Bulgaria,  Croatia,  Czech  Republic, 
Greece, Hungary, Portugal, Romania, Slovakia and Spain, in total covering more than 80 % 
of the EU Roma population. The 2011 Roma survey343 covered the same countries apart 
from  Croatia.  However,  information  on  the  situation  in  Croatia  was  collected  in  the 
UNDP/World Bank/EC 2011 regional Roma survey344. The Member States included in 
the  FRA  surveys  represent  approximately  between  80 %  and  85 %  of  the  EU’s  Roma 
population. 

The  surveys  were  carried  out  using  a  similar  methodology,  applying  a  multi-stage 
selection  of  respondents.  To  optimise  the  sampling  approach,  EU-MIDIS II  refined  the 
methodology  applied  in  2011.  Despite  the  similar  approaches,  the  surveys  have  some 
limitations as to their direct comparability. In 2017, FRA addressed the limitations on the 
comparability of the surveys. Given the relative similarity of the unweighted samples of 
the 2011 and 2016 surveys for the nine Member States, the 2011 sample was weighted to 
reflect  the  differences  between  the  two  surveys  as  regards  regional  coverage  and  the 
urban  nature  of  surveyed  localities.  For  Croatia,  the  same  approach  was  applied  to  the 
dataset from the UNDP/World Bank/EC survey. 

The scoreboard presents 18 indicators in four main thematic areas (education, housing, 
employment  and  health)  and  the  cross-cutting  area  of  poverty.  It  also  presents  average 
values for the Member States in question. For 2011, the average does not include Croatia, 
which at that time was not a Member State. The caveats that need to be considered when 
analysing  values  for  2011  and  2016  are  provided  alongside  each  indicator.  All  sample 
surveys are affected by sampling error, as the interviews cover only a fraction of the total 
population.  Therefore,  all  results  presented  are  point  estimates  underlying  statistical 
variation.  Small  differences  of  a  few  percentage  points  between  groups  of  respondents 
are  to  be  interpreted  within  the  range  of  statistical  variation  and  only  more  substantial 
divergence  between  population  groups  should  be  considered  as  evidence  of  actual 
differences. A difference of a few percentage points between the 2011 and 2016 values 
may be assessed as ‘no change’. 

Survey  data  on  Roma  comes  with  many  caveats.  Sampling  of  Roma  is  difficult  due  to 
lack  of  reliable  data  on  the  actual  demographic  composition  or  geographic  distribution 
from  census  data345.  Capturing  migrant/mobile  Roma  populations  with  an  adequate 
sample  is  even  more  challenging.  Even  in  Member  States  where  Roma  constitute 

341 See  also  FRA  Roma  integration  scoreboard,  SWD(2017)  286  final  and  FRA  2018  ‘A  persisting  concern:  anti-

Gypsyism as a barrier to Roma inclusion’. 

342   http://fra.europa.eu/en/publication/2016/eumidis-ii-roma-selected-findings 
343   http://fra.europa.eu/en/survey/2012/roma-pilot-survey 
344   http://www.eurasia.undp.org/content/rbec/en/home/ourwork/sustainable-development/development-planning-

and-inclusive-sustainable-growth/roma-in-central-and-southeast-europe/roma-data.html 

345 During national censuses, typically more than a third of Roma across the EU do not declare their ethnicity. 

80 

 
 
 
 
 
 
                                                           
significant  parts  of  the  population  sampling  is  a  challenge:  similarly  to  census  data 
collection,  Roma  do  not  always  declare  their  ethnicity  but  rather  that  of  the  majority 
population or of other minority groups. The survey language may also be an issue, as the 
variety  of  Roma  languages  or  dialects  makes  it  difficult  to  conduct  a  single  survey  in 
‘Romani’. Surveys are typically in the national language, which may limit the responses 
from  Roma who face more exclusion due to  inadequate language skills.  Phone surveys 
may also lead to under-representation of the poorest Roma, who do not have access to a 
phone. As a result. the scale of certain problems may be overestimated or underestimated 
(depending on the issue346). 

Comparability  of  FRA  data  with  official  government  statistics  or  Eurostat  data  on  the 
majority population also is an issue. The 2011 FRA Roma survey collected data for the 
general population from non-Roma living nearby the areas surveyed to obtain the Roma 
data.  While  from  a  sociological  point  of  view  this  approach  is  correct  (as  it  makes  it 
possible to better compare gaps by accounting for regional or local social and economic 
disparities), it makes comparison of data more difficult.  

The  above  limitations  required  certain  mitigation  measures  to  be  adopted  when 
presenting or analysing the data: 

  Data  based  on  FRA  surveys  were  only  commented  on  if  changes  to  three 

 

percentage points outside of the statistical margin of error were apparent; 
Information from the FRA’s ‘online data explorer’ presenting data from the 2011  
Roma survey is not weighted and therefore cannot be used for comparison with 
information from the 2016 EU-MIDIS II, which used weighted data. 

UNDP/WB/EC regional Roma surveys 2011 and 2017347 

A  survey  carried  out  by UNDP,  the  World  Bank,  and  the  European  Commission in 
2011 interviewed 750 Roma and 350 non-Roma households living in  or  close to  Roma 
communities  in 12  countries  of  central  and  south-eastern  Europe.  The  survey  collected 
basic socio-economic data on households and on individual household members, as well 
as  perception  data  of  selected  adult  members  from  each  household.  The  2017  regional 
Roma survey348 was the first major collection of data on marginalised Roma in Albania, 
former  Yugoslavia  Republic  of  Macedonia, 
Bosnia  and  Herzegovina, 
Montenegro,   Serbia  and  Kosovo*  since  2011.  By  following  a  similar  procedure  to  the 
2011  survey,  the  2017  regional  Roma  survey  allows  for  a  level  of  comparability  over 
time. 

the 

Data on EU and national funding dedicated to Roma integration, in particular ESIF 

Information  about  exact  amounts  of  funding  specifically  spent  on  Roma  integration  is 
scarce.  With  regard  to  ESIF  2014-2020,  investment  priority  (IP)  9.2 349  covers 

346 For instance, some  more ‘integrated’ Roma or others aspiring upward social  mobility  may self-identify  with the 
ethnicity of the majority population. In other cases, respondents who are more sensitive to discrimination or have 
experienced more discrimination may also identify as the majority population to avoid stigmatisation. 

347 http://www.eurasia.undp.org/content/rbec/en/home/ourwork/sustainable-development/development-planning-and-

inclusive-sustainable-growth/roma-in-central-and-southeast-europe/roma-data.html 

348 http://www.eurasia.undp.org/content/rbec/en/home/library/roma/regional-roma-survey-2017-country-fact-

sheets.html 

349 ‘Socio-economic integration of marginalised communities such as the Roma’. 

81 

 
 
 
 
 
 
 
 
 
                                                           
marginalised communities in general (not exclusively Roma), so the exact amounts spent 
on  Roma  integration  cannot  be  identified.  Equally,  the  ESF  output  indicator  for 
participants  encompasses  ‘migrants,  participants  with  a  foreign  background,  minorities 
(including marginalised communities such as the Roma)’ and is therefore not sufficiently 
specific  to  provide  a  clear  picture.  The  2007-2013  programming  period  provides  even 
less  information,  taking  a  mainstream  approach  with  no  such  investment  priority  and 
indicator. 

Online Roma integration reporting tool 

Since  the  2015  reporting  year,  annual  reporting  by  Member  States  on  the  NRIS 
implementation  has  been  carried  out  online.  The  data  in  the  online  Roma  integration 
reporting  tool  in  particular  include  a  description  of  targeted  and  mainstream  policy 
measures implemented across 12 thematic areas in total. It also attempts to gather data on 
funding  and  beneficiaries.  At  the  time  of  the  evaluation,  data  for  2015  and  2016  were 
available from 24 Member States. 

To  have  a  comprehensive  view  of  the  cost  and  benefits  of  individual  measures 
implemented  in  all  Member  States,  it  would  have  been  necessary  to  rely  fully  on  the 
information  available  from  the  reporting  tool.  However,  serious  data  limitations  were 
identified  around  the  information  provided  in  the  reporting  tool.  These  are  reported 
below, in turn for costs and benefits. 

Costs 

  There  are  gaps  in  the  data,  as  some  Member  States  have  not  provided  any 

financial information. 

  Mainstream measures included in the reporting generally provide no estimate of 
the share of funding spent on Roma. Nor can it be assumed that the amount spent 
on  Roma  is  proportionate  to  the  share  of  the  Roma  population  compared  to  the 
general population. 

  There are limitations and inconsistencies as to how Member States report on the 
costs when they choose to report, in particular regarding which costs they include. 
For  example:  (i)  do  they  include  ‘implementation’  or  ‘management’  costs  of 
policy  measures?;  (ii)  are  all  costs  quantified?  Often  policy  measures  that  have 
long-term  financial  impact  are  immediately  quantified  —  e.g.  an  educational 
measure that may require more teachers or more schools at some future point in 
time. 

Benefits 

  The number of Roma beneficiaries was indicated by the Member States only for 
some of the measures implemented in 2015 and 2016. This is particularly the case 
for mainstream measures. 

  Outcome indicators are not reported in the standard reporting form for 2015 and 

2016. 

  There are numerous  social  and economic exogenous factors affecting direct  and 
indirect  benefits;  it  is  not  possible  to  attribute  them  directly  to  the  individual 
measures. 

82 

 
 
 
 
 
 
 
 
In  light  of  these  limitations,  this  evaluation  took  the  approach  not  to  use  the  reporting 
tool data350. 

CF to cot information under sensitivity analysi 

350 This was in contrast to the external study which, however, also concluded that there is a high degree of uncertainty 
about  the  values.  One  of  the  lessons  learnt  from  this  evaluation  is  the  need  to  improve  data  collection  and 
comparability in the reporting tool (see conclusions). 

83 

 
 
 
 
 
 
                                                           
ANNEX 5: COSTS AND BENEFITS 

a) The table below provides an overview of the costs and benefits identified during the 
evaluation  and  attributable  to  the  EU  framework,  followed  by  an  explanation  of  the 
calculations of administrative costs at national level: 

Table 12: Costs and benefits identified and directly attributable to the EU framework 
Costs for 

Type of cost 

Value 

EU budget 

Administrative costs (staff) 

EU budget 

EU  funding  supporting  measures 
directly  stemming  from  the  EU 
framework  

National budget 

Administrative costs (staff) 

Benefits for 
EU level 

Type of benefit 
as 
of 
Perception 
promoting  values,  equality  and 
Roma integration 

the  EU 

national 

EU, 
local 
stakeholders  working  on  Roma 
integration 

and 

Cooperation and capacity  

of 

six 

EUR 

DG  JUST  Roma  team  with  staff 
full-time 
consisting 
equivalents. 
Additional 
contributions via the Commission 
internal Roma task force. 
Estimate of around EUR 800 000 
annually  (in  particular  bi-annual 
NRCP  meetings;  co-financing  of 
national  Roma  platforms;  annual 
EU Roma  platform; development 
and  maintenance  of  the  reporting 
tool). 
Quantitative estimate of  
EUR 3 300 000 
annually351 
(staff  required  for  compliance 
with  the  EU  framework  in  terms 
of 
and 
implementation 
monitoring of NRIS). 
Value 
Qualitative — the evaluation (EU 
added  value)  found  that  the  EU 
framework: 
put  Roma 
(i) 
inclusion on the political EU and 
raised 
national  agendas; 
attention  to  Roma  issues  also  in 
countries  with  smaller  Roma 
populations;  and  (iii)  ensured  a 
stable  focus  on  Roma  integration 
through its multiannual character. 
Qualitative  — 
the  evaluation 
found  increased  cooperation  and 
capacity  of  stakeholders  working 
on  Roma 
in 
particular  through  the  setting  up 
and  strengthening  of  instruments 
Roma 
for 
structures 
and 
integration,  but  also 
through 
increased  awareness  of  Roma 
issues. 

integration, 

(ii) 

Explanation and limitations of the calculation of administrative costs at national level: 

351 ICF, Chapter 3.6.2 — see limitations in Annex 4 to this SWD. 

84 

 
 
 
 
 
 
 
                                                           
The  tasks  undertaken  by  the  NRCPs  are  varied.  They  include  policy  coordination, 
monitoring  how  NRIS  are  implemented,  compiling  reports,  coordinating  and  meeting 
with  local  authorities  and  other  actors  supporting  Roma  integration,  running  Roma 
platforms  and  engaging  with  the  Roma  community.  Member  States  with  larger 
populations  of  Roma  tend  to  have  a  higher  number  of  staff  (and  costs)  devoted  to  the 
NRIS. Using the standard cost model approach, the interviews carried out by the external 
consultant  with  NRCPs  and  analysis  of  information  in  the  online  reporting  tool  were 
intended to provide clarity about the time required to carry out all activities related to the 
EU framework. These include reporting to the EU level and implementing the NRIS in 
each Member State.  
During  the  interviews,  only  nine  Member  States  could  provide  information  on  the  of 
number of staff and time spent on NRIS implementation and monitoring.  Based on the 
responses and information provided in the reporting tool, in total, the number of people 
performing  this  task  was  estimated  to  be  164  individuals.  The  number  of  people  was 
multiplied  by  an  average  labour  cost  for  public  administrative  and  support  services  in 
Member  States  (hourly  labour  cost)  and  the  average  number  of  hours  worked,  also 
calculated on the basis of the interviews. In total, the administrative cost was estimated to 
be EUR 3.3 million annually, primarily driven by a high number of staff reported in three 
Member States (SK, HU and IT). Overall, however, the number of staff contributing to 
implementation  and  monitoring  of  NRISs  is  likely  to  be  an  underestimate,  mostly 
because  additional  individuals  and  organisations  are  involved,  in  particular  at  regional 
and local level. 

b) In addition to the above, the following table provides and overview of costs and 
benefits  of  Roma  integration  not  directly  or  exclusively  attributable  to  the  EU 
framework or not yet identified because they are long-term costs and benefits. 

Table 13: Costs and benefits of Roma integration not attributable or not identified 
Costs for 
EU budget 

Roma 

Value 
Not directly attributable 
No  sufficiently  reliable  data  covering 
2011-2016  available  (see  Annex  4  for 
explanations) 

for 

Type of cost 
EU 
funding 
integration measures: 
-  Funding  for  Roma  integration 
measures  under  ESIF  (and  its 
predecessors) 
-  Funding  for  Roma  integration 
measures 
other  EU 
under 
programmes 

National budget 

National 
integration measures  

funding 

for  Roma 

Benefits for 
Roma 

Type of benefit 
Less discrimination 

85 

Not directly attributable 
No  sufficiently  reliable  data  covering 
2011-2016 available (limitations described 
in Annex 4) 
Value 
Not directly attributable 
Less discrimination on grounds of race and 
ethnicity  has  a  wide  range  of  interrelated 
impacts, 
including  better  employment 
conditions,  better  educational  outcomes, 
less  criminal  victimisation  including  hate 
crimes and harassment. Individual impacts 
of  less  discrimination  can,  for  example, 
take  the  form  of  increased  earnings,  less 
risk  of  physical  assault,    better  physical 
less  mental  health 
health  status  and 

 
 
 
 
 
 
education, 

Increased participation in quality 
an 
mainstream 
increased proportion of Roma in 
paid  employment  and  increased 
earnings 
from 
participation. 

resulting 

problems352.  
Not directly attributable 

of 

being 

probability 

Education: 
Participation  in  ECEC  increased  between 
2011 and 2016 in six Member States (BG, 
EL,  ES,  HR,  HU,  SK).  According  to 
specialist  literature  in  the  field,  children 
who  remain  in  education  longer  have  an 
increased 
in 
employment  and  of  higher  average 
earnings once they are employed. 
The  share  of  early  leavers  from  education 
and 
in  seven 
Member States (BG, CZ, ES, HU, PT, RO, 
SK).  Early  school  leavers  face  barriers  in 
entering  the  labour  market  (obtaining  a 
job)  and  on  average  have  lower  earnings 
once they are employed. Additionally, their 
levels  of  health,  wellbeing  and  social 
participation  are  lower  than  those  who 
remain  in  school.  This  in  turn  generates 
additional social costs. 

training  has  decreased 

No  quantification  of  benefits  was  carried 
out for the reasons explained below. 

Employment 
that  occurred 
The  positive  changes 
between  2011  and  2016  have  been 
quantified353 as follows: 

for 

HU 

in  an 

additional 

in  paid 
The  proportion  of  Roma 
employment  increased  in  PT  and  HU, 
resulting 
increase  of  55 500 
employed  Roma  in  HU  and  6 800  in  PT. 
This  number  was  then  multiplied  by  the 
for  Roma, 
average  annual  earnings 
wages 
resulting 
in 
(EUR 209 300 000 
and 
EUR 56 200 000  for  PT).  These  in  turn 
were multiplied by the average tax rate for 
the  two  Member  States  to  estimate  the 
increase  in  tax  receipts  (EUR 59 300 000 
for HU and EUR 14 200 000). 
The only Member State with a decrease in 
the  share  of  NEETs  among  the  Roma 
between 2011 and 2016 is PT (from 79 % 
to  52 %)  resulting  in  1 500  fewer  NEETs 
in that country. The costs of being a NEET 
as  identified  by  Eurofound  relate  to  the 
costs 
(benefit 
the  public  purse 
payments)  and  private  costs  (lack  of 
income  generated).  The  decrease  in  the 
in  PT  has  been 
number  of  NEETs 

to 

352 Compare European Parliamentary Research Service; Cost on Non-Europe Report ‘Equality and the Fight against 
Racism and Xenophobia’, March 2018; based on conceptual framework for the impacts of discrimination 
presented by Milieu (see below). 
353 See explanations and limitations below 

86 

 
 
 
 
 
 
 
                                                           
National budget 

Increased  tax  receipts  through 
increased earnings of Roma 

to 

of 

benefit 

to 
around 

the  cost  per  NEET 

multiplied  by 
estimate 
a 
EUR 12 000 000. 
Not  directly  attributable 
framework 
Additional  wages  (EUR 209  300  000  for 
HU  and  EUR 56  200  000  for  PT)  were 
multiplied  by  the  average  tax  rate  for  the 
two Member States to estimate the increase 
in  tax  receipts  (EUR 59  300  000  for  HU 
and EUR 14 200 000 for PT). 

the  EU 

Savings in health care provision 

Not  identified  —  future  savings  expected 
via  more  health  prevention  and  earlier 
access to health services 

National budget 

Employers 

Macroeconomy 

Society 

Savings  in  unemployment  and 
other social benefits 

Not identified 

Higher  public  revenues  due  to 
higher  income  and  consumption 
(VAT) tax payment 

Not identified 

from 

Benefits 
increased  and 
skilled  labour,  in  particular  in 
light  of  adverse  demographic 
developments 

growth; 

productivity 
GDP 
growth  through  upskilling  of  a 
previously excluded labour force 
Longer  term  social  and  political 
cohesion,  tolerance,  integration, 
equality,  diversity; 
  improved 
inter-ethnic  relations,  increased 
security and stability 

sustainability 

of 
Improved 
pension  systems  with  upskilled 
Roma 
labour  market  entrants 
counterbalancing  ageing  non-
Roma societies 

Not identified 

Not identified 

Not identified 

Not identified 

Explanations: 

a) Quantification of EU and national funding for Roma integration 

To ensure that the evidence on funding has been explored as comprehensively as possible, 
the following information sources have been investigated: 

87 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  The online reporting tool, in which Member States provide information on EU 

(ESIF only) and national funding for Roma integration. 

  ESF/ERDF  Member  State  operational  programmes  implementation  reports 
(2007-2013  and  2014-2020)  to  identify  projects  and  funding  targeting  the 
Roma and the number of beneficiaries targeted. This included examining the 
project  descriptions  to  identify  financial  information  outside  programme 
indicators, which were found to be mostly incomplete. 

  The  Instrument  for  Pre-Accession  Assistance  (IPA)  evaluation,  to  establish 

funding through IPA for Roma. 

  Other European funding programmes, such as  PROGRESS, ERAMUS+, the 
Fund  for  European  Aid  to  the  Most  Deprived  (FEAD)  and  the  Rights, 
Equality and Citizenship (REC) programme, Horizon 2020. 

  National research to identify any further funding streams. 
  A  review  of  interviews  and  survey  responses,  where  some  funding 

information is provided. 

Despite examining all these different information sources, it was not possible to robustly 
identify  the  current  or  previous  levels  of  expenditure  on  Roma  integration  due  to  the 
limitations set out above. However, as highlighted in the effectiveness, EU value added 
and coherence sections, EU funding has overall been well aligned with the objectives of 
the  EU  framework.  This  is  particularly  the  case  for  the  ESF,  the  IPA,  the  REC 
Programme  and  the  European  Instrument  for  Democracy  and  Human  Rights  (EIDHR). 
This  is  acknowledged  in  this  evaluation  by  presenting  the  data  that  follows  even  if 
insufficient to draw overall conclusions. 

ESF allocation to IP 9.2 targeting Roma explicitly but not exclusively 

Table 14: Total 2014-2020 ESF and IP 9.2  allocations and Roma population 

88 

 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
TOTAL 
ESF (€m) 

AT  553 
BE  407 
BG  1 200 
CZ   1 900 
ES  3 500 
EL   1 300 
HU  2 400 
FR   3 200 
 IT  4 000 
 PL   19 000 
 RO   3 400 
 SK  1 400 
42 260 

Total allocation 
to IP 9.2 (€m) 
4 
10 
143 
200 
48 
73 
470 
8 
71 
19 
372 
99 
1 518 

9.2 allocation as a 
% of total ESF 
0.7 % 
2.4 % 
11.9 % 
10.5 % 
1.4 % 
5.6 % 
19.6 % 
0.3 % 
1.8 % 
0.1 % 
10.9 % 
7.1 % 
3.6 % 

Share of Roma in 
total population 
0.4 % 
0.3 % 
9.9 % 
1.9 % 
1.6 % 
1.6 % 
7.4 % 
0.6 % 
0.3 % 
0.1 % 
8.6 % 
9.0 % 

Total 
Source: European Court of Auditors (2016) EU policy initiatives and financial support for Roma 
integration: significant progress made over the last decade, but additional efforts needed on the 
ground 

Instrument for Pre-Accession Assistance (IPA)354 

The EU provides funding to  countries  which are candidates and potential  candidates to 
join the EU. The funding can be used to fund projects aiming to improve the following 
topics, which may include spending on the Roma: 

- regional development (transport, environment, regional and economic development) 
- human resources (strengthening human capital and combating exclusion) 
- rural development. 

Table 15:  IPA  I  — period 2007-2013 (actual  amount spent, subject  to update based on 
final reporting) 

Country 

Albania 

Bosnia and Herzegovina 

former Yugoslav Republic of Macedonia 

Kosovo 

Montenegro 

Serbia  

Turkey 

Croatia 

354 Data provided by Commission departments. 

89 

IPA I spent *) **) ***) 
(2007-2013) 
M€ 

2.72 

9.21 

9.55 

15.27 

5.67 

43.72 

16.25 

6.44 

 
 
 
 
 
                                                           
  
 
 
People to People (P2P) programme 

TAIEX 

Multi-country 

Source: Commission, July 2018 

TOTAL 

0.26 

12.72 

121.80 

*) Actual amount spent 2007-2013, continuously updated based on implementation reports 
**)    The  estimates  are  based  on  the  Thematic  Evaluation  on  IPA  Support  to  Roma  Communities.  The  report  was 
published  in  June  2015  as  the  final  deliverable  of  Contract  No 2014/344098/1.  The  evaluation  referred  to  the  2007-
2013  IPA  programme  and  estimated  the  contribution  to  Roma  inclusion  in  eight  enlargement  countries,  including 
Croatia which at the time was a candidate country. In 2017, the figures on the approx. 80 identified Roma inclusion 
IPA I projects were updated based on reporting on actual project implementation in the enlargement countries. Some of 
the  IPA  I  Roma  integration  projects  are  still  being  implemented  and  the  figures  will  be  updated  pending 
implementation reports; ***) These figures are subject to updates based on IPA implementation reports. 

Table 16: IPA II — indicative allocations for assistance to Roma integration during the 
first half of IPA II 2014-2016*) **) ***) 

Country 

Albania 

Bosnia and Herzegovina 

former Yugoslav Republic of Macedonia 

Kosovo 

Montenegro 

Serbia  

Turkey 

People to People (P2P) programme 

TAIEX 

Multi-country 

TOTAL 

Source:  Commission, July 2018 

IPA II estimates  
(2014-2016) 
M€ 

8.85 

8.78 

8.42 

13.46 

6.94 

24.75 

22.00 

0.15 

0.30 

18.85 

112.50 

*) Actual amount spent 2014-2016; **) These figures are subject to updates based on IPA 
implementation reports . Updated figures will be included in future EU reports. 

Rights, Equality and Citizenship Programme (REC) 

Since  2014,  the  REC  programme  has  funded  43  projects  specifically  addressing  the 
Roma.  Their  main  focus  was  on  promoting  non-discrimination  and  on  promoting 
cooperation  at  national  level  through  co-funding  of  national  Roma  platforms.  These 
projects received EUR 6.9 million of EU funding (with total funding of EUR 8.4 million). 

90 

 
 
  
 
 
 
 
 
REC represents both a higher number of projects and a higher average annual spending 
on  Roma  compared  to  the  previous  programming  period  and  the  related  spending  for 
Roma projects under PROGRESS. 

European Instrument for Democracy and Human Rights 355 

The  European  Instrument  for  Democracy  and  Human  Rights  (EIDHR)  is  a  thematic 
funding  instrument  for  EU  external  action  aiming  to  support  projects  on  human  rights, 
fundamental freedoms and democracy in non-EU countries. This instrument is designed 
to support civil society to become an effective force for political reform and defence of 
human rights. 

Table  17:  EIDHR  allocations  for  assistance  to  Roma  integration  2007-2013*)  and 
2014-2016 **) ***) 

Country 

Albania 

Bosnia and Herzegovina  

former Yugoslav Republic of Macedonia 

Kosovo 

Montenegro  

Serbia 
Turkey 

TOTAL 

EIDHR  
(2007-
2013) 
M€ 

EIDHR 
(2014-
2016) 
M€ 

EIDHR 
(2007-
2016) 
M€ 

2.04 

 0.28 

0.68 

 0.10 

3.10 

1.25 

0.31 

0.36 

0.23 

0.23 
1.35 

3.73 

0.00 

3.29 

 0.31 

0.64 

0.91 

0.33 
1.35 

6.83 

Source: Commission, July 2018 
*) Actual amount spent 2007-2013; **) Actual amount spent 2014-2016; ***) These 
figures are subject to updates based on implementation reports. Updated figures will be 
included in future EU reports.  

b) Quantification of benefits resulting from actual  progress  made in between 2011 
and 2016 in education, employment, housing and health 
Even if not directly attributable to the EU framework, an attempt was made to quantify 
benefits  resulting  from  actual  progress  between  2011  and  2016  for  the  following 
indicators used in the FRA surveys: 

  early childhood education and care (4 to compulsory schooling age); 
  early leaving from education and training; 
  self-declared main activity status ‘paid work’; 
  share  of  young  people  aged  16-24  years  neither  in  employment,  education  or 

training (NEETs). 

355 Data provided by Commission departments. 

91 

 
 
 
 
 
  
 
 
  
 
 
 
 
  
 
 
 
 
                                                           
The  evaluation  found  that  research  has  been  done  on  the  economic  returns  from  social 
investment in general, but that there is scant research on returns of investment in Roma 
integration  more  specifically.  Transferability  of  this  research  is  thus  restricted,  in 
particular as socio-economically marginalised groups such as the Roma are considered to 
be further from the labour market than other groups in the population. Potential sources 
were identified and their transferability for the evaluation assessed, as set out below. 

Education 

The positive effects of early childhood education and care (ECEC) have been explored in 
several  studies 356 .  Children  who  remain  in  education  longer  have  an  increased 
probability  of  being  in  employment  and  of  higher  average  earnings  once  they  are 
employed.  Research  has  estimated  the  effect  of  an  individual  attending  pre-primary 
education to be equivalent to an increase in the PISA score in mathematics of 51 points. 
This is equivalent of more than a year of formal schooling (PISA 2013357). 

In  turn,  the  effect  of  an  individual’s  extra  year  in  education  is  estimated  to  be  an 
additional 10 % of earnings358. Using this increase in earnings would assume that the 
quality  and  quantity  of  ECEC  provided  for  Roma  children  is  comparable  to  the 
quality and quantity of ECEC for the general population. This evaluation finds that 
there  is  limited  transferability  of  the  above  quantification  to  the  particular  situation  of 
Roma and therefore refrains from using it to quantify benefits359. Other benefits of early 
childhood  education  according  to  the  specialist  literature  in  the  field  include:  (i)  lower 
dropout  rates,  improved  health  and  social  wellbeing;  (ii)  increased  labour  force 
participation;  (iii)  reduced  welfare  dependency;  (iv)  increased  tax  receipts;  and  (iv) 
reduced crime rates360. 

Early school leavers face barriers in entering the labour market (i.e. obtaining a job) and 
on  average  have  lower  earnings  once  they  are  employed.  Additionally,  their  levels  of 
health,  wellbeing  and  social  participation  are  lower  than  those  who  remain  in  school. 
This in turn generates additional social costs361. Research into early school leaving (ESL) 
and  its  cost  sometimes  uses  estimates  of  the  cost  of  being  NEET  as  a  proxy 362 . 
Following this approach, the change in the number of Roma who are early school leavers 
could  have  been  multiplied  by  the  annual  cost  of  being  NEET  to  estimate  the  costs  or 
benefits  of  changes  in  ESL  in  the  Roma  population.  Again,  however  the  mentioned 
research is not Roma-specific and the transferability of the data is questionable. 

Employment 

356 Magnuson and Duncan (2014), ‘Can early childhood interventions decrease inequality of economic opportunity?’; 

Heckman, J.J (2006), Skill formation and the economics of investing in disadvantaged children. 

357 Janna van Belle (2011) Early Childhood Education and Care (ECEC) and its long-term effects on educational and 

labour market outcomes 

358  West, A (2016) L’école maternelle à la source de la réduction des inégalités sociales: une comparaison 
internationale;  Unesco (2011), Education Counts, Toward the Millennium Development Goals, p. 6. 

359 In contrast to the external evaluation study. 
360 Janna van Belle (2011) Early Childhood Education and Care (ECEC) and its long-term effects on educational and 

labour market outcomes 

361 Brunello, G and De Paola, M (2013) The costs of early school leaving in Europe; the study found that these costs 
are private, fiscal and social. Costs due to lost private benefits include the expected gains in earnings and wealth, 
improved health and life expectancy and higher lifetime satisfaction. Costs related to lost fiscal benefits include 
increased tax payments, lower reliance on government transfers and reduced expenditures on criminal justice. 
Social costs related to lost social benefits include productivity externalities, the social value of better health and 
the gains from reduced crime. 

362 See for example European Commission: Overview and examples of costs of early school leaving in Europe 

92 

 
 
 
                                                           
The  quantifications  below  are  based  on  developments  in  the  two  Member  States  with 
positive  changes.  They  assume  stability  in  the  size  of  the  specific  age  cohorts  between 
2011  and  2016  and  have  been  carried  out  to  illustrate  benefits  to  be  gained  if  more 
progress was made, including in other countries. 

a) Changes in the share of Roma in paid employment 

By increasing the share of Roma in paid employment, there are benefits to be gained in 
terms of increased earnings of Roma and increased tax receipts for the state budget. 

The  increase  in  the  share  of  Roma  in  paid  employment363 between  2011  and  2016  in 
Hungary (from 25 % to 36 %) and Portugal (from 14 % to 34 %) was multiplied by the 
Roma working-age population364 in  the two countries to  estimate the difference in  the 
numbers of Roma workers. This resulted in an increase of 55 500 in Hungary and 6 800 
in Portugal. This number was then multiplied by the average annual earnings for Roma 
(taken  from  UNDP-WB-European  Commission  2011  data  and  inflated  to  2016  prices 
using GDP deflators, resulting in EUR 3 800 for Roma in HU and EUR 8 300 for Roma 
in Portugal). The additional wages (EUR 209 300 000 for Hungary and EUR 56 200 000 
for  Portugal)  were  multiplied  by  the  average  tax  rate  for  the  two  Member  States  to 
estimate  the  increase  in  tax  receipts  (EUR 59 300 000  for  Hungary  and  14 200 000  for 
Portugal). 

b)  Changes  in  the  share  of  young  people  aged  16-24  years  neither  in  employment, 
education or training 

According  to  Eurofound  estimations  the  cost  to  the  economy  of  NEETs  was  over 
EUR 150 billion in Europe in  2011 (more than 1.2 % of EU GDP). The countries most 
affected  were  Bulgaria  and  Greece365,  where  there  are  significant  Roma  populations. 
The  Eurofound  research  also  presents  annual  costs  of  NEETs  by  Member  State.  These 
costs,  which  are  presented  in  the  table  below,  relate  to  the  costs  to  the  public  purse 
(benefit payments) and private costs (lack of income generated). 

Table 18: Cost of NEETs 

MS 

BG 

HR 

CZ 

EL 

HU 

PT 

RO 

Cost of NEETs (€m)  Number of NEETs  Cost  per  NEET 

837 

- 

1 493 

4 043 

1 632 

2 131 

1 181 

468 400 

- 

295 400 

416 300 

375 400 

264 600 

706 600 

2008 

1 800 

1 800 

5 100 

9 700 

4 300 

8 100 

1 700 

Cost  per  NEET 
2016 

2 000 

1 900 

5 100 

10 100 

4 700 

8 200 

1 900 

363 Figures are for Roma aged 16+ 
364 Age group 16-64; Source: World Bank 
365 Eurofound (2015), Young People and ‘NEETs’. 

93 

 
 
 
 
 
                                                           
SK 

ES 

517 

10 794 

126 800 

1 029 300 

4 100 

10 500 

4 100 

10 500 

Eurofound (2012) Young people not in employment, education or training: Characteristics, costs and policy responses 
in Europe 

The only Member State with a decrease in the share of NEETs among the Roma between 
2011 and 2016 is Portugal (from 79 % to 52 %), resulting in 1 500 fewer NEETs in that 
country. This decrease has been multiplied by the cost per NEET to estimate a benefit of 
around EUR 12 000 000. 

Long-term benefits from Roma inclusion 

This SWD does not attempt to quantify the long-term benefits resulting from closing the 
education, employment and earnings gaps between Roma and non-Roma. This is because 
long-term projections fall outside the scope of the evaluation. Attempts to quantify this 
have been made in the past366 but are based on a number of assumptions including equal 
educational  attainment  levels,  equal  access  to  employment,  equal  pay  and  equal 
productivity levels. 

366 See in particular World Bank (2010) — Roma Inclusion: An economic opportunity for Bulgaria, Czech Republic, 
Romania and Serbia; The World Bank (2010) used survey data from 6 EU MS to compare average working-age 
Roma  and  non-Roma.  It  built  a  model  that  draws  on:  (i)  the  probability  of  employment;  and  (ii)  the  average 
wage,  conditional  on  employment  of  both  groups.  The  difference  between  the  average  expected  earnings  for 
Roma and non-Roma is the average earnings gap per working-age individual. Total productivity losses are then 
calculated  using  the  estimated  number  of  Roma  working-age  individuals.  Using  the  same  data  on  wages  and 
probability  of  employment,  the  World  Bank  estimated  (partially)  the  fiscal  contribution  opportunity  costs  of 
Roma exclusions, i.e. — the forgone income tax and social security payments. 

94 

 
 
 
 
                                                           
 ANNEX 6: OBJECTIVES AND MEASURES 2011 AND 2013 

Policy focus 

Discrimination 

Poverty 

Education 

What does the 2011 
Communication say? 
COM(2011) 173 of 05 April 
2011 on an EU framework for 
national Roma integration 
strategies;  
Member States need to ensure 
that Roma are not discriminated 
against but treated like any 
other EU citizen with equal 
access to all fundamental rights. 
They are already under an 
obligation, laid down in 
Directive 2000/43/EC, to give 
Roma non-discriminatory 
access to education, 
employment, vocational 
training, healthcare, social 
protection and housing. 
Action is needed to break the 
vicious cycle of poverty moving 
from one generation to the next. 
Strong and proportionate 
measures are still not yet in 
place to tackle the social and 
economic problems of a large 
part of the EU’s Roma 
population. Non-discrimination 
is not sufficient to combat the 
social exclusion of Roma. 
Roma integration goal 
Access to education: Ensure 
that all Roma children 
complete at least primary 
school (minimum 
requirement) 

What should be done? 

  All Roma children 

should have access to 
quality education. 
  No Roma children 

should be subject to 
discrimination or 
segregation, regardless 
of whether they are 
sedentary or not. 
  Member States should 
widen access to quality 
ECEC. 

  Member States should 

95 

What does the 2013 Council 
Recommendation say? 
Council Recommendation 
(EPSCO) of 9 December 2013 on 
effective Roma integration 
measures in the Member States 

Recommends that Member 
States take effective measures to 
ensure equal treatment and full 
access to quality and mainstream 
education and ensure that all 
Roma pupils complete at least 
compulsory education. 

States that this goal could be 
attained through measures such 
as those listed below. These 
measures include and go beyond 
those set out in the 
Communication: 
(a) eliminating any school 
segregation; 
(b) putting an end to any 
inappropriate placement of Roma 
pupils in special needs schools; 
(c) reducing ESL throughout all 
levels of education, including at 
secondary level and vocational 

 
 
 
 
 
 
 
 
 
Employment 

reduce the number of 
early school leavers 
from secondary 
education. 

  Young Roma should 
also be strongly 
encouraged to 
participate in secondary 
and tertiary education. 

Roma integration goal 
Access to employment: Cut 
the employment gap between 
Roma and the rest of the 
population 

What should be done? 

  Member States should 
grant Roma people full 
non-discriminatory 
access to vocational 
training, to the job 
market and to self-
employment tools and 
initiatives. 

 

  Access to microcredit 
should be encouraged. 
In the public sector, due 
attention should be 
given to employment of 
qualified Roma civil 
servants. 

96 

training; 
(d) increasing access to, and 
quality of, ECEC, including 
targeted support, as necessary; 
(e) considering the needs of 
individual pupils and addressing 
those accordingly, in close 
cooperation with their families; 
(f) using inclusive and tailor-made 
teaching and learning methods, 
including learning support for 
struggling learners and measures to 
fight illiteracy, and promoting the 
availability and use of 
extracurricular activities; 
(g) encouraging greater parental 
involvement and improving 
teacher training, where relevant; 
(h) encouraging Roma 
participation in and completion of 
secondary and tertiary education; 
(i) widening access to second-
chance education and adult 
learning, and providing support for 
the transition between educational 
levels and support for the 
acquisition of skills that are 
adapted to the needs of the labour 
market. 
Recommends that Member 
States take effective measures to 
ensure equal treatment of Roma 
in access to the labour market 
and to employment 
opportunities. 

States that this goal could be 
attained by means of measures 
such as those listed below. These 
measures include (with the 
exception of microcredit) and go 
beyond those set out in the 
Communication. 

(a)  supporting first work 

experience, vocational 
training, on-the-job 
training, lifelong learning 
and skills development; 

(b)  supporting self-

employment and 
entrepreneurship; 

(c)  providing equal access to 

mainstream public 
employment services, 
alongside services to 

 
 
 
 
 
 
Health 

Housing 

support individual job-
seekers, focusing on 
personalised guidance and 
individual action planning 
and, where appropriate, 
promoting employment 
opportunities within the 
civil service; 

(d)  eliminating barriers, 

including discrimination, 
to (re)entering the labour 
market. 

Recommends that Member 
States take effective measures to 
ensure equal treatment of Roma 
in access to universally available 
healthcare services on the basis 
of general eligibility criteria. 

States that this goal could be 
attained by means of measures 
such as those listed below. These 
measures include and go beyond 
those set out in the 
Communication: 
(a) removing any barriers to access 
to the healthcare system accessible 
for the general population; 
(b) improving access to medical 
check-ups, prenatal and postnatal 
care and family planning, as well 
as sexual and reproductive 
healthcare, as generally provided 
by national healthcare services; 
(c) improving access to free 
vaccination programmes targeting 
children and vaccination 
programmes targeting in particular 
groups most at risk and/or those 
living in marginalised and/or 
remote areas; 
(d) promoting awareness of health 
and healthcare issues. 
Recommends that Member 
States take effective measures to 
ensure equal treatment of Roma 
in access to housing. 

States that this goal could be 
attained by means of measures 
such as those listed below. These 
measures include and go beyond 
those set out in the 
Communication: 

Roma integration goal 
Access to healthcare: Reduce 
the gap in health status 
between the Roma and the 
rest of the population 

What should be done? 

  Member States should 
provide the Roma with 
access to quality 
healthcare, especially 
for children and 
women, as well as 
preventative care and 
social services at a 
similar level and under 
the same conditions as 
for the rest of the 
population  
  Where possible, 

qualified Roma should 
be involved in 
healthcare programmes 
targeting their 
communities. 

Roma integration goal 
Access to housing: Close the 
gap between the proportion of 
Roma with access to housing 
and to public utilities (water, 
electricity, gas) and that of the 
rest of the population 

What should be done? 

  Member States should 

97 

 
 
 
 
 
 
 
 
 
promote non-
discriminatory access to 
housing, including 
social housing. 

  Member States should 
address the particular 
needs of non-sedentary 
Roma (e.g. provide 
access to suitable 
halting sites for non-
sedentary Roma). 

Structural 
preconditions/horizontal 
measures 

When developing their NRIS, 
Member States should bear in 
mind the following 
approaches: 

  Set achievable national 

goals for Roma 
integration to bridge the 
gap with the general 
population. These 
targets should, as a 
minimum, address the 
four EU Roma 
integration goals. 
  NRIS to identify 

disadvantaged micro-
regions or segregated 
neighbourhoods where 
communities are most 
deprived. 

  Allocate sufficient 

funding from national 
budgets, complemented 
where appropriate by 
international and EU 
funding. 

  NRIS to include strong 
monitoring methods to 
evaluate the impact of 
Roma integration 
actions and a review 
mechanism for the 
adaptation of the 
strategy. 

  NRIS to be designed, 
implemented and 
monitored in close 
cooperation and 
continuous dialogue 
with Roma civil 

98 

(a) eliminating any spatial 
segregation and promoting 
desegregation; 
(b) promoting non-discriminatory 
access to social housing; 
(c) providing halting sites for non-
sedentary Roma, in proportion to 
local needs; 
(d) ensuring access to public 
utilities (such as water, electricity 
and gas) and infrastructure for 
housing in compliance with 
national legal requirements. 

Funding: 

Recommends that Member States 
allocate adequate funding to 
implement and monitor their 
national and local strategies and 
action plans from any available 
sources of funding (local, national, 
Union and international). The aim 
is to achieve the objective of Roma 
integration through mainstream or 
targeted measures. 

Recommends that Member States 
target the allocation of public 
funding for implementing NRIS or 
integrated sets of policy measures 
to the specific needs of Roma, or 
to the geographical areas most 
affected by poverty and social 
exclusion. Member States should 
also take into consideration the 
gender dimension. 

Recommends horizontal policy 
measures: 

measures: 
Anti-discrimination 
enforcement  of  Directive  2000/43; 
carry  out  desegregation  measures; 
ensure  that  forced  evictions  are  in 
law; 
compliance  with 
EU 
to  combat 
implement  measures 
Roma, 
against 
prejudice 
sometimes 
as 
referred 
in  all  areas  of 
antigypsyism, 
society.  This  should  be  followed 
by  examples  of  such  measures 
(combat  anti-Roma  rhetoric  and 
hate  speech,  raise  awareness  of 
benefits of Roma inclusion etc.) 

to 

 
 
 
 
 
 
 
 
 
 
 
society, regional and 
local authorities. 
  Appoint a national 

contact point for the 
national Roma 
integration strategy 
with the authority to 
coordinate the 
development and 
implementation of the 
strategy or, where 
relevant, rely on 
suitable existing 
administrative 
structures. 

99 

Includes 

Protection  of  Roma  children  and 
women  (fight  violence,  including 
domestic 
trafficking, 
violence, 
underage  and  forced  marriage, 
begging involving children). 
Poverty  reduction  through  social 
investment. 
the 
recommendation:  ‘depending  on 
the  size  and  social  and  economic 
Roma 
situation 
consider  making 
populations, 
Roma 
important 
issue  within  their  national  reform 
programmes or their national social 
reports in the context of the Europe 
2020 Strategy.’ 

integration  an 

their 

of 

Empowerment  (active  citizenship 
of Roma by promoting their social, 
economic,  political  and  cultural 
participation  in  society,  including 
at the local level etc.) 

Recommends 
the 
structural measures: 

following 

monitor 

to  work 
the 

Local  action  (local  action  plans 
with  baselines,  benchmarks  and 
objectives,  strengthen  capacity  of 
local  authorities 
in 
cooperation  with 
families 
concerned). 
and 
Appropriately 
evaluate 
effectiveness  of 
the 
national  strategies  or  integrated 
sets  of  policy  measures  within 
broader  social  inclusion  policies. 
This  could  be  done 
through 
measures  such  as  setting  baselines 
or  measurable 
targets  or  by 
collecting  relevant  qualitative  or 
quantitative  data  on  the  social  and 
economic effects of such strategies 
or measures, in line with applicable 
national  and  EU  law,  particularly 
regarding 
of 
personal  data.  Make  use  of  any 
relevant core indicators or methods 
of empirical social research or data 
collection 
for  monitoring  and 
evaluating  progress  on  a  regular 
basis, particularly at the local level, 
enabling  efficient  reporting  on  the 
situation  of  Roma  in  the  Member 

protection 

the 

 
 
 
 
 
 
States, with the optional support of 
the  European  Union  Agency  for 
Fundamental Rights. 

Support the work of bodies for the 
treatment 
promotion  of  equal 
(promote  regular  dialogue  with 
NRCPs;  granting  them  adequate 
resources) 

NRCPs  should  have  adequate 
mandate  and  resources  so  that 
they  can  coordinate  the  cross-
sectoral  monitoring  of  Roma 
integration 
involve 
decision-making 
NRCPs 
processes 
the 
and 
development, 
implementation of relevant policies 

regarding 
funding 

policies; 

in 

Transnational cooperation 

100 

 
 
 
 
 
 
 
 
 
ANNEX 7A: INTERVENTION LOGIC I 

RATIONALE FOR INTERVENTION 
(needs) 
• 

Prevent  socio-economic  exclusion 
and discrimination of Roma 
•  Reap long-term societal benefits 

from Roma inclusion 

OBJECTIVES 

• 

Fight discrimination, combat 
poverty and improve Roma 
access to quality inclusive 
mainstream education, 
employment, healthcare and 
housing. 

EXTERNAL FACTORS 
 
  Refugee crisis 
  Domestic political situation — rise of 

Economic and labour market situation 

 

nationalist parties 
Social media and discriminatory 
rhetoric 

Effects 

IMPACTS 
 

Sustainably improved socio-
economic inclusion of Roma; 
sustainably reduced gaps 
between Roma and non-
Roma 
Economic, fiscal, and social 
benefits to EU societies 

Relevance 

 

Effectiveness 

EU/ National/ local interventions for Roma integration 

INPUTS 
Instruments: 
European 
• 
• 
• 

• 
• 

• 

EU Framework for NRIS 
Council recommendation on effective Roma integration measures 
European legislation: Racial Equality Directive, Council Framework Decision on 
Racism and Xenophobia 
Europe 2020 Strategy and European Semester 
European  Structural  and  Investment  Funds  (ESIF)  /  Instrument  for  pre-
Accession Assistance (IPA) 
Directly  managed  EU  funding,  e.g.  under  the  Rights,  Equality  and  Citizenship 
Programme and the Employment and Social Innovation Programme 
Enlargement negotiations under Chapter 23 
Visa liberalisation roadmaps for enlargement countries 

• 
• 
National 
• 
National Roma integration strategies or integrated sets of policy measures 
•  Mainstream public policies in education, employment, healthcare, housing 
• 
• 
Local 
• 
• 
• 

Local Roma integration measures 
Local level policy making and equality practices 
Local funding 

National equality legislation 
National funding 

Coherence 

Governance: 
European 
• 
• 

• 
National 
• 
• 

National Roma Contact Point Network 
Consultation meetings with European umbrella Roma and pro-Roma civil society 
and international organisations and partners 
European Platform for Roma inclusion 

National Roma contact points and national Roma platforms 
Equality bodies 

ACTIVITIES 
European 
Monitoring and guidance 
•  Monitoring under the EU Framework of NRIS: 

Roma-targeted country-by-country and horizontal 
guidance, including for enlargement countries 

•  Monitoring under the European semester: 

Country-specific recommendations and country 
reports 

•  Monitoring under ESIF/IPA: Guidance e.g. on the 

use of ESIF for inclusive policies 

•  Monitoring under Racial Equality Directive: e.g. 

infringement on segregation of Roma in education 

•  Monitoring for enlargement under the bi-annual 

Roma seminars and annual sub-committees. 

Other initiatives 
• 

• 

• 

• 

• 

• 

European initiatives, such as the ‘for Roma, with 
Roma’ awareness raising campaign 
Joint programmes with the Council of Europe: 
JUSTROM, ROMACT, ROMED, ROMACTED, 
Calls for proposals under relevant European 
programmes in direct management (Rights, 
Equality and Citizenship Programme, Europe for 
Citizens Programme, Employment and Social 
Innovation Programme, IPA). 

National 
• 

Implementation, monitoring and review of NRIS or 
integrated sets of policy measures 
Implementation of CSRs, national reform, 
programme 
Partnership agreement, Operational 
Programmes, European Structural and 
Investment Funds 
Transposition and application of European 
equality legislation 

101 

•  Guidance, training and capacity building to 

EU added value 

Sustainability 

OUTCOMES 
 

Increased capacity of 
national and local actors to 
implement Roma integration 
measures 
Strengthened cooperation 
between stakeholders of 
Roma integration 
Reduced gaps in education, 
employment, health and 
housing between Roma/non-
Roma 

 

 

Coordination 

OUTPUTS 
 

Increased number of 
targeted and mainstream 
Roma integration measures 
•  Mainstream public policies 
which are more inclusive of 
Roma issues 
Increased investment in 
Roma integration measures  
•  More Roma beneficiaries of 

• 

national or local actors under the NRIS, ESIF, 
equality legislation. 

targeted and mainstream 
measures 

Efficiency 

 
 
 
 
 
 
 
 
ANNEX 7B: INTERVENTION LOGIC II-ZOOM ON THE EU FRAMEWORK 

Effects 

RATIONALE FOR INTERVENTION 
(needs) 
• 

Prevent  socio-economic  exclusion 
and discrimination of Roma 
•  Reap long-term societal benefits 

from Roma inclusion 

OBJECTIVES 

• 

Fight discrimination, combat 
poverty and improve Roma 
access to quality inclusive 
mainstream education, 
employment, healthcare and 
housing. 

EXTERNAL FACTORS 
 
  Refugee crisis 
  Domestic political situation — rise of 

Economic and labour market situation 

 

nationalist parties 
Social media and discriminatory 
rhetoric 

General objectives of fighting 
discrimination and reducing poverty 

Four EU Roma integration goals in the 
areas of employment, education, 
housing and health 

Activities the EU Framework proposes 

for Member States (in proportion to the size of the Roma 
population and taking into account different starting points): 

 
 
 
 

 
 

 
 

design national Roma integration strategies 
set achievable national goals for Roma integration 
appoint national Roma contact points 
implement measures in the four areas of employment, 
education, housing and health 
allocate sufficient funding from national budgets 
identify disadvantaged micro-regions or segregated 
neighbourhoods where communities are most 
deprived 
include monitoring and review mechanisms in NRIS 
design, implement and monitor NRIS in close 
cooperation with civil society, regional and local 
authorities 

for the enlargement countries 

 

The EU Roma integration goals are equally relevant to 
enlargement countries. Their national Roma 
integration strategies and action plans should be 
reviewed in line with these goals. 

Accompanied by: 

102 

  Mobilisation and alignment of other legal, policy and 
financial instruments and governance structures 
  Making European and national policies more sensitive 
to Roma  needs, including through empowering civil 
society 

  Putting in place a robust monitoring system 

 

 

 

IMPACTS 
 

Sustainably improved socio-
economic inclusion of Roma; 
sustainably reduced gaps 
between Roma and non-
Roma 
Economic, fiscal, and social 
benefits to EU societies 

OUTCOMES 
 

Increased capacity of 
national and local actors to 
implement Roma integration 
measures 
Strengthened cooperation 
between stakeholders of 
Roma integration 
Reduced gaps in education, 
employment health and 
housing between Roma and 
non-Roma 

OUTPUTS 
 

Increased number of 
targeted and mainstream 
Roma integration measures 
•  Mainstream public policies 
which are more inclusive of 
Roma issues 
Increased investment in 
Roma integration measures  
•  More Roma beneficiaries of 

• 

targeted and mainstream 
measures 

 
 
 
 
 
 
 
 
 
 
 
 
 
ANNEX 8: THEMATIC MAPPING OF INSTRUMENTS AND PROCESSES 

E
U
R
O
P
E
A
N
L
E
V
E
L

N
A
T
I
O
N
A
L

L
E
V
E
L

L
O
C
A
L

L
E
V
E
L

EU Framework for National Roma Integration Strategies up to 2020 

European legislation 
(Racial Equality Directive, Council 
Framework Decision on Racism and 
Xenophobia) 

Mainstream policies, such 
as the European Semester 

Council Recommendation on 
effective Roma integration 
measures in Member States 

EU funding such as European Structural and Investment Funds 

National Roma integration strategies & measures 

National legislation 

Mainstream public policies 

National funding 

Local Roma integration strategies & measures 

Local legislation/practices 

Local level policy-making and implementation 

Targeted policy 

Mainstream policy 

Legal instruments 

Funding instruments 

Local funding 

FIGHT DISCRIMINATION, COMBAT POVERTY, PROMOTE ROMA ACCESS TO 
EDUCATION, EMPLOYMENT, HEALTH, HOUSING   

103 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
104