EUROPEAN COMMISSION Brussels, 4.12.2018 SWD(2018) 480 final COMMISSION STAFF WORKING DOCUMENT Evaluation of the EU Framework for National Roma Integration Strategies up to 2020







Accompanying the document COMMUNICATION FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL

Report on the evaluation of the EU Framework for National Roma Integration Strategies up to 2020 {COM(2018) 785 final}



TABLE OF CONTENTS 1.Introduction 2.Background to the intervention 3.Implementation / state of play 4.Method 5.Analysis and answers to the evaluation questions 5.1 Effectiveness 5.2. Coordination 5.3. Relevance 5.4. Equity 5.5. Coherence 5.6. Efficiency 5.7. EU added value 5.8 Sustainability 6.Conclusions Annex 1: Procedural information Annex 2: Stakeholder consultation Annex 3: Evaluation criteria and questions

Annex 4: Methods and analytical models Annex 5: Costs and benefits Annex 6: Objectives and Measures 2011 and 2013 Annex 7a: Intervention logic I Annex 7b: Intervention logic II-zoom on the EU Framework Annex 8: Thematic mapping of instruments and processes Glossary Term or acronym Meaning or definition CSR Country-specific recommendation ECEC Early childhood education and care ERDF European Regional Development Fund EPSCO Employment, Social Policy, Health and Consumer Affairs Council ESF European Social Fund ESIF European structural and investment funds ESL Early school leaving EU-MIDIS European Union minorities and discrimination survey EURoma European network on social inclusion and Roma under structural funds FRA Fundamental Rights Agency IP Investment priority IPA Instrument for pre-accession assistance NGO Non-governmental organisation NRCP National Roma contact point NRIS National Roma integration strategies and integrated sets of policy measures OP Operational Programme OPC Open public consultation OSCE Organisation for Security and Cooperation in Europe PA Partnership agreement REC Rights, Equality and Citizenship Programme SFC System for fund management in the EU UNDP United Nations Development Programme 1.Introduction This staff working document (SWD) is a midterm evaluation of the ‘EU framework for national Roma integration strategies up to 2020’ (EU framework), adopted as a Communication by the Commission on 5 April 2011 and welcomed by the Council in May 2011 and the European Council in June 2011 . This midterm evaluation responds to the request to the Commission made in the EPSCO Council conclusions of 8 December 2016 entitled ‘Accelerating the Process of Roma Integration ’. It aims to provide information to the Commission’s departments and to stakeholders to help steer the work for the remaining implementation period of the EU framework (2020) and to prepare for the period thereafter. The evaluation assesses the relevance, effectiveness, efficiency, coherence and EU added value of the EU framework. It also assesses three further evaluation criteria, namely coordination, equity and sustainability. In terms of geographic scope, the evaluation mainly focuses on EU Member States, with the exception of Malta, which does not have a Roma community. However, it also covers the enlargement region (Bosnia and Herzegovina, the former Yugoslav Republic of Macedonia, Serbia, Turkey, Albania, Kosovo *and Montenegro ). Regarding the thematic scope, the evaluation focuses on how the EU framework and its objectives are working, notably in the areas of poverty, discrimination, education, employment, health and housing. The evaluation covers in particular:

·changes in the situation of Roma in the areas of poverty, discrimination and access to education, employment, healthcare and housing since the EU framework was adopted and the extent to which changes are attributable to this initiative; ·the existence of preconditions and structures for Roma integration at national level as provided for in the EU framework, such as national Roma integration strategies (NRIS) and national Roma contact points (NRCPs); ·alignment of EU level and national policies and mobilisation of legal and financial instruments (e.g. the European Semester and EU funding programmes) to help meet the objectives of the EU framework; ·the role of the EU framework and NRIS in integrating Roma inclusion into mainstream policies and in stimulating cooperation and dialogue between stakeholders, including civil society; ·the use of EU and national funding to implement NRIS and Roma integration measures; ·the impact of the EU framework on promoting Roma integration in the enlargement region; and ·lessons learnt for the remaining implementation period and for the period after 2020. The thematic scope does not include an evaluation of other legal, policy or funding instruments that are directly or indirectly relevant for Roma integration, such as the Racial Equality Directive, the European Semester or the European structural funds. For these instruments, only their alignment with the EU framework’s objectives and their mobilisation to advance Roma integration and fight discrimination is assessed . Where available and relevant, the SWD draws on existing evaluations of these instruments. The evaluation covers the period April 2011 (start of the intervention) to the end of 2017. Comparable data on the living conditions of Roma for the period 2011-2016 for EU Member States and 2011-2017 for enlargement countries have been used. The evaluation has been informed by an external study requested by the Commission and prepared by ICF Consulting Services Ltd and Milieu (hereafter ‘ICF’).

2.Background to the intervention 2.1. Description of the intervention and its objectives Vice-President Viviane Reding, 08 April 2011: ‘The EU Framework comes at a moment when the situation of Roma has become the centre of attention — and a matter of collective shame for Europe. (…) Action is needed both at national and at EU level. We need to join forces and step up our efforts to end discrimination against the Roma and to make sure they enjoy the same rights as any other EU citizen. That is why the European Commission has come up with a new and innovative instrument: an EU Framework designed especially for Roma economic and social integration in all Member States ’. Wider policy developments at the time of adoption of the EU framework include: From an economic perspective, the financial and economic crisis wiped out years of economic and social progress, leaving millions of people unemployed, exposing structural weaknesses in Europe’s economy and putting new pressure on social cohesion . In response, in its Europe 2020 strategy adopted in 2010, the Commission proposed five measurable EU targets for 2020 to be converted into national targets. Of these targets, the three related to employment, education and poverty are directly relevant for Roma integration.

From an equality perspective, a decade had passed since adoption of the two anti-discrimination Directives which Member States were required to implement by 2003 and which, among other grounds, prohibit discrimination on grounds of race and ethnic origin. The Framework Decision combating racism and xenophobia was adopted in 2008 and implementation of all three pieces of legislation was being monitored . With regard to other equality policies, the Commission adopted its 2010-2015 strategy for equality between women and men in September 2010. This strategy underlined the cumulative effect of discrimination on two or more grounds, for example, for being Roma and a woman . Concerning the free movement of people, the dialogue between the Commission and some national authorities on how the relevant EU law should be applied to address the situation of the Roma had intensified in 2010. The Commission had called upon Member States to respect the rules laid down in the 2004 Directive on free movement with respect to the fundamental rights of EU citizens and avoiding discrimination, notably on the grounds of nationality or ethnic group . In addition, transitional arrangements on the free movement of workers that had been agreed in the Accession Treaties of Bulgaria and Romania still allowed the remaining Member States to temporarily restrict workers from Bulgaria and Romania from working in their country. This restriction had an effect on Roma from these two countries, as the last restrictions on the free movement of workers from Bulgaria and Romania were only lifted on 1 January 2014. International, European and national policies and initiatives for Roma integration started from 2005 and continued in the years preceding the EU framework. These include: ·the Decade of Roma Inclusion 2005-2015 ; ·the 2008 Commission Communication highlighting, among other things, that the tools to advance the social inclusion of Roma should be better applied; ·the common basic principles on Roma inclusion attached to the EPSCO Council conclusions on Roma inclusion of June 2009; ·a 2010 Commission Communication focusing on the social and economic integration of Roma in Europe ; and ·the European Parliament’s Resolution of 9 March 2011 on the EU strategy on Roma inclusion . Despite these efforts, the economic and social marginalisation of Roma persisted and many of the estimated 10-12 million Roma in Europe continued to face prejudice, intolerance, discrimination and social exclusion in their daily lives. EU countries were (and are) characterised by different sizes of the Roma population living in their territories and different starting points in terms of Roma integration policies. The estimated shares of Roma in the EU countries in 2012 range from 10.3% in Bulgaria, 9.1% in Slovakia, 8.3% in Romania, 7% in Hungary, 2.5% in Greece, 2% in the Czech Republic, 1.6% in Spain to less than 1% in most of the other countries. The fact that Roma were marginalised and lived in very poor socio-economic conditions was increasingly seen as an economic disadvantage, resulting in welfare dependency and limiting the labour markets and tax revenues , in addition to being a question of European values and human rights The Commission’s policy response was to adopt the EU framework. This was the first EU initiative specifically concerning Roma which included a follow-up mechanism . The Commission proposed that Member States, in proportion to the size of the Roma population living in their territories and taking into account their different starting points, design national Roma integration strategies (NRIS) and set achievable national goals for Roma integration or, where they already exist, adapt them to meet four EU Roma integration goals: 1. Ensure that all Roma children complete, as a minimum, primary school education. 2. Cut the employment gap between Roma and the rest of the population. 3. Reduce the gap in health status between the Roma and the rest of the population. 4. Close the gap between the share of Roma with access to housing and to public utilities (such as water, electricity, and gas) and that of the rest of the population. The framework did not set a specific EU Roma integration goals in the area of non-discrimination. Instead, it underlined that Member States are already under an obligation to ensure that Roma are not discriminated against, as laid down in the Racial Equality Directive, i.e. to give Roma non-discriminatory access to education, employment, vocational training, healthcare, social protection and housing. Its conclusions stress that the framework is complementary to existing EU legislation and policies in the areas of non-discrimination, fundamental rights, the free movement of persons, and the rights of the child.

The framework underlines that non-discrimination is not sufficient to combat the social exclusion of Roma. Member States have the primary responsibility as well as the competence to improve the situation of marginalised populations. Action to support Roma integration lies first and foremost in their hands, taking into account specific national circumstances, needs and required solutions. The EU framework acknowledges this, but due to shared values (such as equality and fundamental rights) and common objectives (such as economic prosperity, social cohesion and solidarity between Member States) , it calls for a European role in policies for Roma integration. A key driving force for the EU framework was transnational mobility of Roma in the context of freedom of movement, making Roma integration a joint EU-level objective. To achieve each of the four Roma integration goals mentioned above (the specific objectives), the EU framework took a non-binding approach, listing a set of measures Member States should take to make progress. These measures take account of different situations and starting points in terms of national strategies already in place and the size of the Roma population. The EU framework also requests that when developing their NRIS Member States: ·allocate sufficient funding from national budgets, complemented where appropriate by international and EU funding; ·include robust monitoring methods to evaluate the impact of Roma integration actions, and a review mechanism so that the strategy can be adapted if necessary; ·design, implement and monitor the strategy in close cooperation and continuous dialogue with Roma civil society and regional and local authorities; and ·appoint a national contact point for the NRIS, with the authority to coordinate the development and implementation of the strategy or, where relevant, rely on suitable existing administrative structures. Overall, the EU framework was ambitious in terms of EU guidance and new features (such as EU Roma integration goals and national goals, development of the National Roma Integration Strategies, appointment of National Roma Contact Points). And it was flexible in terms of concrete approaches and responses expected from Member States, taking into account the size of the Roma population and different starting points and having a non-binding approach for national measures. In addition, the EU framework: (i) encouraged Member States to make use of available EU funds to address the needs of Roma; (ii) stressed the importance of making European and national policies more sensitive to Roma needs, including through empowering civil society; and (iii) highlighted the need to put in place a robust monitoring system. With regard to the enlargement region, the EU framework emphasised that the EU Roma integration goals are equally relevant to enlargement countries. It included three main goals: i) improve the delivery of pre-accession assistance, ii) strengthen the involvement of the civil society, iii) enhance the monitoring. 2.2 Intervention logic and baseline The EU framework’s theory of change began with the observation that discrimination on the basis of racial or ethnic origin in education, employment, health and housing and other areas was already prohibited by EU law, but that non-discrimination alone is not sufficient to combat the social exclusion of Roma . Therefore, to fight discrimination and combat poverty (general objectives), countries need to develop and implement an integrated and sustainable approach, with efforts spanning different policy areas — education, employment, health and housing in particular (Roma integration goals specific objectives). The expectation was that in the long term this approach would lead to social and economic benefits for Roma and non-Roma alike. When the EU framework was adopted, no intervention logic had been prepared. It has therefore been developed for this evaluation and has two elements: Annex 7a (intervention logic I ) provides an overview of key instruments, governance structures and activities available to advance Roma integration at EU, national and local levels. Inputs and activities directly related to the EU framework are highlighted by colouring them in red. This annex illustrates that in working towards the Roma integration goals, the EU framework does not operate independently of other legal, policy and financial instruments but mobilises and aligns these instruments and their concrete activities to reach its specific objectives. It highlights that outputs, outcomes and impacts of the EU framework depend on the smooth running of these other instruments as well as on commonly shared external factors. The annex also illustrates that the expected key outputs included increased funding for Roma integration, a higher number of Roma beneficiaries, and more effective targeted and mainstream policy measures. It shows that the key expected outcomes of the intervention were the achievement of the Roma integration goals in 2020, increased capacity of those involved at national and local level to implement Roma integration measures as well as strengthened cooperation between stakeholders; and that the expected key impacts include socio-economic inclusion, less discrimination of Roma and economic, fiscal and societal benefits such as a rise in GDP through better education outcomes and a more skilled workforce. Some external factors influencing effectiveness were already apparent when the framework was adopted. These include the effect of the economic crisis on the labour market, high levels of discrimination preventing some goals from being quickly or fully achieved and the political will required to achieve the EU framework’s objectives. Annex 7b (intervention logic II) focuses more specifically on the EU framework, setting out its objectives, its interaction with Member States and enlargement countries and its role in mobilising and aligning other EU and national policy, legal and financial tools for Roma integration. The expected outputs, outcomes and impacts remain the same. Information about the baseline situation of Roma in the EU and the enlargement region is still incomplete, with gaps in most Member States, at least for certain subgroups such as migrant or EU-mobile Roma . The fact that no impact assessment was carried out when the EU framework was prepared also affects this evaluation. Furthermore, early studies and reports by European institutions showed a lack of systematic and in-depth knowledge of the situation of the Roma . Most information was collected and made available thanks to efforts of the United Nations Development Programme (UNDP) in central and eastern Europe . The baselines sources mentioned in the 2011 Communication therefore refer to partial studies covering a limited number of Member States . The EU institutions’ first systematic (but still incomplete) mapping of the situation of the Roma (in Member States where this could be done using quantitative methods) followed the adoption in 2011 of the EU framework. This first attempt was the survey of 11 EU Member States by the Fundamental Rights Agency (FRA), which found that in 2011, Roma people were significantly discriminated against across the EU. Eurobarometer surveys from 2012 also provide insights into the prevalence and nature of discrimination faced by Roma. When asked how comfortable citizens in their country would feel if their children had Roma schoolmates, 34% thought that citizens in their country would feel uncomfortable with such situation (28% fairly comfortable and 31% comfortable). The 2011 UNDP/World Bank/EC Regional Roma survey revealed significant gaps between Roma and non-Roma in Western Balkan countries. The table below shows the situation of EU Roma in 2011, versus the general population, for a number of indicators. As explained above, such data on Roma did not yet exist at the time of adoption of the EU framework. Eurobarometer findings on perception of Roma in the general public were not included as questions in the 2012 and 2015 surveys were different and findings thus not comparable. Table 1: Comparing the situation of Roma and the general population in 2011 Baseline indicators Roma General population Poverty At risk of poverty 86 % 19 % Education Share of children +4 attending ECEC 47 % 90 % Early leavers from education and training 87 % 21 % Share of children in compulsory schooling age (5-17 depending on the country) attending education 86 % 98 % Share of people that felt discriminated in the past 5 years because of being Roma when in contact with school 17 % - Share of Roma children aged 6-15 attending classes where all classmates are Roma 10 % - Employment Share of people with ‘paid work’ as their self-declared main activity status, 16+ (%) 26 % 70 % Share of young people aged 16-24 years old with current main activity as neither employment, education or training (NEET), household members (%) 56 % 16 % Share of people that felt discriminated in the past 5 years because of being Roma when looking for a job 50 % - Share of people that felt discriminated against at work in the past 5 years because of being Roma 19 % - Health Share in ‘very good’ or ‘good’ health (self-reported) 55 % 68 % Share with medical insurance coverage 78 % 94.5 % Housing Share of people living in households having neither toilet, shower, nor bathroom inside the dwelling 36 % 7 % Average number of rooms per person in the household 0.6 1.46 Share of people that felt discriminated against in the past 5 years when looking for housing because of being Roma 45 % -

For the Roma integration goals more specifically, this translates into the following reconstructed baseline not available at the time of adoption of the EU framework: Table 2: 2011 reconstructed baseline for the Roma integration goals Roma integration goal Value/Gap 2011 Access to education: Ensure that all Roma children complete at least primary school 86 % Access to employment: Cut the employment gap between Roma and the rest of the population 44 pps Access to healthcare: Reduce the gap in health status between the Roma and the rest of the population 13 pps Access to housing: Close the gap between the share of Roma with access to housing and to public utilities (water, electricity, gas) and that of the rest of the population 29 pps 0.66 rooms



3.Implementation / state of play The EPSCO Council conclusions from May 2011 welcomed the EU framework and ‘invited Member States to set or continue working towards their goals, in accordance with the Member States’ policies, in the fields of education, employment, healthcare and housing with a view to closing the gaps between marginalised Roma communities and the general population’. As such, the conclusions did not explicitly endorse the EU Roma integration goals. The political importance of Roma integration was recognised by the European Council in June 2011 which called for the Council’s conclusions to be implemented rapidly. It particularly requested that Member States prepare, update or develop NRIS or integrated sets of policy measures for improving the situation of Roma within their broader social inclusion policies, by the end of 2011. By 2012, all Member States except for Malta had submitted their strategies or integrated sets of policy measures . That same year, the Commission presented the results of a first assessment of all NRISs and invited Member States to consider a number of adjustments for the future. The assessment concluded that much more needed to be done at national level. In particular, more concrete measures, explicit targets for measurable deliverables, clearly earmarked funding at national level and a robust national monitoring and evaluation system would be necessary. In its Communication of 26 June 2013 entitled ‘Steps forward in implementing NRIS’ , the Commission stressed the need for further action on the preconditions necessary to promptly carry out measures to speed up Roma integration. In response, in December 2013 the Council Recommendation on effective Roma integration measures in the Member States was adopted unanimously. The Recommendation, which was the first EU soft law instrument explicitly targeting Roma: ·provided guidance to Member States to make their measures for Roma integration more effective and strengthen implementation of their NRIS; ·recommended that Member States take effective policy measures to ensure equal treatment for Roma and the respect of their fundamental rights, including equal access to education, employment, healthcare and housing; ·indicated how each of the four goals — ensuring equal access to education, employment, healthcare and housing — could be met; ·set out that these goals could be achieved either through mainstream or targeted measures, including specific ones to prevent or compensate for disadvantages, or by a combination of both, paying special attention to the gender dimension; ·reinforced the focus on anti-discrimination, referring explicitly to antigypsyism , and extended the work on Roma integration to new horizontal and structural areas beyond employment, education, health and housing; and ·called on Member States to communicate to the Commission on an annual basis, starting from 2016, the measures taken in line with the Recommendation and progress achieved in implementing the strategies. For the enlargement countries it is important to note that in 2013, the Communication on the enlargement strategy and main challenges 2013-2014 identified access to civil documentation as a priority for enlargement countries, in addition to the four areas of employment, education, housing and health .

Individual assessments of Member States’ NRIS were presented by the Commission in 2012, 2014 and 2016. The 2016 assessment identified some positive trends, such as the growing focus on early childhood education, but underlined that — overall — efforts had not prevented further deterioration in living conditions of Roma or widespread hostility among the general population. In its Communication, the Commission called on Member States to reaffirm political commitment to Roma integration so that the legal, policy and financial instruments in place could be fully used to bring tangible results on the ground . In response, the Council adopted Conclusions in December 2016 on accelerating the process of Roma integration . Member States reconfirmed their commitment to the Roma integration process and their determination to ensure that all policy, legal and financial instruments put in place at European and national level be used to close the gap between Roma and non-Roma. The Commission’s 2017 midterm review of the EU framework took stock of progress achieved since 2011 by presenting the findings of the Roma integration scoreboard. It urged Member States to intensify efforts and provided guidance on how to prioritise actions. It announced that a midterm evaluation would be launched in order to analyse the EU framework's achievements and challenges in more depth, consult stakeholders and use external expertise. The EU framework led to the adoption of NRIS and the establishment of NRCPs at national level (see table below). In total, 15 Member States have adopted strategies and the remaining 12 have adopted integrated sets of policy measures within their broader social inclusion policies . The reasons provided by these 12 Member States for not having developed strategies are: (i) the small size of the Roma population ; (ii) the legal impediments to adopting a strategy that targets one specific ethnic group ; and (iii) insufficient evidence of discrimination against Roma, which deemed a strategy unnecessary . Most Member States that adopted NRIS did so in response to the EU framework. As of 2016, all enlargement countries had adopted an NRIS and action plans and had put in place NRCPs . Some Member States had bodies in place prior to the EU framework to coordinate matters on ethnic minority issues, including Roma issues but they were formally nominated as NRCPs following adoption of the EU framework . Table 3: Which of the following instruments and governments structures were established at Member State in response to the EU framework for NRIS ? Yes No Pre-existed (year of adoption) NRIS AT, BE, BG, CY, CZ, DE, DK, EL, ES, IE, LT, LU, LV, PT, SK, Former Yugoslav Republic of Macedonia, Montenegro, Turkey NL, UK, EE, SE FI (1999), HR (2003), SI Albania (2003), Bosnia and Herzegovina (2005), Kosovo (2009), Serbia (2009) NRCP AT, BE, CY, CZ, DE, DK, EE, EL, IE, LT, LU, LV, NL, PT, UK, Albania, Bosnia and Herzegovina, former Yugoslav Republic of Macedonia, Kosovo, Montenegro, Serbia, Turkey SE BG, ES, FI (1956), HR (1991), SI, SK Source: Interviews with NRCPs; *The table is based on interviews with NRCPs — not all NRCPs responded to these questions and therefore not all Member States are covered. In addition, enlargement countries were only partially covered by interviews. DG NEAR provided additional information. 4.Method An external evaluation carried out by a team of consultants from October 2017 to May 2018 provided information for the current midterm evaluation. The methodological approach chosen for the external study is described in detail in Annex 4. In this SWD, the Commission has built on the findings and conclusions of the external evaluation, unless highlighted otherwise in the text. In addition, to prepare the midterm evaluation, the Commission put together a set of draft evaluation questions relating to the five main criteria of effectiveness, relevance, coherence, efficiency and EU added value, as per the evaluation design and road map . The evaluation framework developed by the external evaluator under the guidance of an inter-service steering group (ISSG) built on and further elaborated these questions, also covering data collection and analysis methods. The evaluation also addresses the criteria of coordination, sustainability and equity as the ISSG deemed them relevant for this evaluation . Data collection and analysis Evaluation findings are based on the following range of data sources in particular: ·literature review (175 sources ) ·phone and face-to-face interviews of key stakeholders in 27 Member States (191 interviews) ·phone and face-to-face interviews of 10 key stakeholders in three enlargement countries ·8 phone and face-to-face interviews at EU level (Commission staff, MEP, EU-level NGO) ·country analysis studies (11 Member States with a total share of 90.2 % of the EU Roma population) pulling together quantitative and qualitative country-specific information ·an open public consultation (OPC), which generated 240 responses and 28 position papers

·a targeted NGO survey, which generated 65 responses ·data provided by Member States in the online reporting tool for annual reporting to the Commission (years covered: 2015 and 2016) ·ESIF programming documents , including through the use of dedicated portals provided by the Commission (SFC 2014) ·2011 FRA Roma Survey and 2016 FRA EU-MIDIS II survey ·2011 UNDP/ World Bank/ EC regional Roma survey and 2017 UNDP/World Bank/EC survey on the Western Balkans ·discussions at the evaluation workshop on preliminary findings of this evaluation, at an event discussing the evaluation at the EU Roma Week, at the 2017 European Platform for Roma Inclusion (focusing on the transition from education and employment) and other stakeholder and consultation events. Method strengths, limitations and mitigation actions The lack of solid Roma specific performance indicators and quantitative data affected this evaluation in several ways, in particular the ability to assess the effectiveness and efficiency criteria. Other limiting factors included the relative unreliability of the data from the online reporting tool (used by Member States to report to the Commission on NRIS implementation) and reduced availability of data on quantifiable costs and benefits (see Annex 4 for a detailed description and the mitigation actions taken). b) Limitations regarding attribution of outputs, outcomes and impacts to the EU framework A number of social and economic factors (economic developments, political priorities etc.) unrelated to the EU framework, directly or indirectly impact efforts for Roma integration. In addition, given that the policy areas covered by the EU framework fall largely under the competence of Member States, the aim of the EU framework was to provide guidance for the NRISs and their implementation. It was expected that the EU framework would enable exchange and encourage cooperation of relevant stakeholders as well as mobilise other EU level policy, legal and funding instruments for Roma integration. In turn, it was for the NRISs to mobilise and align the use of national legal, policy and funding instruments, and provide guidance for local-level action plans and strategies, which would interact with local-level policies (see Annex 8 ). Therefore, the results and impacts of the EU framework on the situation of Roma can thus only be considered to be indirect and as being influenced by the instruments and actors at national and local levels. Consequently, a prudent approach was chosen for this evaluation, particularly with regard to efficiency criterion. The evaluation focuses primarily on costs and benefits attributable to the EU framework that have so far been identified. On the other hand, it does not attempt to assess the extent to which EU and national funding in the area of Roma integration is directly justified and proportionate to the long-term benefits from Roma integration in inclusive, quality mainstream education, employment, healthcare and housing .Such an assessment would have needed more solid Roma-specific indicators as well as a clear relationship between cause and effect of interventions. Validity and reliability of the findings Despite the lack of a quantified counter-factual scenario and the limitations set out above, the evaluation findings are valid and reliable. Mitigation measures were taken and the evaluation was underpinned by a large body of qualitative evidence which provides a solid basis for drawing conclusions. In particular the workshop disseminating and discussing preliminary findings helped to further verify the correctness of the findings and fine-tune the findings.





5.Analysis and answers to the evaluation questions 5.1 Effectiveness The analysis of effectiveness considers the extent to which the EU framework has made progress towards its objectives at the midterm stage, using appropriate points of comparison and assessing the role of the EU framework in delivering the observed changes. The effectiveness of the EU framework so far has been assessed as being largely limited regarding progress towards the Roma integration goals. The assessment is more positive with respect to the setting up and strengthening of instruments and structures and increased cooperation among and capacity of stakeholders. The assessment has been informed by 14 evaluation questions that looked into the extent to which: ·progress has been made so far on the expected outcomes, in particular towards reaching the Roma integration goals in the areas of access to education, employment, housing and health, as well as the reasons for this progress; ·changes in the situation of Roma in the four areas can be attributed to the EU framework and NRIS; ·the EU framework contributed to the setting up of instruments and structures for Roma integration at EU and national level, increasing cooperation among stakeholders and capacity of national and local actors; ·the EU framework has indeed served as a framework for NRIS; ·the EU framework and NRIS contributed to an increase in the number of and funding for Roma integration measures as well as an increase in the number of Roma beneficiaries. 5.1.1. Progress towards the Roma integration goals, attribution and outlook Attributing progress towards the attainment of the Roma integration goals is difficult in light of: (i) numerous external factors such as the economic and financial crisis (and the resulting economic and labour market challenges in various countries), the 2015 refugee crisis (which in some Member States led to lowering the priority of Roma integration in the national policy agenda) or the rise of nationalist parties and spread of discriminatory rhetoric; and (ii) the distribution of competences between the EU and the Member States. The policy areas covered by the EU framework fall largely under the competences of Member States. The EU framework therefore provided guidance for NRIS implementation, enabled exchange and cooperation between relevant stakeholders and mobilised other policy, legal and funding instruments for Roma integration. These aspects are elaborated in detail below and in the coherence section. In turn, it was for the NRIS to mobilise and align the use of national legal, policy and funding instruments, and provide guidance for local-level action plans and strategies, which would interact with local-level policies. The effects of the EU framework on the situation of Roma can therefore only be indirect, and mitigated by the instruments and actors at play at national and local levels. The area of education observed the most pronounced progress . Participation of Roma in early childhood education and care (ECEC) has improved since 2011 in most countries , from 47 % in 2011 to 53 % in 2016. School attendance at compulsory schooling age has improved, from 86 % to 90 % and more specifically in four out of the nine countries surveyed (CZ, EL, HR, PT); early leaving from education and training has decreased from 87 % to 68 % and more specifically in seven out of the nine countries (BG, CZ, ES, HU, PT, RO, SK). However, substantial gaps remain both between Roma and the general population and between countries (ranging from a 28 % participation rate of Roma children in Greece to 94 % in Spain). Attendance of education when in compulsory schooling age is not far from the general population (90 % Roma compared to 96 % non-Roma). The share of Roma students attending classes where ‘all classmates are Roma’ has increased (from 10 % to 15 %) and one to two thirds of Roma children attend schools where most or all children are Roma. For the Western Balkan countries, the situation shows improvement between 2011 and 2017 especially as regards enrolment in compulsory education (ages 7-15). However,, pre-primary school enrolment rates slightly decreased in 3 out of 6 countries (Albania, Bosnia and Herzegovina, the former Yugoslav Republic of Macedonia). Substantial improvement was registered in Montenegro only and appears to be directly linked with the implementation of a UNICEF project. Importantly, in most of the cases, there was no reduction of the gap in education between e Roma and non-Roma . FRA survey data shows a lack of observable improvement in access to employment across the analysed period. The proportion of Roma who are in paid work as their main activity — including self-employment and occasional work — remains similar (around 25 % of Roma aged 16+). The share of young Roma aged 16 to 24 whose current main activity is cited as being neither in employment, education or training (NEET) has increased from 56 % to 63 %. The gap with the general population remains wide. Until now, the improvements observed in education have barely resulted in higher employment rates, and the transition from education to employment remains a challenge. In respect to the Western Balkan countries, there has been has been a general worsening of the situation since 2011; only the former Yugoslav Republic of Macedonia has shown an improvement related to reduction of unemployment and informal employment of Roma. Most likely reasons for this lack of improvement in employment rates include: (i) the consequences of the economic crisis; (ii) existing prejudices and discrimination; (iii) the absence of adapted support measures; (iv) insufficient qualifications . Country analysis studies and interviews mention that low educational status, the lack of basic skills and discrimination are not tackled effectively. Also, mainstream measures for active employment typically lack explicit references to Roma and do not provide individualised support. Therefore, coupled with adverse economic conditions, the employment measures have failed to effectively reduce the employment gap between Roma and non-Roma. However, given the duration of the educational cycle and considering its long-term qualitative effects, improvement in the employment situation is likely in the medium-to long-term. Health self-perception has improved but medical health insurance coverage is still limited . The reported average self-perceived health status of Roma is equal to that of the general population. As compared with previous years, more Roma consider that their health is good or very good (from 55 % to 68 %). Nevertheless, healthcare coverage has not increased (around 75 % of Roma aged 16+), with differences between countries ranging significantly from 45 % in Bulgaria to 98 % in Spain . Access to health insurance and health services, including health promotion and disease prevention, poses special problems for many Roma and results in difficulties particularly in the area of sexual and reproductive health services and vaccination . In the Western Balkan countries, general improvements in health insurance coverage and reduced inequality were observed. However, Montenegro had poorer outcomes for both of these indicators and health insurance coverage reduced in Albania. Over 70 % of OPC respondents claimed that their country had not made progress on the implementation of Roma integration measures on healthcare. Most likely reasons for this limited progress over the short time frame are a range of barriers such as language and literacy, a lack of trust, a lack of identification documents and physical distance to healthcare facilities . There has been very little change in Roma housing conditions. The housing situation for many Roma families continues to be difficult. In 2016, the average number of rooms per person was 0.7 (0.6 in 2011), 30 % of households did not have tap water inside the dwelling (29 % in 2011) and the share of people living in households having neither a toilet, shower, nor bathroom inside the dwelling was reportedly 38 % (36 % in 2011). There is still a gap between Roma and the general population in these areas . However, most households reportedly have an electricity supply (92 % to 96 %). Most likely reasons identified for the lack of progress are discrimination and lack of funding. The data does not indicate any decrease in discrimination in access to housing. On the contrary, in some countries a notable increase is observed . Many housing conditions are substandard and a gap remains in access to public utilities. Many respondents to the NGO survey (31 out of 65 ) believe that since 2011, the NRIS of their Member State did not contribute to ensuring national funding for Roma integration in the area of housing. Forced evictions of Roma from their homes continue to be reported across many Member States and enlargement countries . Regarding the Western Balkan enlargement region, the housing situation of Roma varies considerably between countries, with improvements in Serbia and Bosnia and Herzegovina, and a deterioration in Albania and Montenegro . The following table presents a shortened and simplified 'nutshell assessment' of progress towards the specific objectives of the EU framework:

Table 4: Progress towards the Roma integration goals Objective Progress status 2011 2016 Access to education: Ensure that all Roma children complete at least primary school Some progress: attendance of education when in compulsory schooling age has increased 86 % 90 % Access to employment: Cut the employment gap between Roma and the rest of the population No progress: the share of Roma in paid work has not increased in comparison to the employment rates of the general population. gap of 44 pps gap of 46 pps Access to healthcare: Reduce the gap in health status between the Roma and the rest of the population Limited progress: self-reported health status ‘very good’ or ‘good’ has reached the same level as that of the general population However, this is self-reported and no progress in access to medical insurance coverage could be observed . Gap of 13 pps Gap of 0 pps Access to housing: Close the gap between the share of Roma with access to housing and to public utilities (water, electricity, gas) and that of the rest of the population No progress: The housing situation for Roma remains difficult. The gap between Roma and the general population remains and is even widening. gap of 29 pps gap of 0.66 rooms per person gap of 33 pps gap of 0.79 rooms per person Source: Roma: Based data quoted in FRA (2018) — A persisting concern: anti-Gypsyism as a barrier to Roma inclusion; General Population (Eurostat); Data: Percentage point difference between Roma and General Population indicators for nine Member States covered by FRA 2011 Roma / 2016 EU-MIDIS II surveys. Modest progress has been observed for the general objective of fighting poverty. According to the 2011 FRA Roma survey, on average 86 % of the Roma population surveyed were at risk of poverty. The situation had improved somewhat by 2016, when 80 % of Roma surveyed were found to be at risk . Table 5: FRA Roma survey results on poverty 2011 2016 Gap Roma General population Roma General population 2011 2016 People at risk of poverty 86 % 19 % 80 % 20 % 67pps 60pps Discrimination continues to be of high concern in European societies, despite evidence of some improvement in a few areas. The enforcement of anti-discrimination law regarding Roma in the period 2011-2016 shows a scattered, mixed picture among Member States. However, in most countries the situation has not changed in this five-year period . According to surveys the overall share of Roma who felt discriminated against when in contact with schools (as a parent or student) has not decreased significantly (17 % to 14 %). There has, however, been a significant decrease when accessing health services (20 % to 9 % in the case of men and 19 % to 7 % in the case of women) . In the past 5 years, the share of Roma who felt they were discriminated against when looking for housing has dropped slightly (from 45 % to 41 %) but the extent of discrimination felt by Roma ‘when looking for a job’ and ‘at work’ remains high (40 % and 17 % respectively), even though for the former there has been a considerable drop of 10 %. In 2016, almost every third Roma (30 %) had experienced some form of harassment in the past 12 months that they felt was due to their ethnicity . For 76 % of the respondents, harassment due to ethnicity is a recurring experience. Instances of antigypsyism in the form of hate crime against Roma diminishes Roma people’s trust in their public institutions, in particular in law enforcement and justice, thus undermining social inclusion efforts . Antigypsyism is reported to have increased, fuelled by the stigmatisation of the Roma community by some mainstream political parties . The majority of the OPC respondents identify rising discrimination and antigypsyism as main challenges to Roma inclusion at both EU and national level . The OPC respondents confirmed that the most significant progress was achieved in education. In health, employment and housing, most respondents see no change. Moreover, while on health more people see improvement than deterioration, opinions are divided on employment, whereas on housing those who consider the situation to be worsening outnumber those that see improvement. On discrimination, the biggest response is ‘no change’, and many more people refer to a worsening situation than to improvements. Figure 1: OPC— Perceived changes in the situation of Roma since 2011 Overall, 6 years is too short a period to make significant improvements. Structural changes require time and the real impact of progress (including a change in employment trends, resulting from positive steps in education) may only be seen in a generation. However, the evaluation indicates that further progress can still be made in the remaining period provided that efforts to combat discriminatory attitudes and tackle antigypsyism increase. Favourable economic developments, falling unemployment rates and increasing labour shortages in the EU could provide some conditions for progress in terms of improving the employment situation of Roma. 5.1.2 Alignment between the EU framework and NRIS

All Member States and enlargement countries have a NRIS as well as a National Roma Contact Point with the authority to coordinate the development and implementation of NRIS in place. A total of 71 % of OPC respondents stated that one of the main achievements of the EU framework had been that Member States developed NRISs. Member States make a differentiated use of these instruments, depending on country specificities. In total, 15 Member States have adopted strategies, while the remaining 12 Member States have adopted integrated sets of policy measures within their broader social inclusion policies . The human and financial resources, as well as the mandate and tasks of NRCPs vary. While with few exceptions they are in charge of coordination and monitoring of NRIS and almost all of them also facilitate civil involvement, fewer are involved in development of other policies and the planning of the use of EU funds, and less than half have regular contacts with equality bodies. The placement of NRCPs within the national government structures also vary: Some are placed in institutions with a wider social inclusion agenda with relatively strong mandate and influence on mainstream policies, while others are in well-established institutions responsible for a specific narrower policy agenda (such as housing). Some NRCPs rather have a weaker technical or policy communication role with other institutions in place coordinating social inclusion. The EU framework called on Member States to set achievable national targets for Roma integration — addressing the four EU Roma integration goals as a minimum — to bridge the gap with the general population. The below analysis shows that in particular those countries with the highest share of Roma fully addressed the Roma integration goals in their NRIS (Bulgaria, Spain, Hungary, Romania, Slovakia), complemented by Finland. Table 6: Inclusion of national goals explicitly addressing the Roma integration goals Roma integration goal No Yes Partial Ensure that all children complete at least primary education CZ, DE, EE,FR IE, IT, CY LV, LT, LU, NL, PL, PT, SI, SE, UK BE, BG, DK, EL, ES, FR, HR, HU, RO, SK, FI former Yugoslav Republic of Macedonia, Kosovo, Serbia, Turkey AT Albania, Bosnia and Herzegovina, Montenegro Cut the employment gap between Roma and the rest of the population DK, DE, EE, IE, EL, FR, IT, CY, LV, LT, LU, NL, AT, PL, PT, SI, SE, UK BE, BG, ES, HR, HU, RO, SK, FI former Yugoslav Republic of Macedonia, Kosovo, Serbia, Turkey CZ Albania, Bosnia and Herzegovina, Montenegro Reduce the gap in health status between the Roma and the rest of the population BE, CZ, DK, DE, EE, IE, EL, FR, HR, CY, LV, LT, LU, NL, AT, PL, PT, SI, SE, UK BG, ES, IT, HU, RO, SK, FI former Yugoslav Republic of Macedonia, Kosovo, Serbia, Turkey Albania, Bosnia and Herzegovina, Montenegro Close the gap between the share of Roma with access to housing and that of the rest of the population BE, DK, DE, EE, EL, FR, HR, CY, LV, LT, LU, NL, AT, PL, PT, SI, UK BG, CZ, ES, HU, RO, SK, FI, SE former Yugoslav Republic of Macedonia, Kosovo, Serbia, Turkey IE, IT Albania, Bosnia and Herzegovina, Montenegro Source: ICF Research based on NRIS and NRCP interviews; complemented with a contribution from DG NEAR The cases of incomplete alignment with the Roma integration goals above can be explained primarily by the fact that the Council used the following wording to endorse the Roma integration goals: ‘four goals of ensuring equal access in education, employment, healthcare and housing ’. The scope was therefore broader, providing more flexibility to Member States in shaping their national goals, taking into account the size of their Roma population, different starting points and country-specific features, for example in the composition of the Roma population. The same reasoning applies to the measures taken by the Member States in the four key areas. As shown in the comparison between the 2013 Council Recommendation and the EU framework set out in Annex 6 to this SWD, the Recommendation has widened the set of possible measures per area but strengthened their voluntary nature by stating that the Roma integration goals could be attained by means of measures such as those listed. 5.1.3. Setting up other instruments and structures for Roma integration and increasing capacity of those involved at national and local level At EU level, this evaluation finds that that the EU framework was effective in introducing a number of instruments (in particular annual reporting by Member States and the Commission, recent reporting by civil society) and EU-level governance structures (for example the NRCP network — see coordination section). Mirroring the Member States’ practices, similar instruments have been introduced in the enlargement region. With regard to instruments, the 2013 Council Recommendation introduced comprehensive reporting by Member States from 2016. The reporting has been put in place but focuses on policy measures taken by Member States, rather than on outcomes. Independent monitoring, in particular through FRA surveys, is still hampered by the relatively low coverage of Member States (although the nine Member States covered in both 2011 and 2016 surveys make up more than 80 % of the EU Roma population). In 2017 the Commission piloted a project to monitor the implementation of the NRIS by civil society in all Member States but Malta . As for the structures, even if some of them predated the EU framework (the Roma task force was created in 2010, while the European Platform for Roma Inclusion met for the first time in 2009 ), they were strengthened and made permanent following the adoption of the EU framework. Notably, 66.7 % of OPC respondents recognised that the development of these structures at EU level was one of the key achievements of the EU framework. With regard to the capacity of national actors to implement Roma integration measures, the evaluation shows that today there is better awareness and more knowledge about Roma integration. During the interviews with NRCPs, nine Member States in total reported an increase in the number of government officials working on Roma integration in response to the EU framework. Closer engagement with Roma stakeholders either at local or central level, including via formal Roma platform structures at national level, has helped increase understanding of Roma issues among government officials. However, some NGO representatives voiced the criticism that this increased understanding had not sufficiently been translated into action. Roma participation has increased at national level and most NRISs have systems of consultation with Roma civil society in place. However, there remains a need for more effective Roma engagement at all levels, including from the gender perspective. The quality of civil society participation is still considered inadequate in terms of stakeholder involvement in the monitoring progress. Furthermore, cooperation between Roma NGOs and broader social NGOs rarely occurs in practice, and Roma issues are barely present in the mainstream social inclusion agenda. 5.1.4. Increase in number of and funding for Roma integration measures and in Roma beneficiaries Finally, the evaluation finds that there is insufficient quantitative data available at EU and national levels to assess whether the EU framework has contributed to an increase in the number of Roma integration measures, the funding attributed and the number of Roma beneficiaries. As described in the EU added value and coherence sections, the EU framework and EU structural funds have been successfully aligned for the 2014-2020 programming period. In total, 12 Member States (AT, BE, BG, CZ, ES, FR, EL, HU, IT, PL, RO and SK) selected investment priority (IP) 9.2 on socio-economic integration of marginalised communities such as the Roma, allocating it a total of EUR 1.5 billion, or 3.6 % of European Social Fund (ESF) funding. However:

Øthe Investment Priority covers marginalised communities in general (and not exclusively Roma) thus making it more difficult to quantify concrete amounts spent on Roma integration; Øthe ESF output indicator for participants ‘migrants, participants with a foreign background, minorities (including marginalised communities such as the Roma)’ is not specific enough to provide a reliable assessment of the number of Roma beneficiaries ; and particularly Øno comparison can be made with the 2007-2013 programming period, which took a mainstream approach with no such IP and indicator. The evaluation could nevertheless provide some qualitative insights. For example: Øregarding EU funding in 10 Member States, the consensus among interviewed stakeholders was that it has increased , in particular from the ESF; Øin the enlargement countries, all respondents expressed a similar view on the impact of IPA funding ; Øin seven Member States , there was consensus that there had been no increase, while in another eight Member States the government officials interviewed did not know if and how funding had changed; Øthe majority of OPC respondents (53.5 %) also agreed that the EU framework had contributed to an (increased) allocation of EU funding under ESIF (2014-2020) for Roma integration; Øregarding national funding, the consensus from the in-depth interviews in 13 Member States was that there had been no increase ; ØOPC respondents held similar views: 52 % stated that the EU framework had not contributed to (increased) allocation of national funding for Roma integration; Øin the enlargement region, respondents argued that prior to the financial crisis, budgets for Roma integration were generally on the rise, except in Turkey and Kosovo; and Øthe crisis provoked a drop in the budgets in most countries in the region that neither the Roma Decade nor the EU framework reversed .



5.2. Coordination The analysis of coordination considers to what extent actions carried out under the EU framework have been coordinated to maximise effects. It looks at coordination at EU and national level and at cooperation with stakeholders. Coordination is closely linked to effectiveness but treated separately as a key precondition for delivering the EU framework. Coordination of the EU framework has been assessed as being positive overall at EU level and limited at national level. The assessment has been informed by five evaluation questions that looked into the extent to which: ·coordination mechanisms at EU and national levels have been established in response to the EU framework and whether they are effective; and to which ·stakeholders — in particular civil society — both at EU and national level, are involved in implementing, planning and monitoring the EU framework and the NRIS.

5.2.1. Coordination mechanisms At the EU level, several coordination mechanisms supporting the EU framework have been set up at EU level. Box 1: Coordination mechanisms at EU level and in the enlargement region European Commission Roma Team: the specific team within the ‘Non-discrimination and Roma coordination unit’ (in addition to the non-discrimination and the legal teams), coordinates the implementation of the EU Roma framework. In addition, since 2014, a Special Adviser is in place to coordinate Roma integration policies in the enlargement and neighbourhood regions. In the enlargement region, the Special Adviser is supported by a Roma network composed of representatives from the Commission geographic teams and the Roma contact point from each EU delegation. The EU Anti-trafficking Coordinator has also been working closely on the risk of trafficking in Roma, in particular of women and children. The coordinator is responsible for: (i) improving coordination between EU institutions, EU agencies, Member States and those involved at international level; and (ii) developing existing and new EU policies to address trafficking in human beings. European Platform for Roma Inclusion: the platform brings together national governments, the EU, international organisations, representatives of civil society and local authorities on an annual basis. It aims to encourage cooperation and the exchange of experiences on successful Roma inclusion . Network of NRCPs: created in 2012 as a mechanism of coordination among Member States and between Member States and the Commission, this network meets twice per year. Its key role is to facilitate coordination between the Commission and the NRCPs. In 2016, the network was extended to the enlargement region. Task force of the ‘Roma Integration 2020’ initiative in the enlargement region: the task force organises annual task force meetings and regional thematic workshops involving, among others, NRCPs and Roma NGOs from enlargement countries . European network on social inclusion and Roma under the structural funds (EU Roma): This is a learning network which includes NRCPs and managing authorities, focusing on the use of ESI funds for the inclusion of Roma . Roma task force: Commission internal group involving several Directorates-General and the FRA . Consultation meetings with civil society and international organisations: the Commission regularly organises meetings bringing together local authorities, civil society, international institutions and organisations (Council of Europe, UN agencies, OSCE). These mechanisms provide a permanent system of cooperation. During the workshop, several NGOs highlighted the stronger cooperation between all relevant stakeholders resulting from the EU Roma framework, and acknowledged the role of the Commission in achieving it. The network of NRCPs is considered by the NRCPs interviewed as a good opportunity for establishing contacts with colleagues in other Member States dealing with similar situations and for exchanging practices . At the national level, NRCPs are the main coordinators, having the mandate to coordinate the development and implementation of the NRISs. In addition, the national Roma platforms established in most Member States and all enlargement countries aim to ensure inclusive involvement of and coordination with all relevant stakeholders (such as civil society, public authorities, Roma and non-Roma communities and business). In the enlargement region, since 2011 in all countries except Turkey, the government and the Commission organise a national Roma integration seminar every 2 years. The objective is for all partners (central and local authorities, civil society, international organisations, donors) to monitor progress and agree on the Roma integration priorities to be implemented in the next 2 years. Many of the NRCPs interviewed in the EU and enlargement countries (11 out of the 19 that provided an opinion) say they have sufficient administrative capacity to effectively coordinate NRIS implementation . The NRCPs have improved cooperation among public administrations and between public administrations and other stakeholders. However, given their mandate and power, they have a limited influence on the design and implementation of mainstream policies and, more generally, on policies implemented by other institutions at the national and local level . Coordination is not yet well integrated into the policy cycle of planning, budgeting, implementation, monitoring and evaluation. Moreover, NRISs are not well known, even in some cases by key administrative departments in the government. NRCPs have limited power in influencing decision-making processes across policies, which is a major obstacle for their NRIS coordination capacity. In other cases, insufficient coordination seems to stem from a lack of commitment by ministries, the political context or the lack of participation at local and regional levels . 5.2.2. Stakeholder involvement At the EU level, stakeholder cooperation includes the Council of Europe, the Economic and Social Committee, EU agencies such as the FRA, UN agencies, the OSCE, EEA/Norway Grants, the World Bank, civil society associations and foundations, representatives of municipalities, thematic umbrella organisations and others. In enlargement countries, partners include a number of bilateral and multilateral donors. Stakeholder cooperation and dialogue has been developed through regular events such as the EU Roma Summit, Roma platforms, and consultation meetings with civil society and international organisations. Synergies were achieved through the cooperation with the Council of Europe in different joint programmes . With regard to ensuring their participation in decision-making at EU-level, Roma organisations think that : (i) dialogue with the Roma team and other Commission departments could be more permanent; (ii) information and transparency could be improved; and (iii) the Commission should help facilitate dialogue with the NRCPs. At national level, stakeholder participation — including of Roma representatives and civil society — has been boosted through the EU framework. New mechanisms introduced are diverse and include: (i) national Roma platforms; (ii) annual national progress reports that are shared with stakeholders; (iii) dedicated advisory committees; (iv) projects to further strengthen cooperation with Roma NGOs; (v) regular consultation meetings with Roma representatives and other stakeholders (e.g. municipalities); and (vi) expert groups including both government and NGOs representatives . Despite these efforts, weaknesses have been identified with regard to the involvement of civil society in the practical implementation of the NRIS and in the monitoring and evaluation processes. As regards planning and implementation of the ESIF, it is suggested that the degree and quality of stakeholder participation varies across countries, and ‘in many cases the mechanisms and processes for involvement are weak (e.g. informal)’ or limited to specific phases of the policy cycle . The following chart illustrates this mixed assessment. Figure 2: Reply to: ‘Please indicate the extent to which you agree with the following statement: The existing mechanisms for coordination and implementation of the NRIS of my Member State allow for effective cooperation and/or consultation with all key stakeholders.’





5.3. Relevance The analysis of relevance looks at whether the EU framework objectives set in 2011 were appropriate to the needs at the time and whether they continue to be relevant today. Relevance of the EU framework has been assessed as positive overall, with some limitations as set out below. The analysis has been informed by seven evaluation questions that looked into the extent to which: ·the EU framework’s original objectives were appropriate in view of the original needs; and ·the EU framework’s original objectives are appropriate in view of the current needs. Annex 6 provides an overview of the objectives of the EU framework and the proposed measures for Member States, together with an overview of how these have been adapted and extended by the 2013 Council Recommendation. In general, as explained above, while respecting Member State competences, the EU framework provides flexibility so that its objectives can be adapted to specific national contexts and changes that take place over time. This flexibility implies that the relevance of the EU framework hinges closely on the appropriateness and ambition of the concrete objectives and measures set out in the individual NRISs. The flexibility and the non-binding nature of the EU framework have allowed Member States to follow a tailored approach taking account of specific national contexts when selecting measures. This evaluation therefore assesses the relevance of the EU framework’s general and specific objectives but not the relevance of the individual measures proposed. 5.3.1. Relevance in light of the original needs The evaluation finds that the original objectives set in 2011 were appropriate in view of the Roma integration needs at the time, both in the Member States and in the enlargement countries. Overall, taking into account the baseline situation described above and the persistent socio-economic exclusion of and discrimination against the Roma, the general objectives on poverty and discrimination and the specific objectives on employment, education, health and housing have been confirmed as relevant for Roma inclusion. This view was clearly shared by the workshop participants when discussing the preliminary findings of this evaluation. During the OPC, an overwhelming majority of respondents (between 86 and 95 % depending on the thematic area) agreed that targeted public interventions are needed in these areas. The evaluation finds, however, that for education, housing and non-discrimination, relevance of the EU framework could have been stronger. Education: while relevant, the Roma integration goal in the area of education was considered to be insufficiently ambitious by several civil society organisations and interviewees . Indeed, primary school completion rates of Roma children were already quite high at the time the EU framework was introduced. The relevance of the EU Roma framework could have been increased by other objectives such as: (i) the transition of Roma from lower secondary to upper secondary education; (ii) reducing the share of early leavers from education and training; and (iii) focusing on reducing gaps between Roma and the rest of the population in the other areas. Housing: the Roma integration goal to close the housing and essential services gap between Roma and the rest of the population is a key factor for improving the integration of Roma into society. While acknowledging that the EU framework’s specific objective was appropriate in view of the original needs, the civil society stakeholders consulted expressed regret that there were no tailored housing objectives across the Member States to address country-specific needs. The 2011 FRA Roma survey and the data collected for the country analysis reports indicate that deprived living conditions, lack of basic infrastructure and segregation among Roma, was worst in Eastern European countries — particularly so in Hungary, Romania and Slovakia. In countries such as Austria and Finland, the problems faced by Roma mainly related to access to housing due to discrimination, rather than to housing conditions or housing segregation . Non-discrimination: a key conclusion of the workshop was that the EU framework’s relevance was reduced because non-discrimination received insufficient attention. Indeed, the EU framework did not set a specific non-discrimination goal but referred to the existing legal obligations of Member States.

The evaluation also finds that the following factors reduced the relevance of the EU framework in light of the original needs. 5.3.2. Relevance in light of the current needs The evaluation finds that the original general and specific objectives continue to be relevant today, both in Member States and in enlargement countries. As set out in the effectiveness section, although there is visible progress on education, progress in other areas is slower. Despite better economic conditions, Roma continue to have much higher rates of being at risk of poverty, having lower employment and higher numbers of NEETs, substandard housing conditions and unequal access to healthcare services. The survey of national NGOs also shows that the objectives remain largely relevant to the current needs . a) With regard to the four Roma integration goals (see effectiveness section for more detail on the 2016 situation of Roma in these areas): Education: this goal would be more relevant to current needs if it did not focus exclusively on primary education but was extended to aspects such as transition from lower to upper secondary school attendance, early school leaving (ESL) and transition from education to employment. These are areas were the gaps between Roma and non-Roma remain high. Health: despite some improvements, the analysis finds that the Roma integration goal in the area of health is still appropriate today. Although the gap between Roma and non-Roma largely disappeared for self-reported health status, it remained practically unchanged for medical insurance coverage since 2011 (see effectiveness section). The actual (not self-reported) health status of Roma remains unknown, but EU-wide studies reveal ‘indirect evidence’ such as shorter life expectancy versus the population as a whole, that clearly indicates a health disparity . The Roma population is also disproportionately affected by communicable diseases . Persistently poor living conditions often result in a higher probability of serious illnesses and chronic diseases, even when access to healthcare is provided . Furthermore, little progress has been made on preventative healthcare, as Roma continue to have consistently lower child vaccination rates compared to non-Roma . Employment: the original objective still corresponds to the current needs, as the employment gap between the Roma and the general population remains very high. Furthermore, nearly half of the respondents to the NGO survey considered that the needs of Roma youth and Roma women were not addressed in employment measures and programmes . Housing: this Roma integration goal also continues to be relevant to the needs of the Roma. About half of the country analysis studies identify housing as one of the most important areas for further work — a view that is shared by several stakeholders consulted in the Member States and enlargement countries. The OPC confirms the lack of progress between 2011 and 2016, with more than half (57.3 %) of respondents indicating the housing situation has worsened due to higher levels of housing discrimination. b) In light of the current needs, the evaluation finds that the general objective of non-discrimination needs updating. Already in 2016 the Commission highlighted the importance of more systematic approaches and greater political will on the part of Member States in fighting antigypsyism. Levels of discrimination remain high (see section on effectiveness) and antigypsyism is a critical issue as recently highlighted by the European Parliament and the Fundamental Rights Agency . Stakeholders, including workshop participants , found that compared to the general objective of combating poverty and the four specific objectives, giving more weight to the anti-discrimination objective is critical to address antigypsyism and its manifestations in hate speech and crime . The OPC results show that, in addition to noting a rise in discrimination, respondents find that little has been done to actively fight discrimination within the four main policy areas over the course of the current EU framework.

Figure 3: Perception of progress/implemented measures to fight discrimination in the four main policy areas 2011-2016 Source: OPC — Public consultation on the Evaluation of the EU framework for national Roma integration strategies up to 2020, 2017. In order to make more progress towards the Roma integration goals, more efforts are thus needed to fight discrimination and anti-gypsyism if equal access to education, employment, health and housing is to be ensured. c) The EU framework’s objectives remain relevant also from an economic perspective The effects of Roma exclusion on welfare dependency, the labour market and tax revenues were already clear at the time the EU framework was adopted. Since then, for the five Member States with the highest share of Roma (BG, CZ, HU, RO and SK) the country reports prepared in the context of the European Semester have repeatedly underlined the risks of adverse demographic trends. Low fertility rates are expected to cause the working-age population to shrink, leading to difficulties with labour supply, shortages of skilled and unskilled workers. These factors are projected to hamper potential growth in the next years . In this climate, the economic rationale for providing Roma with skills and integrating them in the open labour market is clear in ageing Central Eastern European societies. Recent research carried out for the European Parliament illustrates the economic impacts of discrimination on grounds of race and ethnicity, ranging from GDP and tax revenue loss to direct costs in mental health provision . Roma integration is also relevant for attaining the Sustainable Development Goals of the United Nations 2030 Agenda for Sustainable Development, as many countries cannot do this without making progress on Roma inclusion . 5.4. Equity For this evaluation, the analysis of equity considers whether the EU framework has the capacity to address the needs of subgroups among Roma. Equity is closely related to effectiveness and relevance (hence its inclusion here directly after the effectiveness and relevance chapters), but looks more deeply into the needs of Roma subgroups. This analysis is important as ‘the Roma’ targeted in the EU framework may experience different forms of disadvantage which need to be addressed. The analysis is therefore presented separately. Equity of the EU framework has been assessed as limited overall. The assessment was informed by four evaluation questions that looked into the extent to which the EU framework addresses the needs of subgroups among Roma. The evaluation concludes that the EU framework, as adopted in 2011, does not sufficiently address the needs of specific subgroups. The country studies and the targeted stakeholder consultations reveal that whenever special attention is given to vulnerable Roma subgroups at national level, Roma children, youth and women are most likely to be included , and Roma with disabilities to a much lesser extent . Various sources, including contributions during the workshop, the country studies and interviews with national stakeholders, indicate that several other subgroups currently receive insufficient attention, if any at all, in the EU framework and at national level. These include Roma migrants , LGBTI Roma and Muslim Roma . Insufficient attention is also given to the distinction between urban and rural Roma . With regard to the three subgroups, the EU framework only very briefly refers to the situation of Roma women, pointing out their lower employment rates and the need for better access to quality healthcare. This was partly rectified by the new orientation given in the 2013 Council Recommendation, which also speaks about the need to fight violence, including domestic violence, violence against women and girls, trafficking in human beings, underage and forced marriages, and begging involving children. However, the absence of an explicit focus on gender equality in the EU framework has meant that the NRISs lack indicators and targets to tackle Roma women’s specific situation, and also lack the political commitment at national level, to take a more proactive approach to addressing gender needs . Many NRIS either take a gender-neutral approach or they lack explicit gender measures in nearly all areas except for health . Some enlargement countries provide a positive example, such as the former Yugoslav Republic of Macedonia, where a national plan for strengthening the position of Roma women includes the goal of Roma women and girls’ enjoyment of their human rights without discrimination . Among the four areas of the EU framework, it is in the employment area that the gap between Roma men and Roma women has increased between 2011 and 2016, both in terms of employment rates and NEET rates.

Table 7: Difference between male and female experiences in employment (2011-2016) Difference b/w male / female 2011 in pp Difference b/w male / female 2016 in pp Roma Male (2016) Roma Female (2016) Share of people with the self-declared main activity ‘paid work’ (including full-time, part-time, ad hoc jobs, self-employment), household members, 16+ (%) 11 18 34 16 Share of young persons, 16-24 years old whose current main activity is neither in employment, education or training, household members (%) 11 17 55 72 Source: Unpublished information from FRA Roma Survey (2011) and EU-MIDIS II survey presented by FRA to Commission services in February 2018 As set out in the relevance section, stakeholders consider that gender equality was highly relevant to the needs of the Roma at the time the EU framework was adopted and thus merited more attention . The above findings in the area of employment confirm this view. The EU framework mentions that young Roma should be strongly encouraged to participate in secondary and tertiary education but provides no further guidance or specific objective. Today however, the rising levels of young Roma aged 16 to 24 who are not in employment, education or training (NEET) — an increase from 56 % in 2011 to 63 % in 2016 — is a challenge that needs to be addressed. Research at national level indicates that only a small number of countries include references to the needs or special situation of young Roma in their NRIS . One factor that could explain the absence of focus on Roma youth at national level is the lack of statistical data on Roma youth. This was identified in several country studies as a factor hindering insight into the situation of young Roma that could help those who advocate for Roma youth measures or analyse the extent to which young Roma benefit from mainstream measures . The results of the NGO survey confirmed the view that key policy measures for Roma integration at national level only address the needs of Roma youth to a limited extent. The respondents were divided on whether Roma young people’s needs were addressed in the area of education, while they largely agreed that they were not sufficiently addressed on healthcare, employment, housing and equality and anti-discrimination. As is the case for Roma women, it was not the EU framework but the 2013 Council Recommendation that put a focus on the needs of Roma children. In 2015, the European Parliament stressed the need to prioritise Roma children in the promotion of the Roma’s access to education, healthcare and housing when implementing the EU framework . In 2016, the Commission called the situation of Roma children ‘particularly worrying ’. Country analysis studies and targeted stakeholder interviews only provided limited information on specific measures for Roma children in the Member States. According to OPC respondents, education is an area in which at least some of the needs of Roma children were addressed. However, 71% of the OPC respondents disagreed that measures were taken in terms of promoting Roma girls’ school participation and 65% of OPC respondents disagreed that measures against misdiagnosis and misplacement of Roma children were taken. Both Roma EU-mobile citizens and Roma third-country nationals are barely considered in the EU framework and NRIS of reception countries . We must remember that restrictions on the free movement of workers from Bulgaria and Romania after EU accession were lifted in 2014, and while this was not specific to Roma, it also affected them. Since that date, a significant number of people from these countries moved to EU-15 Member States . The 2013 Council Recommendation already highlighted the need to ‘respect the right to free movement’ of EU citizens and ‘the conditions for its exercise, including the possession of sufficient resources and of a comprehensive sickness insurance cover, in accordance with Directive 2004/38/EC of the European Parliament and of the Council, while also seeking to improve the living conditions of Roma and pursuing measures to promote their economic and social integration in their Member States of origin as well as their Member States of residence.’ With regard to Roma who are third-country nationals staying legally in a Member State, the 2013 Council Recommendation states that they can also be put in a vulnerable position, particularly when they share the same poor living conditions as many Roma who are citizens of the EU, while also facing the challenges of many migrants coming from outside the EU. Country studies confirm the high level of discrimination faced by Roma migrants and EU-mobile Roma for being Roma and migrants at the same time, their precarious living conditions and issues related to their residence status . Several interviews with stakeholders have highlighted concerns about Roma migrants and the need to recognise this group as a specific area for action under the EU Roma framework .



5.5. Coherence The analysis of coherence focuses on how well the EU framework works together with other EU and national instruments. The coherence of the EU framework has been assessed as positive at EU level and as limited at national level. The assessment has been informed by nine evaluation questions that looked into the extent to which : ·the EU framework is consistent with other EU policies; ·policy, legal and financial instruments at EU level were mobilised and aligned to accomplish the objectives of the EU framework; and ·policy, legal and financial instruments at national level were mobilised and aligned to meet the objectives of the EU framework. 5.5.1. EU level: internal coherence The EU framework was the first comprehensive EU approach to Roma integration across the four policy areas. However, the formulation of the Roma integration goals was not consistent or in line with the SMART criteria (specific, measurable, achievable, relevant and time-bound ). While the goal for education was not particularly ambitious, considering the situation in 2011 the goal for housing was very ambitious. The goals were not easy to measure given issues of comparability of data for Roma and for the general population, nor were they adequately specific. The fact that three goals aim to bridge gaps between Roma and non-Roma while the education goal does not relate to gaps, is another inconsistency.

The evaluation also finds it less coherent that in contrast to having Roma integration goals in the areas of employment, education, housing and health, the EU framework did not include a specific non-discrimination goal . Non-discrimination was rather set as a cross-cutting theme , which was identified as a significant shortcoming of the existing EU framework by the European Court of Auditors and others. However, consistency between the EU framework and the work of the FRA is found to be strong. The FRA has generated valuable evidence and data on living conditions and discrimination of Roma, in particular through its 2011 Roma pilot survey, the 2016 second EU minorities and discrimination survey (EU-MIDIS II) and the Roma integration indicators scoreboard (2011-2016) . In 2012, the FRA, in close cooperation with the European Commission, established a working party on Roma integration indicators to help EU Member States develop indicators to monitor the progress of the NRIS. 18 Member States participated in the process, represented by their NRCPs, which consulted with a variety of national stakeholders including competent ministries, statistical agencies, structural funds managing authorities, regional and local authorities, Roma civil society and communities, and other institutions that have a role in implementing the NRIS . This process resulted in the development of the online reporting system used by the Commission to collect annual information on implementation of the NRISs. 5.5.2 EU level: external coherence Intervention logic Annex 7a illustrates that progress on Roma integration depends on consistency between the EU framework’s objectives and those of other EU and national policy, legal and financial instruments. Table 8: Coherence at EU level (partly based on ICF, Chapter 3.3) EU Policies Coherent Partial Not coherent Comment Mainstreaming Roma integration in EU policies X There is consistency between the EU framework and other key EU policies (for example European Semester) but some gaps were detected. Fighting discrimination and racism legislation X The EU framework is externally overall consistent with instruments in this area but the lack of measurable anti-discrimination target leads to partial coherence EU funding X The introduction of the ESIF ex ante conditionality and IP 9.2 has increased consistency. Mainstreaming into policies Overall, progress has been made in mainstreaming Roma integration in EU policies. On employment, education and poverty, 3 out of 5 Europe 2020 headline targets are directly linked to the EU framework targets for Roma integration: (i) the fight against poverty and social exclusion; (ii) raising employment levels; and (iii) reducing school dropout rates while increasing attendance in tertiary education. Making sufficient progress towards these three Europe 2020 targets requires addressing the situation of Roma . The 2013 Council Recommendation on Roma integration therefore called on Member States to ‘… depending on the size and social and economic situation of their Roma populations, consider making Roma integration an important issue within their national reform programmes or their national social reports in the context of the Europe 2020 strategy.’ Country-specific recommendations (CSRs) have been addressed to Bulgaria, the Czech Republic, Hungary, Romania and Slovakia in the context of the European Semester since 2012. In addition, the ESIF regulations established a link between the European Semester and EU funding in the 2014-2020 ESIF programming period. Member States which at the time of programming had received CSRs on Roma integration were to include the above-mentioned IP 9.2 in their operational programmes (OP) and use EU funds for necessary reforms. This in turn made the Roma-related thematic ex ante conditionality and its fulfilment criteria applicable, closely linked to the objectives of the EU framework. Other policies also include an explicit link to the EU framework. ØRoma integration has been fully mainstreamed into European enlargement policy and explicitly identified as one of the key priority areas. Progress on Roma integration is now also fully included in the negotiations of enlargement Chapter 23 ‘Judiciary and Fundamental Rights’. Furthermore, the Commission analyses the state of play by country in the context of the annual enlargement package. ØRoma integration has been reflected in the EU urban agenda, allowing for a better mainstreaming of Roma integration into urban policies . ØThe framework is coherent with EU anti-trafficking policies which often address the situation of Roma and their particular risks and vulnerability to trafficking in human beings .

ØIn the area of health, the European Commission’s recent proposal for a recast of the Drinking Water Directive refers directly to the 2011 EU framework. There are also policy initiatives with no explicit link to the objectives of the EU framework. Roma integration is not explicitly considered in social policy and education initiatives such as the Youth Guarantee , the Skills Agenda, the Alliance for Apprenticeships and the Pillar of Social Rights. These initiatives take a mainstream approach to vulnerable and disadvantaged groups, mentioning Roma specifically either fleetingly or not at all. The non-explicit reference to Roma in initiatives such as the ones mentioned may be the reason why a large share of the OPC respondents (88.9 %) identified the lack of effective mainstreaming of Roma integration in policy as one of the main challenges at EU level in the 2011-2016 period. Legislation on fighting discrimination and racism The Racial Equality Directive, together with the Council Framework Decision on combating racism and xenophobia of 2008 and other instruments, has provided a normative ground for monitoring the fundamental rights situation of Roma . The Directive’s implementation has been monitored by the Commission And a review of the enforcement of anti-discrimination law in relation to Roma found that they do not benefit equally from the non-discrimination legal instruments, and a lack of enforcement is particularly evident in the areas of education and housing. Consequently, the Commission initiated infringement procedures over the school segregation of Roma children against three countries and put in place several policy tools to help national authorities implement anti-hate speech and hate crime legislation to combat antigypsyism among other issues. The EU framework’s socio-economic inclusion standards and the provisions of the EU legislation complement each other significantly in terms of aims and scope. The Commission’s enforcement of the EU legislation can strengthen the EU framework’s policy objectives, while the monitoring and implementation of the EU framework informs the Commission on the state of play of the EU legislation and supports its practical application. About 83 % of the OPC respondents considered the measures relating to the monitoring and enforcement of EU anti-discrimination laws as very useful for increasing political commitment to ambitious public policies for Roma inclusion . However, as set out under the ‘internal coherence’ section, the lack of specific objectives on non-discrimination is seen as a missed opportunity for full coherence. EU funding In the 2007-2013 programming period, EU funding regulations targeted disadvantaged groups in general, but not explicitly Roma. For the 2014-2020 period, however, substantial changes were made. The Rights, Equality and Citizenship Programme (REC) as well as the instrument for pre-accession assistance (IPA), made funding for Roma integration a priority. Going further than this, the ESIF regulations established an explicit link between the EU framework and EU funding. ØIP 9.2 of the ESF entitled ‘Socio-economic integration of marginalised communities such as the Roma’ was introduced. ØThe thematic ex ante conditionality ‘a national Roma inclusion strategic policy framework is in place’ was linked to the above IP and three European Regional Development Fund (ERDF) IPs dealing with investment in health and social infrastructure, support for regeneration of deprived communities and investment in education and training infrastructure . The fulfilment criteria for this thematic ex ante conditionality were fully aligned with the EU framework’s objectives. ØIn 2011 and 2015, the Commission issued thematic guidance on the use of ESIF in tackling educational and spatial segregation . For its internal organisation, it then prepared in 2014, thematic guidance for ESF desk officers on how to operationalise the objectives of the EU framework for the ESI funds . These measures have been considered to be an 'element of progress' , 'a promising practice' , as 'having provided additional opportunities to align Roma needs with mainstream policies' and as being 'instrumental in mobilising public administrations to work together and connecting Roma integration goals to mainstream policies' . In total, 12 Member States selected this IP (AT, BE, BG, CZ, ES, FR, EL, HU, IT, PL, RO and SK) allocating a total of EUR 1.5 billion, or 3.6 % of ESF funding. Across all ESIF, six Member States reported action plans related to the Roma-specific ex ante conditionality (BG, EL, HU, LT, RO and SK) in their partnership agreements. Hungary and Romania succeeded in meeting the conditions between the submission of the partnership agreement and the OP. Examples of actions related to the Roma strategy are: (i) developing strategies (e.g. RO) or action plans (e.g. LT); (ii) adopting revised strategies (e.g. HU); and (iii) developing an adequate monitoring system (e.g. EL, CZ and BG ). 5.5.3 National level At national level, limited progress has been made overall.

Table 9: Coherence at national level (partly based on ICF, Chapter 3) National Policies Coherent Partial Not coherent Comment Mainstreaming X The EU framework and NRIS have contributed to mainstreaming Roma integration into policies to a limited extent. National funding X National funding has been allocated to the implementation of the NRIS, complementing EU funding. Fighting discrimination and racism legislation X NRIS have contributed to fighting discrimination, hate speech and hate crime in some countries, while their contribution was minor or non-existent in others. Mainstreaming The evaluation finds that the EU framework has contributed to incorporating Roma integration into all policies to a limited extent at national level. Evidence of coherence is particularly limited in the health sector. In terms of employment, objectives were generally part of mainstream policies, but often without specific targeted measures or a specific reference to Roma inclusion into the labour market. On housing, Roma inclusion measures have been short-term actions rather than longer-term programmes. There is some reluctance to include specific references to Roma and their specific needs into mainstream policies to avoid singling out a specific vulnerable group . Thematic policies often address vulnerable or disadvantaged groups in general, without referring to the particular needs of specific groups, including Roma. Indeed, a large share of the OPC respondents (88.9 %) identified the lack of effective inclusion of Roma in policies as one of the main challenges at national level. National funding In the 11 countries analysed in depth for this evaluation, national funding was allocated for implementing the NRIS, in addition to or topping up EU funding. In the enlargement countries, all but Turkey have allocated specific funding (national/EU) to implement the strategy. In some countries, a consistent approach has developed over time between NRIS and funding mechanisms at national level, either through co-financing or an increase in national budget allocations for Roma integration measures. Quantification of amounts is difficult despite Member States providing the Commission with information on the implementation of their NRIS, including financial data, since 2015, through the Roma integration measures reporting tool. The information is incomplete due to the limitations of the tool explained in Annex 4. Fighting discrimination and racism The evaluation finds that the NRISs have contributed to fighting discrimination, hate speech and hate crime against Roma in some Member States, while in others their contribution was minor or non-existent. Overall, combating discrimination has not been at the core of the NRIS. The NGO survey respondents considered that the NRIS of their respective Member State contributed to some extent to reducing discrimination against Roma (40 %), reducing hate speech (50 %), reducing hate crimes (53 %) and improving the enforcement of anti-discrimination legislation (44 %) during 2011-2017 . Targets and measures to combat trafficking in human beings are included in the NRIS of only two of the countries researched for this evaluation (BG and HU). However, other countries have taken measures against trafficking in Roma people outside the scope of the NRIS.



5.6. Efficiency The analysis of efficiency assesses the relationship between the resources used by an intervention and the changes generated by it. The efficiency of the EU framework has been assessed as limited with regard to monitoring and reporting systems. Furthermore, costs and benefits could not be conclusively evaluated. Efficiency was analysed through five evaluation questions covering the following areas in particular: ·efficiency of the monitoring and reporting system, including scope for simplification; and ·costs and benefits in the context of the EU framework and NRIS. 5.6.1.