COMMISSION STAFF WORKING DOCUMENT Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS on the interim evaluation of the implementation of the Rights, Equality and Citizenship Programme 2014-2020
Date: 2018-06-29
Institutions: Directorate-General for Justice and Consumers, European Commission
Subjects: EU programme, European citizenship, area of freedom, security and justice, children's rights, civil society, consumer protection, data protection, equal treatment, gender equality, human rights

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 29.6.2018  SWD(2018) 358 final  COMMISSION STAFF WORKING DOCUMENT Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS on the interim evaluation of the implementation of the Rights, Equality and Citizenship Programme 2014-2020 {COM(2018) 508 final} - {SWD(2018) 359 final}  
 
 
Table of contents 

1. 

2. 

3. 

INTRODUCTION ................................................................................................................................ 2 

BACKGROUND TO THE INTERVENTION ..................................................................................... 3 

IMPLEMENTATION / STATE OF PLAY ........................................................................................ 10 

4.  METHOD ........................................................................................................................................... 19 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS .......................................... 23 

6.   CONCLUSIONS AND WAY FORWARD ........................................................................................... 32 

ANNEX 1: PROCEDURAL INFORMATION ............................................................................................ 35 

ANNEX 2: STAKEHOLDERS CONSULTATION .................................................................................... 38 

ANNEX 3: METHODS ................................................................................................................................ 40 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Purpose and scope of the evaluation 

1. 

INTRODUCTION 

This staff working document sets out the results of the interim evaluation of the Rights 
Equality and Citizenship (REC) Programme for the period 2014-2020.  

The evaluation was carried out under the provisions of Article 13(2)(b) of the European 
Parliament  and  Council  Regulation  establishing  the  Rights  Equality  and  Citizenship  
Programme for the period 2014-20201.  

It aimed mainly to assess the Programme’s outputs and results compared to its objectives 
and to assess qualitative and quantitative aspects of its implementation. It also assessed 
whether the current Programme is on track to achieving its objectives and the extent to 
which  the  recommendations  of  the  previous  2007-2013  ex-post  evaluation  have  been 
followed.  The  ex-post  evaluation  carried  out  in  2015  concerned  the  three  predecessor 
programmes  which  were  merged  in  the  current  Programme  (namely,  Daphne  III, 
Fundamental Rights and Citizenship and Progress)2. 

The  current  interim  evaluation  has  informed  the  Report  from  the  Commission  to  the 
European  Parliament,  the  Council  of  the  European  Union,  the  European  Economic  and 
Social  Committee  and  the  Committee  of  Regions  which  this  document  accompanies.  It 
also  helped  the  reflection  on  the  Programme's  future  by  providing  evidence-based 
information.  

Under  the  Regulation,  the  Commission  is  requested  to  present  an  ex-post  evaluation 
report for the Programme by 31 December 2021. 

The Programme was assessed based on the following main evaluation criteria: 

1.  Relevance:  whether  and  to  what  extent  the  Rights  Equality  and  Citizenship  
Programme  addresses  needs  and  problems  of  the  target  groups  identified  in  the  2011 
Impact Assessment3 and in the legal basis of the Programme, as well as emergent issues, 
and  whether  its  objectives  are  still  relevant  for  the  needs  and  problems  of  the 
beneficiaries; 

2. Efficiency: whether and to what extent the costs of the Programme were proportionate 
given the benefits achieved and which parameters/factors participated in these results; 

3.  Effectiveness:  whether  and  to  what  extent  the  Rights  Equality  and  Citizenship   
Programme has achieved its general objective, as well as its nine specific objectives, and 
which are the factors that have contributed to these achievements; 

4. Coherence/Complementarity/Synergies: whether and to what extent the Programme 
is  coherent  with  other  interventions  at  the  EU  and  international  level,  such  as  with  the 

Report. 

Specific 

1  Regulation  (EU)  No  1381/2013  of  17  December  2013  establishing  a  Rights,  Equality  and  Citizenship  Programme  for  the  period 
2014 to 2020 (OJ L 354, 28.12.2013). 
2  European  Commission  (2015),  Ex-post  evaluation  of  five  programmes  implemented  under  the  2007-2013  financial  perspective  – 
Programme, 
Final 
programme 
Specific 
http://ec.europa.eu/justice/grants1/files/expost_evaluations_2007_2013/daphne_programme_evaluation__final_report.pdf; 
Citizenship, 
programme 
http://ec.europa.eu/justice/grants1/files/expost_evaluations_2007_2013/frc_programme_evaluation_final_report.pdf; 
Ex-post 
evaluation of the Programme for employment and social solidarity – PROGRESS 2007-2013 and recommendations for the successor 
programmes to PROGRESS 2014-2020 http://ec.europa.eu/social/BlobServlet?docId=12610&langId=en  
3  European  Commission.  'Commission  Staff  Working  Paper  –  Impact  Assessment  -  Accompanying  the  document  Proposal  for  a 
Regulation of the European Parliament and of the Council establishing for the Period to 2020 the Rights and Citizenship Programme. 
Impact Assessment’, SEC (2011) 1364 Final. 

Fundamental 

evaluation: 

evaluation: 

Daphne 

Rights 

and 

2 

 
 
                                                            
predecessor  EU  programmes  in  the  field4,  with  activities  supported  by  other  Union 
instruments  and,  in  general,  with  the  European  priorities  in  the  fields  covered  by  the 
Programme; 

5. EU added-value: to what extent the effects from the EU action are additional to the 
value that would have resulted from action at the national level only; 

6.  Equity:  whether  and  to  what  extent  the  Programme  has  distributed  the  available 
resources fairly among beneficiaries in different Member States, took into consideration 
the  needs  of  target  groups,  promoted  gender  mainstreaming,  the  rights  of  the child  and 
the rights of people with disabilities; 

7. Scope for simplification: whether and to what extent the management of the Rights 
Equality and Citizenship Programme could be further simplified. 

Geographically,  the  Programme  is  open  to  all  EU  Member  States,  but  also  to  the 
European Free Trade Association States that are party to the European Economic Area, 
candidate countries, potential candidates and countries acceding to the Union, provided 
that  they  conclude  an  agreement  with  the  Union  laying  down  the  details  of  their 
respective participation in the Programme. Iceland joined for all specific objectives of the 
Programme,  while  Liechtenstein  participates  only  in  the  specific  objectives  related  to 
non–discrimination,  racism,  xenophobia,  the  rights  of  persons  with  disabilities  and 
equality between women and men. 

The reference period for this interim evaluation is the first half time of the Programmes’ 
implementation from 2014 to mid-20175. 

2.  BACKGROUND TO THE INTERVENTION 

Since  1948,  when  the  United  Nations  General  Assembly  adopted  the  Universal 
Declaration  of  Human  Rights,  an  elaborate  body  of  international  human  rights  law  has 
developed  through  state  practice,  the  work  of  international  courts  and  agreement  of 
multilateral treaties. Human rights law is currently enshrined in dozens of human rights 
treaties and  protected by organisations,  such  as  the UN, the Council of  Europe  and the 
African Union. 
Within EU borders, human rights are embedded in the EU treaties and reinforced by the 
EU Charter of Fundamental  Rights. These  rights include protection of individual  rights 
for  EU  citizens  as  well  as  promotion  of  equality  and  non-discrimination  for  all 
individuals in the EU. 
Since 2014, the Rights, Equality and Citizenship Programme aims at further developing a 
Europe of rights and equality in accordance with the Europe 2020 Strategy6.  
At the time  of its definition, the following  key problems and needs were identified in 
the Impact assessment of the three predecessor programmes7: 1. Limited effectiveness in 
the  implementation  of  activities  against  discrimination  on  the  grounds  of  sex,  racial  or 
ethnic  origin,  religion  or  belief,  disability,  age  or  sexual  orientation,  and  more  Roma 
integration  and  gender  equality;  2.  Insufficient  promotion  and  protection  of  the  rights 
enshrined in the EU Charter of Fundamental Rights, including the rights of the child and 

4 As already mentioned, the Rights, Equality and Citizenship Programme has replaced three programmes in the field of security and 
citizenship which were in force during the 2007-2013 Programming period. 
5 Given its relatively early stage of implementation, the 2017 annual work programme have been analysed only in terms of design and 
structure, not in terms of its execution. 
6  For  more  info  on  the  Europe  2020  Strategy,  see  https://ec.europa.eu/info/business-economy-euro/economic-and-fiscal-policy-
coordination/eu-economic-governance-monitoring-prevention-correction/european-semester/framework/europe-2020-strategy_en . 
7  European  Commission  (2015),  Ex-post  evaluation  of  five  programmes  implemented  under  the  2007-2013  financial  perspective  – 
Final Report, ibid. 

3 

 
                                                            
the rights of persons with disabilities, and in the UN Convention on the Rights of Persons 
with  Disabilities;  3.  Too  high  levels  of  gender  based  violence  and  too  little  support  to 
victims  of  violence;  4. Insufficient  protection  of  EU  citizen’s  personal data;  5.  Limited 
knowledge and implementation of the rights deriving from the citizenship and law of the 
Union. 

Description of the programme 

The  introduction  of  the  Rights,  Equality  and  Citizenship  Programme  was  meant  to 
address  all  these  challenges  with  the  general  objective  of  contributing  to  the 
strengthening of equality and the rights of persons, as enshrined in the Treaties, the EU 
Charter  of  Fundamental  Rights  and  in  international  human  rights  conventions  to  which 
the European Union has acceded.  
In  order  to  do  so,  the  Programme  addresses  fundamental  rights  in  nine  areas, 
corresponding to its specific objectives: 

  Specific objective 1: Promoting the effective implementation of the principle of non-
discrimination  on  grounds  of  sex,  racial  or  ethnic  origin,  religion  or  belief, 
disability,  age  or  sexual  orientation,  and  to  respect  the  principle  of  non-
discrimination on the grounds provided for in Article 21 of the Charter  

The  Programme  supports  projects  aiming 
towards  preventing  and  combatting 
discrimination on the grounds of sex, racial or ethnic origin, religion or belief, disability, 
age or sexual orientation. In particular, the promotion of Roma integration is a key topic 
of this specific objective. In this regard, the Rights, Equality and Citizenship Programme 
supports the National Roma Platforms and works in close cooperation with EU Member 
States and their National Roma Contact Points and with civil society organisations active 
on non-discrimination and Roma integration. The Programme finances also the activities 
of  the  European  Network  of  Equality  Bodies.  The  core  task  of  this  network  includes 
strengthening  the  cooperation  and  exchange  of  information  between  the  National 
Equality  Bodies  designated  by  the  Member  States  on  topics  related  to  non-
discrimination. 

  Specific  objective  2:  Prevent  and  combat  racism,  xenophobia,  homophobia  and 

other forms of intolerance  

Projects  financed  in  this  area  contribute  to  better  implementation  of  existing  EU 
legislation  in  the  Member  States  and  to  assisting  the  victims  of  hate  crime  and  hate 
speech.  In  particular,  the  funded  activities  support  the  protection  of  refugees  and 
migrants against the surge of populism, extremism and intolerance. 

  Specific objective 3: Promote the rights of persons with disabilities  

The  operational  aim  of  this  specific  objective  is  to  increase  awareness  and  better 
implementation of the rights of people with disabilities leading to a reduction of barriers, 
to  their  full  participation  in  society  and  enjoyment  of  their  rights.  This  is  mostly  done 
through operating grants supporting the activities of European level networks advocating 
for the rights of people with disabilities and through procurement activities focusing on 
data  collection,  training  and  awareness-raising  activities.  Under  the  2014  annual  work 

4 

 
 
 
 
programme,  the  European  Disability  Card8  was  launched  with  support  from  the 
Programme in order to facilitate cross-border travelling for persons with disabilities. 

  Specific objective 4: Promote equality women and men 

The Programme promotes projects to support Member States and relevant stakeholders in 
increasing gender balance in all spheres of life, especially in economic decision-making, 
and  promoting  equal  economic  independence  of  women  and  men.  Funding  under  this 
objective  also  aims  to  contribute  to  reducing  the  gender  gaps  in  pay,  earnings  and 
pensions.  

  Specific  objective  5:  Prevent  and  combat  violence  against  children,  young  people 

and women, and other groups at risk  

The Programme finances primarily projects which help to protect and support the most 
vulnerable victims of violence, such as women and children, as well as activities to raise 
awareness and prevent violence, engaging with non-governmental organisations working 
on  the  ground.  It  also  supports  the  treatment  of  perpetrators  of  violence.  Strengthening 
child  protection  systems  is  one  of  the  Commission's  priorities  and  the  Programme 
supports  the  rights  of  migrant  children  by  funding  international  organisations  working 
directly for and with these children.  

  Specific objective 6: Promote the rights of the child  

The rights of the child are promoted through projects focusing on child-friendly justice 
and on the protection of vulnerable children (like children leaving foster care systems and 
children in conflict with the law), by supporting, in particular, training activities of civil 
servants, non-governmental organisations and lawyers dealing with them. 

  Specific objective 7: Protect privacy and personal data  

With  regard  to  data  protection,  the  Programme  is  the  EU  main  funding  source.  The 
activities  carried  out  within  this  specific  objective  are  strongly  linked  with  the  Data 
Protection  Reform9,  adopted  in  2016.  Through  this  objective,  the  Programme  aims  at 
financing activities to support, in particular, the transposition and implementation of the 
new data protection legislation by the Member States and the training of data protection 
authorities and data protection officers. 

  Specific objective 8: EU citizenship  

The Programme finances projects, mostly awareness raising activities, aimed at fostering 
the inclusion of EU citizens in the civic and political life of the EU. Therefore, it supports 
projects  which  can  help  citizens  to  be  more  aware  of  their  rights  deriving  from  the 
citizenship  of  the  Union.  Information  campaigns  on  EU  citizenship  rights  are  financed 
through this specific objective and a particular focus is on their electoral rights in view of 
the 2019 European elections. 

  Specific objective 9: Consumers' or entrepreneurs' rights 

The  Programme  finances  also  projects  which  support  individuals  in  their  capacity  as 
consumers or entrepreneurs in the internal market in order to enforce their rights deriving 

8 For more info on the European Disability Card, see http://ec.europa.eu/social/main.jsp?catId=1139 . 
9 For more info on the Data Protection Reform, see https://ec.europa.eu/info/law/law-topic/data-protection_en.  

5 

 
                                                            
from  UE law.  EU  funds in  this  area  aim at contributing to increase the knowledge  and 
awareness of consumers' rights, especially in the digital market. 

Budget 

The total Programme's budget for the period 2014-2020 is EUR 439 473 000 (see Table 
1).  The  highest  commitment  rate  is  observed  in  2016,  while,  according  to  available 
sources, the total committed EU contribution is lower for the previous two years.  

Table 1: Total annual amount planned and committed in 2014-2016 

Budget year 

Annual amount 
planned (in euro) 

Annual amount 
committed (in euro) 

Ratio 
committed/planned 

2014 

2015 

2016 

54 158 000.00  

49 719 588.56  

56 269 000.00  

48 621 670.69 

58 852 000.00  

55 854 156.79  

Total 2014-2016 

169 279 000.00  

154 195 416.04 

91.80 % 

86.41 % 

94.91 % 

91.09% 

Sources: Annual monitoring reports (reports on the implementation of the annual work programmes and Project Database for 2016) 

Procurement  activities  represent  about  23%  of  the  budget,  though  the  distribution 
between  the  specific  objectives  is  not  homogeneous.  Indeed,  procurement  activities 
represent the total planned spending of the specific objective on the rights of consumers, 
65% of the specific objective on citizenship and less than 5% of the total planned budget 
on  the  specific  objective  on  the  prevention  of  racism.  All  other  specific  objectives 
allocate to procurement activities between 11% and 55% of the planed budget. 

In order to allow the achievement of its specific objectives, the Programme identifies a 
wide  range  of  activities  to  be  implemented.  In  particular,  according  to  Article  5  of  the 
Regulation  establishing  the  Rights,  Equality  and  Citizenship  Programme  for  the  period 
2014 to 2020, the Programme can finance: 

  Analytical  activities,  in  particular:  collection  of  data  and  statistics;  development  of 
common  methodologies  and,  where  appropriate,  indicators  or  benchmarks;  studies, 
researches, analyses,  surveys  and  evaluations; elaboration and  publication  of  guides, 
reports  and  educational  material;  workshops,  seminars,  experts'  meetings  and 
conferences; 

  Mutual learning,  cooperation, awareness  raising and dissemination activities, in 
particular:  identification  of,  and  exchanges  concerning,  good  practices,  innovative 
approaches  and  experiences;  organisation  of  peer  reviews  and  mutual  learning; 
organisation  of  conferences,  seminars,  media  campaigns,  including  in  the  online 
media, information campaigns, including institutional communication on the political 
priorities  of  the  Union  as  far  as  they  relate  to  the  objectives  of  the  Programme; 
compilation  and  publication  of  materials  to  disseminate  information  about  the 
Programme  and  its  results;  development,  operation  and  maintenance  of  systems  and 
tools using information and communication technologies; 

  Training  activities,  for  instance:  staff  exchanges,  workshops,  seminars,  train-the-
trainer events and the development of online training tools or other training modules; 

6 

 
 
 
 
 
 
  Actions  to  support  the  main  actors  whose  activities  contribute 

the 
implementation  of  the  objectives  of  the  Programme,  such  as  support  for  Non-
governmental  organisations  in  the  implementation  of  actions  with  European  added 
value;  support  for  key  European  actors,  European-level  networks  and  harmonised 
services  of  social  value;  support  for  Member  States  in  the  implementation  of  Union 
law  and  policies  and  support  for  networking  activities  at  European  level  among 
specialised bodies and entities  as well as  national, regional and local authorities and 
non-governmental  organisations,  including  support  by  way  of  action  grants  or 
operating grants10. 

to 

The  stakeholders  eligible  for  support  are  public  or  private  organisations  (usually  non-
profit-oriented), duly established in one of the countries participating in the Programme, 
or  international  organisations  (such  as  the  UN  High  Commissioner  for  Refugees, 
UNICEF and the International Organization for Migration).  

In terms of target groups, intended as the groups that can benefit directly or indirectly 
from  the  Programme,  public  authorities,  non-governmental  organisations  and  other 
research entities, concerned with pursuing its objectives, are included. Target groups are 
also  all  citizens,  since  its  objectives  and  initiatives  aim  at  promoting,  protecting  and 
effectively implementing equality and the rights of persons, in particular those subjected 
to  discrimination,  intolerance  or  violence,  including  migrants  and  minorities.  Specific 
objectives  focus  on  support  for  people  with  disabilities,  Roma,  women,  EU  citizens, 
consumers and entrepreneurs and children11. 

The Rights, Equality and Citizenship Programme has three main funding mechanisms: 
action grants, operating grants and procurement actions12.  

  Action  grants  are  addressed  mainly  to  civil  society  organizations,  Member  States 

authorities and universities;  

  Operating grants ("support to Networks") fund mainly European networks active in 
the  following  areas:  non-discrimination;  racism,  xenophobia,  homophobia  or  other 
forms  of  intolerance;  disabilities;  equality  between  women  and  men;  preventing  and 
combating all forms of violence and promoting the rights of the child. 

  Procurement actions ("Commission initiatives") fund mostly conferences, seminars, 

studies, surveys, awareness-raising activities, but also specific IT projects.  

The Programme intervention logic is outlined in the Figure 1. 

10 To see the types of activities funded by the Programme in 2014, 2015 and 2016, see the Mid-term evaluation of the Rights, Equality 
and Citizenship Programme 2014-2020, Ernst & Young Financial-Business Advisors, Final Report, April 2018, p. 64-65. 
11 For more information, see below p. 19 and see also section 6.2 on "Target groups and beneficiaries" in the Interim report, p. 71 et 
seq. 
12  For  more  information  on  the  number of  action  grants  and  operating  grants  awarded  per year and  on  the  number of  procurement 
activities, see section 6 "Implementation state of play" in the Interim report, in particular p. 61-63. 

7 

 
                                                            
Figure 1: Intervention Logic of the Rights Equality and Citizenship Programme 2014-2020 

8 

 
 
 
External factors could influence the Programme's outcomes and make challenging to 
identify changes as a result of its intervention. In particular, the economic crisis has led to 
a lack of interest on the part of public sector stakeholders (more interested in economic 
issues)  and,  therefore,  to  a  related  reduction  of  national  funding  for  social  and  human 
rights  issues.  This  is  more  accentuated  in  certain  Member  States  where  the  political 
climate is less supportive13.  

Baseline and points of comparison  

The  interim  evaluation  assessed  the  Rights,  Equality  and  Citizenship  Programme's 
performance starting from the situation described in the 2011 Impact Assessment14 and in 
the  ex-post  evaluation  of  2007-2013,  concerning  the  three  predecessor  programmes 
(Daphne III, Fundamental Rights and Citizenship and Progress), carried out in 201515.  
These documents contain the baselines and main points of comparison for measuring the 
Programme’s achievements during the reporting period.  
The  baseline  analysis  showed  that,  if  no  changes  had  been  made  between  the  2007-13 
and 2014-20 programming period, the three previous programmes would have continued 
to  be  successfully  implemented,  but  at  a  reduced  potential,  in  particular  due  to:  1.  The 
lack of flexibility in the funding instruments which did not reflect the pace of change and 
reform in this policy area; 2. The fragmentation of funding that reduced the capacity of 
the  programmes  to  deliver  results  in  horizontal  and  cross-cutting  issues;  and  3.  The 
elevated  number  of  different  funding  instruments  which  increased  the  administrative 
burden. 
Given these difficulties, the Impact Assessment suggested the option to consolidate the 
three programmes in the current Rights, Equality and Citizenship Programme. 
This  possibility  was  assessed 
relevance/scope,  effectiveness, 
terms  of 
complementarity,  European  added  value,  efficiency  and  potential  for  simplification. 
Since the option to proceed with only one programme was undertaken, the findings of the 
Impact  Assessment,  together  with  the  recommendations  of  the  ex-post  evaluations  of 
2007-2013,  provide  a  useful  baseline  to  verify  whether  the  Rights,  Equality  and 
Citizenship Programme achieved its expected goals. 

its 

in 

Follow-up on the ex post evaluation report of the period 2007-2013 

All  the  ex-post  evaluation  reports  concerning  the  previous  funding  programmes 
confirmed  their  overall  effectiveness  and  highlighted  that  their  specific  objectives  and 
priorities  were  largely  specific,  attainable  and  realistic,  but  not  always  measurable. 
Indicators,  allowing  the  measuring  of  progress  toward  the  attainment  of  the  specific 
objectives, were not included.  

Moreover, the reports pointed out the need to make improvements on other issues which 
could have increased the impact, added value, effectiveness and efficiency in delivering 
result of the programmes. 

In  particular,  the  reports  have  identified  the  following  needs  common  to  the  three 
programmes:  

1.  Better  definition  of  the  priorities  in  order  to  ensure  that  they  can  be  adequately 

13 Interim report, ibid. 
14 SEC(2011) 1364 Final, ibid. 
15 European Commission (2015), Ex-post evaluation of five programmes implemented under the 2007-2013 financial perspective  – 
Final Report, ibid.  

9 

 
 
 
                                                            
achieved within an earmarked budget;  

2.  Realistic  assessments  of  project  risks  and  better  risk  mitigation  strategies 
throughout the projects duration;  

3. Increase focus on assessment of impacts at all levels and not merely on outputs, as 
regards monitoring and evaluation;  

4.  Explore  ways  of  enhancing  the  uptake  of  project  outputs,  results  and  best 
practices by other organisations, including in other Member States;  

5. More dissemination/use of results and outputs of the funded activities.  

The  ex-post  evaluation  reports  of  the  period  2007-2013  have  also  highlighted  the 
following  problems: 1. The  dilution of  funds amongst  many small-scale projects with 
limited  impact  and  EU  dimension;  2.  No  balanced  geographical  spread  among  the 
organisations which receive funding; 3. The complex and bureaucratic procedures for 
the applicants; 4. The high administrative burden on the Commission and an increase 
of  the  length  of  procedures  due  to  the  multiplication  of  procedures  for  the  different 
programmes.  

These issues were taken into consideration and were either integrated into the Regulation 
establishing  the  current  Rights,  Equality  and  Citizenship  Programme  and  in  the  annual 
work programmes or translated into technical implementation measures and included in 
the Guides for applicants to projects calls. 
In particular, the merger of the three funding programmes has provided a significant 
positive impact on the identified problems of scope, effectiveness, fragmentation and 
efficiency  (see  further  details  in  section  5  “Analysis  and  answer  to  the  evaluation 
questions”).   

3. 

IMPLEMENTATION / STATE OF PLAY 

Programme management 

The  Rights,  Equality  and  Citizenship  Programme  is  implemented  via  direct  centralised 
management by the European Commission. This was meant to ensure a close relationship 
between  the  Programme  management  and  EU  policy-making  and  to  contribute  to  the 
alignment of budget implementation with EU policy priorities in line with the "budget for 
result"  approach.  This  management  mode  allows  the  Commission  to  tailor  funded 
activities to policy priorities and policy needs and to target directly the relevant groups of 
stakeholders. According to the evaluation carried out, it also allows a close contact with 
the programmes' beneficiaries and better knowledge of the needs on the ground. Indeed, 
most  of  the  stakeholders  interviewed  referred  to  improvements  in  the  management 
systems,  as  well  as  good  communication  with  the  Commission  and  contact  persons.  In 
this respect, several stakeholders agreed that the policy staff in the current Programme is 
much more engaged with civil society16. 

State of play  

The  purpose  of  this  section  is  to  present  the  state  of  play  of  the  Programme’s 
implementation  and  its  key  initiatives  in  the  period  2014-2017.  It  will  also  provide  the 

16  Although,  the  beneficiaries  interviewed  were  not  specifically  asked  to  describe  their  experience  with  other  kinds  of  Programme 
management. 

10 

 
 
 
 
 
 
                                                            
qualitative and quantitative results achieved in the same period, in general terms and for 
each specific objective. 
In  order  to  measure  the  progress  from  the  baseline  situation,  a  series  of  Programme 
indicators (see below in this section17) have been devised for the current Programme. The 
introduction  of  a  system  of  indicators  has  proved  to  be  adequate  for  measuring  the 
achievements of the Programme since the selected indicators show if the targets are close 
to being achieved or have already been achieved and this contributes to better focus the 
Programme's outcomes.  

Concerning the general objective of the Programme, the 2020 target appears difficult 
to be achieved for three of the five selected indicators18.  
In particular, the increase of the employment rate of women and people with disabilities 
is  hampered  by  an  external  factor,  i.e.  the  European  financial  recession.  Also  the 
percentage of women among non-executive directors on boards of listed companies is not 
reaching  the  2020  target.  However,  the  Rights,  Equality  and  Citizenship  Programme  is 
contributing to the achievement of these objectives through projects aimed at promoting 
the exchange  of good practices, developing tools to  collect data on women and  men in 
decision  making  and  lobbying  at  the  EU  and  national  level  for  promoting  the  labour 
market inclusion of women. 
Concerning the employment rate of people with disabilities, the target is  difficult to  be 
achieved since the participation in employment of people with disabilities is lower than 
people  without  disabilities  and  the  recession  has  only  worsened  it.  The  Programme  is, 
however, contributing to the achievement of the target by funding projects empowering 
organisations  representing  people  with  disabilities  towards  an  effective  lobbying  at  the 
EU  and  national  level  for  designing  policies  favouring  the  labour  market  inclusion  of 
people with disabilities. 
Instead,  the  gender  pay  gap  is  showing  decreasing  trends  and  the  target  is  likely  to  be 
reached. Projects financed by the Programme promote wage transparency and awareness-
raising activities that can enable employees and employers to reduce the gender pay gap. 
Finally,  the  percentage  of  Europeans  who  consider  themselves  as  “well  or  “very  well” 
informed of  the rights  they enjoy as  citizens  of the Union is  showing increasing trends 
and the target is likely to be reached if awareness raising activities are adequately tailored 
to different targets (i.e. primary school students, migrants, rural communities) (see Table 
2  below).  However,  it's  challenging  to  gauge  the  specific  contribution  of  the  Rights, 
Equality and Citizenship Programme to such results since this topic is addressed by other 
EU level initiatives (i.e. Erasmus+)19.   

Table 2: Result indicators, targets and actual values 

Result indicators 

Baseline 2013 

2015-2017 

Targets 2020 

1.  Female  employment  rate  20-64 
age group 

62,4 % (2012) 

66,6% (2017) 

75%  (for  both  women 
and  men):  Europe  2020 
headline target; 71% for 
women 

2.  Employment  rate  of  people  with 
disabilities 

(2013) 

48,5% 
EU-Statistics 
on  Income  and 
Living 

47,4% (2015) 

55 % 

17 In particular, see below pages 15-16. 
18 See Interim report, ibid. 
19 There are also other EU level initiatives which contribute to such result, such as the Erasmus+ Programme. 

11 

 
 
 
                                                            
Conditions 

3. The gender pay gap 

16,8% (2013) 

16,3% (2015) 

4. The percentage of women among 
non-executive directors on boards of 
listed companies 

16,2 % (2011) 

25,7 % (2017) 

14 % 

40 % 

5.  Percentage  of  Europeans  who 
consider  themselves  as  “well  or 
“very  well”  informed  of  the  rights 
they enjoy as citizens of the Union 

(2010, 

32% 
Source: 
Eurobarometer) 

42% (2015) 

51 % 

Concerning  the  specific  objectives  of  the  Programme,  the  following  indicators  have 
been used to evaluate its performance: 

  Specific objective 1: To promote the effective implementation of the principle of 
non-discrimination  on  the  grounds  of  sex,  racial  or  ethnic  origin,  religion  or 
belief,  disability,  age  or  sexual  orientation,  and  to  respect  the  principle  of  non-
discrimination on the grounds provided for in Article 21 of the Charter 

The  Programme's  performance  concerning  this  specific  objective  is  measured  through 
two main indicators: the number of Europeans aware of their rights if they fall victims of 
discrimination  and  the  number  of  Member  States  that  set  up  structural  coordination 
mechanisms  with  all  stakeholders,  including  Roma,  on  the  implementation  of  the 
National Roma Integration Strategies20.  
For both of these indicators, it seems likely that the stated goals for 2020 will be reached 
(see  Table  3).  Concerning  the  first  indicator,  the  target  will  be  attained  faster  if  the 
increased  awareness  of  the  rights  of  victims  is  combined  with  training  activities  of 
stakeholders in direct contact with them (i.e. social assistants, police officers, lawyers). 
Concerning the  second indicator, the  Rights,  Equality and Citizenship Programme  has 
launched ad hoc calls on the implementation of the National Roma Integration Strategies. 
Therefore,  the  achievement  of  the  target  depends  largely  on  the  investments  of  the 
Programme in this regard.   

Table 3: Result indicators, targets and actual values 

Result indicators 

Baseline 2013 

2015-2017 

Targets 2020 

1.  The  number  of  Europeans  aware  of  their 
rights if they fall victims of discrimination 

(2012, 

37% 
Source: 
Eurobarometer) 

45% (2015) 

70% 

2.  The  number  of  Member  States  that  set  up 
structural  coordination  mechanisms  with  all 
stakeholders, 
the 
implementation  of 
the  National  Roma 
Integration Strategies  

including  Roma,  on 

No 
State (2013) 

Member 

21 
States (2016) 

Member 

27 Member States 

  Specific objective 2: To prevent and combat racism, xenophobia, homophobia and 

other forms of intolerance 

20  The  European  Commission  invited  EU  Member  States  to  adopt  national  strategies  aim  at  improving  the  economic  and  social 
situation  of  Roma  and  reducing  existing  disparities  with  the  rest  of  the  population  by  2020.  For  more  info,  see  https://eur-
lex.europa.eu/legal-content/EN/TXT/?uri=LEGISSUM%3Aem0049 . 

12 

 
 
 
 
 
 
                                                            
For  this  specific  objective,  no  specific  indicator  was  set  up,  but  the  same  indicator  on 
non-discrimination could be used. 

  Specific objective 3: To promote and protect the rights of people with disabilities 

Also for this specific objective, no specific indicator was set up. However, the operating 
grants funded under this specific objective also cover the promotion of social and labour 
market  inclusion  of  people  with  disabilities.  The  Programme's  performance  concerning 
this  specific  objective  is,  therefore,  measured  through  the  indicator  on  the  employment 
rate of people with disabilities (see above Table 2, indicator 2). 

  Specific objective 4: To promote equality between women and men and to enhance 

gender mainstreaming 

No specific indicator was set up also for this specific objective. However, the indicators 
used  for  the  general  objective  are  adequate  for  measuring  the  achievements  under  this 
specific objective. As a matter of fact, these indicators capture the expected results of the 
projects funded by the Programme that aim at promoting the increased gender balance in 
economic decision-making positions at all levels (see above Table 2).  

  Specific objective 5: To prevent and combat all forms of violence against children, 
young  people  and  women,  as  well  as  violence  against  other  groups  at  risk,  in 
particular groups at risk of violence in close relationships, and to protect victims of 
such violence 

The  Programme's  performance  concerning  this  specific  objective  is  measured  through 
one  indicator:  the  percentage  of  people  that  consider  that  domestic  violence  against 
women is unacceptable. Until 2015 there were no progresses in this regard, but it is likely 
that  the  awareness  raising  actions  of  the  projects  funded  by  the  Programme,  which 
represent  a  significant  portion  of  the  activities  funded  in  this  area,  will  lead  to  getting 
closer to the target value (see  Table 4). However, this indicator is rather extensive, and 
many external factors such as economic growth, national policies (including the Istanbul 
Convention ratification) influence its value.  

Table 4: Result indicators, targets and actual values 

Result indicator 

1.  Percentage  of  people  that  consider  that 
is 
domestic 
unacceptable  

against  women 

violence 

Baseline 2013 

84 % (2010)  

Source: 
Eurobarometer 

2017 

96%  

Targets 2020 

100% 

  Specific objective 6: To promote and protect the rights of the child 

The Programme's performance concerning this specific objective was, at first, measured 
through  one  indicator:  the  percentage  of  children  aware  that  they  enjoy  specific  rights. 
However, it soon emerged that this specific indicator is not fully adequate to capture the 
achievements  of  the  Programme  within  this  specific  objective.  The  Commission  is 
currently developing a more suitable indicator for measuring and reporting the progress 
achieved by the Programme on the promotion and protection of the rights of the child. 

  Specific  objective  7:  To  contribute  to  ensuring  the  highest  level  of  protection  of 

privacy and personal data 

13 

 
 
 
The  Programme's  performance  concerning  this  specific  objective  is  measured  through 
one  indicator:  the  number  of  complaints  received  by  data  protection  authorities  from 
individuals  relating  to  data  protection.  The  contribution  of  the  Rights,  Equality  and 
Citizenship  Programme  to  this  specific  objective  is  expressed  by  the  capacity  of  the 
Programme to raise awareness among public authorities and companies about the content 
and technical requirements of the 2016 Data Protection Reform. However, no data were 
available so far (see Table 5).  

Table 5: Result indicators, targets and actual values 

Result indicator 

Baseline 2013 

2016 

Targets 2020 

1.  Number  of  complaints  received  by 
data 
from 
individuals relating to data protection  

authorities 

protection 

NA 

60 000 

68  569  (2012, 
source: 
Directorate-
General 
Justice 
Consumers) 

for 
and 

  Specific  objective  8:  To  promote  and  enhance  the  exercise  of  rights deriving  from 

the citizenship of the Union 

The  Programme's  performance  concerning  this  specific  objective  is  measured  through 
three indicators: 1. The share of the population that knows the meaning of “Citizenship of 
the Union”; 2. The share of the population considering themselves as well or very well 
informed  of  the  rights  they  enjoy  as  citizens  of  the  Union;  3.  The  percentage  of 
population aware of the right to vote and to stand as candidate in European election in the 
Member State of residence, without having the nationality of that Member State.  
All the stated goals of the three indicators are likely to be achieved in 2020 (see Table 6). 
Concerning  the  first  indicator,  the  target  was  almost  achieved  in  2015  and  it  is  likely 
that  has  been  achieved  meanwhile.  However,  it  is  difficult  to  gauge  the  specific 
contribution of the Rights, Equality and Citizenship Programme to such results since this 
topic  is  addressed  by  other  EU  level  initiatives  (i.e.  Erasmus+).  This  last  consideration 
holds also for the second indicator. 
Concerning the third indicator, despite the drop in 2015 compared to the baseline, the 
target  may  still  be  achieved  by  2020  since  projects  and  Commission  awareness  raising 
campaigns  financed  under  this  specific  objective  aimed  at  encouraging  the  political 
participation  of  mobile  EU  citizens  and,  therefore,  to  increase  their  participation  to 
European elections. 

Table 6: Result indicators, targets and actual values 

Result indicators 

1.  Share  of  the  population  that  knows  the 
meaning of “Citizenship of the Union” 

2.  Share  of  the  population  considering 
themselves as well or very well informed of 
the  rights  they  enjoy  as  citizens  of  the 
Union 

3.  Awareness  of  the  right  to  vote  and  to 
stand  as  candidate  in  European  election  in 
the  Member  State  of  residence,  without 
having the nationality of that Member State 

Baseline 2013 

(2013, 

46% 
Source: 
Eurobarometer) 

(2013, 

36  % 
Source: 
Eurobarometer) 

Targets 2020 

52% (2015) 

53% 

42% (2015) 

51% 

72% (2013, source: 
Eurobarometer) 

67% (2015) 

80% 

14 

 
 
 
 
 
 
  Specific objective 9: To enable individuals in their capacity as consumers or entrepreneurs 
to  fully  benefit  from  the  European  Single  Market  by  removing  remaining  legal  obstacles 
and  ensuring  enforcement  of  their  rights  deriving  from  Union  law,  having  regard  to  the 
projects funded under the Consumer Programme 

The  Programme's  performance  concerning  this  specific  objective  is  measured  through 
four indicators: 1. The perception of consumers of being protected; 2. The percentage of 
consumers  who  are  aware  of  their  right  to  keep  the  unordered  product  sent  to  them 
together  with  an  invoice;  3.  The  percentage  of  retailers  in  the  EU  who  know  that 
including invoices with marketing material (for unordered products) is prohibited; 4. The 
level of consumer confidence in cross-border shopping, as measured by the percentage of 
consumers who have at least equal level of confidence in sellers from their own country 
as from another EU country.  
The  targets  set  for  the  first  and  the  fourth  indicators  have  already  been  achieved. 
However,  the  Programme,  through  the  funded  awareness  raising  activities,  can  surely 
contribute  to  further  increasing  these  achievements.  The  target  of  the  second  indicator 
may not be achieved, and of the third indicator is likely to be achieved (see Table 7). 
Concerning,  in  particular,  the  second  indicator,  the  Programme  is  contributing  to 
achieving  the  target  by  implementing  training  and  awareness  raising  actions  targeted 
directly  at  companies  selling  products  online  in  a  way  that  they  provide  clearer 
information to their customers. 

Table 7: Result indicators, targets and actual values 

Result indicators 

Baseline 2013 

2015-2016 

Targets 
2020 

1. The perception of consumers of being protected  

2. Percentage of consumers who are aware of their 
right  to  keep  the  unordered  product  sent  to  them 
together with an invoice 

3. Percentage of retailers in the EU who know that 
including  invoices  with  marketing  material  (for 
unordered products) is prohibited 

4.  Level  of  consumer  confidence  in  cross-border 
the  percentage  of 
shopping,  as  measured  by 
consumers  who  have  at  least  equal  level  of 
confidence  in  sellers  from  their  own  country  as 
from another EU country 

64  %  (2011,  source: 
Consumer 
Conditions 
Scoreboard) 

36  %  (2011,  source: 
Consumer 
Conditions 
Scoreboard) 

51  %  (2011,  source: 
Consumer 
Conditions 
Scoreboard) 

34  %  (2012,  source: 
Eurobarometer) 

76% (2015) 

75% 

34% (2015) 

39% 

56% (2015) 

60% 

58% (2016) 

50% 

Moreover, a series of Programme-related indicators are also mentioned in the legal basis 
of the Rights, Equality and Citizenship Programme21.  
They  provide  a  basis  for  monitoring  and  evaluating  the  extent  to  which  each  of  the 
Programme's  specific  objectives  has  been  achieved.  These  are:  1.  The  number  and 
percentage  of  persons  in  a  target  group  reached  by  the  awareness-raising  activities 

21 Art. 14 of the Regulation (EU) 1381/2013 of 17 December 2013 establishing a Rights, Equality and Citizenship Programme for the 
period 2014 to 2020. 

15 

 
 
 
 
 
                                                            
funded  by  the  Programme;  2.  The  number  of  stakeholders  participating  in,  training 
activities, exchanges, study visits, workshops and seminars funded by the Programme; 3. 
The  improvement  in  the  level  of  knowledge  of  Union  law  and  policies  and,  where 
applicable,  of  rights,  values  and  principles  underpinning  the  Union,  in  the  groups 
participating  in  activities  funded  by  the  Programme  compared  to  with  the  entire  target 
group;  4.  The  number  of  cases,  activities  and  outputs  of  cross-border  cooperation;  5. 
Participants'  assessment  of  the  activities  in  which  they  participated  and  of  their 
(expected)  sustainability;  6.  The  geographical  coverage  of  the  activities  funded  by  the 
Programme; 7. The number of applications and grants related to each specific objective; 
8. The level of funding requested by applicants and granted in relation to each specific 
objective. 

Programme  indicators  1,  2  and  4  relate  to  the  outputs  and  outcomes  of  funded 
activities and can be calculated through the final reports of the projects. At the moment, 
there  are  not  sufficient  available  data  to  come  to  consolidated  conclusion  on  these 
indicators;  Programme  indicators  3  and  5  relate  to  the  perception  of  target  groups 
concerning  the  activities  they  participated  in  and  level  of  knowledge  they  acquired. 
These  indicators  are  currently  not  measured.  Programme  indicator  6  speaks  to  the 
equity in distribution of activities funded by the Programme amongst beneficiaries in the 
participating  Member  States,  while  indicator  7  and  indicator  8  are  interrelated  since 
they  speak  of  the  relationship  between  the  demand  for  grants  and  for  funding  and  the 
extent to which such demand is being satisfied.  
The assessment of these indicators is provided in the section on the Implementation state 
of  play  and  in  the  section  dedicated  to  answering  evaluation  questions  on  the 
effectiveness  and  equity  of  the  Programme.  However,  in  general,  there  is  no  baseline 
available  for  these  indicators,  as  they  relate  directly  to  the  specific  actions  of  the 
Programme, and there are also no defined targets, except for the geographical coverage.  

Other data, collected during the Interim evaluation, show the current state of play 
of the Programme’s implementation and its key initiatives in the period 2014-2017. 

In  relation  to  the  type  of  partnership  developed  under  the  Programme,  the  data 
collected in the interim evaluation suggest that, on average, the number of participating 
organisations  per  project,  between  2016  and  2017,  is  equal  to  six-seven  for  the  action 
grants  (including  the  coordinator).  Within  the  Programme  network  there  are  236 
organisations  connected  out  of  440  organisations  reported  in  the  database,  i.e.  204  are 
isolated, meaning that one organisation out of two, within the Programme collaboration 
network,  is  isolated22.  This  shows  that  the  Programme  network  is  characterised  by  the 
presence of a set of actors that has no connection at all (isolated) or belong to very small 
groups and tend to participate in a small number of projects.  
Transnational  partnerships,  thanks  to  their  networking  and  interconnectedness  between, 
are essential contributors to the effectiveness of the Programme. Indeed, according to the 
majority of respondents, the partnerships developed under the Programme have improved 
their organisational structures in terms of project implementation, sustainability of results 
and  overall  capacities.  A  smaller  number  of  beneficiaries  reported  also  that  the 
partnerships had a positive impact on their fund-raising capacities.  
The evaluation of the partnerships formed has shown also an appropriate representation 
of beneficiaries by type of organisation, but an uneven geographical composition of the 
participating organisations.  

22 Within the Justice Programme only one organisation out of five is isolated. See Interim report, ibid.  

16 

 
 
 
 
                                                            
As regards the  type  of beneficiaries  of the Programme,  the available data showed that 
39% of the total beneficiaries are non-profit organisations, while only 5% are private for-
profit  entities.  As  for  the  Higher  or  Secondary  Education  entities  category,  92%  are 
universities and 17% of them are projects coordinators. More than half of the universities 
involved  as  coordinators  or  as  partners  in  approved  projects  come  from  only  three 
Member States (Italy, Belgium and Spain). The category of Public Bodies is constituted 
for 14% by Belgian organisations and to the same extent by Spanish organisations, 12% 
by the Dutch, 10% by French and to a lesser extent by organisations from other Member 
States.  The  research  organisations,  the  private  for-profit  entities  and  no-profit 
organisations  are  distributed  fairly  evenly  throughout  Europe,  even  if,  in  the  last 
category, there is still a greater presence of two Member States (Belgium and Italy). 
According to the evaluation, about 20% of beneficiaries participated in multiple projects 
within  the  Programme  and  the  organisations  involved  in  multiple  projects  tend  to  be 
based in Member States with relatively low participation rates, such as Sweden, Latvia, 
Cyprus  and  Denmark.  This  distribution  reflects  the  fact  that  the  Rights,  Equality  and 
Citizenship  network  is  more  “periphery-based”  compared,  for  example,  to  the  Justice 
Programme.  
Taking into account the involvement of partner organisations, it appears clearly that the 
bulk of participants to the Programme come only from few Member States and that 
a balanced geographical spread has still to be achieved (as was the case with the 
predecessor programmes) (see Figure 2). 

Figure 2: Number of applying organisations per country (main applicant or partners) 

900

800

700

600

500

400

300

200

100

0

i

n
a
p
S

e
c
e
e
r
G

y
l
a
t
I

i

m
o
d
g
n
K
d
e
t
i
n
U

i

m
u
g
e
B

l

a
i
r
a
g
u
B

l

a
i
t
a
o
r
C

y
r
a
g
n
u
H

i

a
n
a
m
o
R

e
c
n
a
r
F

y
n
a
m
r
e
G

d
n
a
o
P

l

s
u
r
p
y
C

l

a
g
u
t
r
o
P

a
i
r
t
s
u
A

i

a
n
e
v
o
l
S

d
n
a
e
r
I

l

i

a
n
a
u
h
t
i
L

a
i
v
t
a
L

s
d
n
a
l
r
e
h
t
e
N

c
i
l

b
u
p
e
R
h
c
e
z
C

d
n
a
n
i
F

l

i

a
n
o
t
s
E

n
e
d
e
w
S

a
t
l
a
M

k
r
a
m
n
e
D

a
i
k
a
v
o
l
S

d
n
a
e
c
I

l

y
a
w
r
o
n

g
r
u
o
b
m
e
x
u
L

i

n
e
t
s
n
e
t
h
c
e
i
L

2014

2015

2016

Source: Sygma and Priamos data 

Consequently,  this  affected  the  allocation  of  funds.  For  example,  the  largest  share  of 
available  resources  for  action  grants  is  attributed  to  Italian  organisations,  followed  by 
organisations  in  the  United  Kingdom,  Spain,  Belgium  and  Greece.  Instead,  the  vast 

17 

 
 
 
 
 
majority of resources available for yearly work programmes of funded operating grants is 
distributed to organisations based in Belgium23.   

With regard to the kind of projects/activities funded, in the period covered by the 2014-
2016  Annual  Work  Programmes,  52  calls  for  proposals  were  launched.  Most  calls 
covered  the  specific  objectives  “prevent  and  combat  all  forms  of  violence  against 
the  effective 
children,  young  people  and  women”  (15  calls)  and  “promote 
implementation of the principle of non-discrimination” (10 calls). This distribution is in 
line with what was programmed in the annual work programmes. Under these calls, 352 
projects have been funded. 

The two specific objectives with the highest number of awarded grants are the ones on 
promoting  non-discrimination  and  preventing  all  forms  of  violence,  which  cover  more 
than half of total grants approved (26 % of the Programme's committed budget). Through 
them,  the  Programme  mainly  financed  grants  focused  on  training  and  mutual  learning 
activities. In relation to the implementation of procurements (which represent about the 
23%  of  the  budget),  the  Programme  finances  activities  focused  mostly  on  the  specific 
objective on the rights of people with disabilities, followed by the specific objective on 
non-discrimination.  Most  funded  activities  concerned  the  organisation  of  conferences, 
workshops, the provision of analytical activities and the support to expert networks. 
On average, by looking at the awarding rate of calls for proposals in 2014 and 201524, 
this  ranges  between  7%  and  27%  for  almost  all  the  specific  objectives,  which  is  quite 
low. Only under the specific objective on disability, almost all the applications submitted 
were  awarded  a  grant.  The  second highest  awarding rate,  almost  58  %,  is  observed  for 
2014  in  relation  to  the  specific  objective  on  gender  equality.  As  with  the  case  of  the 
Justice Programme, a drop in the grant applications received (coinciding with the rollout 
of the Participant Portal25) increased the awarding rate of calls for proposals.  

The  specific  objectives  that  attracted  the  highest  number  of  applications  are  the  ones 
related  to  the  promotion  of  non-discrimination  and  prevention  of  violence  that  are, 
therefore, amongst the most competitive ones.  
As shown in Figure 3 below, the most covered target groups in the project proposals 
are young people (including children), women and students.  

23 This is due to the fact that operating grants resources are made available to organisations representing European networks, which 
tend to be headquartered in Belgium (thanks to the proximity with other EU institutions ). 
24 The awarding rate cannot be calculated for 2016 on the basis of the information available. 
25 The Participant Portal is an electronic platform used to manage the applications received for calls published in the framework of the 
Rights,  Equality  and  Citizenship  Programme.  According  to  the  evaluation  results,  the  new  Portal  could  be  further  simplified,  in 
particular in relation to the administrative documentation to be provided, in order to encourage the participation from smaller or less 
experienced organisations. 

18 

 
 
 
                                                            
Figure 3: Text mining of target groups of the Rights, Equality and Citizenship Programme, by 
specific objective 

Source:  Text  mining  analysis  conducted  on  256  application  forms;  RCHI  (rights  of  the  child),  RDAP  (To  prevent  and  combat  all 
forms of violence against children, young people and women), RDIS (non-discrimination), RGEN (gender equality), RRAC (to prevent 
and combat racism), RDIB (rights of the people with disabilities) 

The most covered categories to which the Programme’s activities are addressed are 
children (above all missing children and child victims of violence), (young) women and 
law-related professionals (in particular lawyers). 

Short description of methodology 

4.  METHOD 

The  evaluation  process  was  supported  by  an  external  evaluator  (Ernst  &  Young 
Financial-Business Advisors)26, under Commission's responsibility. The criteria used for 
the  evaluation  include:  relevance,  effectiveness,  efficiency,  coherence,  equity,  scope  of 
simplification of the Programme and EU added value (see Table 8).  

Table 8: Evaluation criteria and corresponding evaluation questions  

Evaluation 

criteria 

Evaluation questions 

1.  EQ1. To what extent have the objectives been achieved considering the 

set indicators?  

i)  What progress has been achieved so far by the Programme in meeting 

ii) 

indicators listed in Art 14.2 of the Regulation? 
Is  the  Programme  meeting  their  respective  general  objective  at  the 
Programme level? 

2.  EQ2.  How  adequate  were  the  actions  funded  under  the  Programme  to 

the objectives of the Programme? 

Effectiveness 

3.  EQ3. What factors influenced the achievements observed?  

i)  Have any specific factors favoured or deterred the achievement of the 

specific objectives of the Programme?  

ii)  Have  any  unintended  effects  influenced  the  effectiveness  of  the 

Programme? 

4.  EQ4:  How  have  the  eligibility  criteria  influenced  the  formation  of 

partnerships and the scope of proposals? 
to 

5.  EQ5.  Compared 

financing 
Programmes, how did the Programme perform in terms of better policy 

the  2007-2013  six  predecessor 

26 Interim evaluation of the Rights Equality and Citizenship Programme 2014-2020, Final Report, April 2018. 

19 

 
 
 
                                                            
Evaluation 

criteria 

Evaluation questions 

targeting, and targeting of the right groups of beneficiaries, Programme 
management and economies of scale?  

i)  How  has  the  consolidation  of  predecessor  programmes  influenced  the 
management  of  the  Programme  in  terms  reduction  of  administrative 
burden and the achievement of economies of scale?  

ii)  Has  the  consolidation  of  predecessor  programmes  resulted  in  better 
policy  targeting,  in  terms  of  a  reduction  of  dispersion/proliferation  of 
calls with similar or overlapping objectives?  

iii)  Have  funded  beneficiaries  been  able  to  enhance  their  capacity  to 

support relevant target groups? 

iv)  Have  any  unintended  effects  influenced  the  effectiveness  of  the 

Programme? 

6.  EQ6: How effective have been the communication activities in informing 
the  potential  applicants  about  upcoming  calls  and  in  increasing  the 
visibility of funded projects? 

7.  EQ7.  Will  the  results  of  the  implemented  actions  be  sustainable  in  the 

long-term?  

i)  Do  funded  actions  achieve  results  that  last  beyond  the  duration  of 

individual projects?  

ii)  Do partnerships among beneficiaries of the Programme endure beyond 

participation in the individual calls? 

Efficiency 

Relevance 

Coherence, 
Complementarit
y, Synergies 

EU added value 

8.  EQ8: Which are the costs and benefits of the intervention? 
9.  EQ9: To what extent has the intervention been cost-effective? 
10. EQ10:  What 
influenced 

the  efficiency  with  which 

factors 
achievements observed were attained? 

the 

11. EQ11:  Is  there  any  scope  for  using  alternative  implementing  measures 
other  than  action  grants  and  operating  grants,  e.g.  innovative  financial 
instruments? 

12. EQ12:  How  relevant  were  the  actions  funded  under  the  Programme  to 

the needs of the different stakeholders? 

i)  Did  the  Programmes  address  stakeholders’  needs,  as  identified  in  the 

intervention logic of the Programmes?  

ii)  Has the Programme provided relevant support to beneficiaries, in terms 

of capacity to address the needs of target groups? 

13. EQ13:  How  well  do  the  (original)  objectives  (still)  correspond  to  the 

needs within the EU? 

14. EQ14: How relevant is the EU intervention to EU citizens? 
15. EQ15: How relevant were the groups targeted by the intervention?  
16. EQ16: To what extent is the Programme coherent / complementary with 
other  EU  and/or  national  policies  and  funding  Programmes  that  have 
similar objectives to the Union bodies' work (external coherence)? 

17. EQ17:  What  synergies  exist  within  the  Programme  and  with  other  EU 

Programmes? 

18. EQ18:  To  what  extent  the  different  actions  and  interventions  lead  to  a 

coherent approach within the Programme (internal coherence)? 

19. EQ19:  To  what  extent  is  the  intervention  coherent  with  international 
obligations, including the 2030 Agenda for sustainable development? 

20. EQ20:  What 

the  EU 
intervention(s), compared to what could be achieved by Member States? 
21. EQ21:  To  what  extent  do  the  issues  addressed  by  the  intervention 

the  additional  value 

resulting 

from 

is 

continue to require action at EU level? 

22. EQ22: What would be the most likely consequences of limiting the level 
of  the  existing  EU  intervention  or  completely  stopping/withdrawing 
from it?  

i)  How have policy areas relevant to the Programme evolved since 2014, 

and how does this affect the added value of the Programme?  

ii)  How  has  the  Programme  influenced  national  policies  in  its  respective 

fields? 

23. EQ23:  How  much  EU  added  value  resulted  from  national  projects 

compared to transnational projects?  

i)  Did any national projects overlap with, and address the same needs as, 

projects funded by the Programme? 

ii)  Did  projects  funded  under  the  Programme  foster  the  development  of 

effective and durable cross-border networks? 

Equity 

24. EQ24:  How  fairly  are  the  different  activities  distributed  across  the 

20 

 
Evaluation 

criteria 

Evaluation questions 

different target groups and EU Member States? 

i)  Were  resources  of  the  Programme  allocated  with  the  differentiated 

needs of target groups from different MS in mind? 

ii)  Were resources allocated also in consideration of the different needs of 

beneficiaries across Member States? 
25. EQ25: How has gender mainstreaming been promoted? 

i)  Was  gender  mainstreaming  (intended  as  a  cross-cutting  theme) 
included  in  the  programming  and  implementation  of  the  Programme, 
including calls for proposals?  

ii)  How  was  the  principle  of  gender  mainstreaming  applied  in  practice  by 

beneficiaries? 

26. EQ26: How have the rights of the child been promoted? 
27. EQ27: How have the rights of people with disabilities been promoted? 
28. EQ28: How can the Programme management with focus in particular on 
the  grant  management,  be  further  simplified  to  alleviate  administrative 
burden of the Commission and of the applicants? 

29. EQ29.  Is  there  any  scope  for  further  simplification  through  changes  in 

the management mode of the Programme? 

Scope for 
simplification 

The  evaluation  deployed  a  series  of  quantitative  and  qualitative  tools,  which  were 
devised  to  provide  detailed  responses  to  specific  evaluation  questions  associated  with 
each  evaluation  criteria.  The  evaluation  methodology  relied  on  a  Mixed  Methods 
approach  which  integrates  and  compares  quantitative  and  qualitative  approaches,  data 
collection,  data  analysis  and  interpretation  in  order  to  strengthen  the  reliability  of  data, 
the  validity  of  the  findings  and  recommendations  by  triangulating  multiple  sources  of 
information.  

The analysis covered the call for proposals and procurement activities financed in 2014, 
2015 and 2016 annual work programmes. The 2017 work programme, on the other hand, 
has been analysed chiefly in terms of design and structure, not in terms of execution.  

One  key  method  to  analyse  the  Programme  was  desk  research,  which  focused  on 
available  documents  at  programme,  policy  and  project  level.  The  desk  research  was 
complemented by fieldwork, which was leveraged to collect relevant data and input from 
stakeholders.  This  helped  to  fill  knowledge  gaps  and  validate  information  which  was 
retrieved during the course of the desk research. The following fieldwork activities were 
performed:  1.  A  web-based  survey  was  launched  to  gauge  and  compare  the  views  of 
relevant  stakeholders  (i.e.  project  applicants,  project  beneficiaries,  Programme 
Committee  Members  and  additional  relevant  stakeholders);  2.  33  interviews  were 
conducted  with  project  beneficiaries  and  Commission  officials;  3.  Finally,  a  webinar 
was  held,  which  provided  a  chance  to  discuss  preliminary  findings  with  a  number  of 
beneficiaries.  

The  collected  data  was  then  used  to  perform  a  series  of  quantitative  and  qualitative 
analyses.  On  the  quantitative  side,  the  analysis  performed  helped  provide  perspective 
both at the Programme level, assessing for instance how and where the Programme has 
deployed the available financial resources, and at the project level, giving a granular view 
of  quantitative  indicators  provided  by  individual  projects.  Also,  a  Social  Network 
Analysis  was  performed;  this  helped  achieve  a  “bird’s-eye”  view  of  the  collaborations 
and  transnational  networks  that  were  developed  as  a  result  of  projects  financed  by  the 
2014-2020  Rights,  Equality  and  Citizenship  Programme.  Finally,  concerning  the 
qualitative  analysis,  automated  text  mining  techniques  reinforced  with  machine 
learning  algorithms  were  deployed  to  help  sift  through  the  large  amounts  of  data 
concerning individual projects. This was done in light of the need to give structure and 

21 

 
 
find meaning within a large and mostly unstructured dataset (namely project applications 
and final reports), so as to create taxonomies of the information present in free text fields. 
Crucially, text mining helped in the analysis to find commonalities and correlations in the 
data that might not been found had solely manual techniques been deployed. 

Robustness and limitations of findings 

The  evaluation’s  findings  and  conclusions  are  deemed  as  credible  and  consistent 
according  to  the  methodology.  There  were,  however,  some  challenges  which  by  their 
nature could not always be mitigated.  

First  of  all,  only  5  projects  had  both  monitoring  and  survey  data  were  available  which 
means  that  no  statistical  inference  has  proved  feasible  and  that  the  triangulation  of 
quantitative  data  at  the  project  level  has  been  limited.  Nevertheless,  the  analysis  of  a 
series of output indicators allowed to have a description of the actions’ features as well as 
progress in terms of target groups reached and activities carried out.  

Moreover, another challenge was that in evaluations of EU legislation “it can be difficult 
to identify a robust counterfactual situation (i.e. what the situation would be if EU laws 
had not been adopted)”, which made quantitative analysis problematic27.  

In view of these limitations, a counterfactual analysis, built on qualitative data, has been 
considered  to  be  the  more  suitable  method.  The  analysis  of  direct  questions  on 
behavioural change assumes that the respondents are able to reflect on their behaviour in 
hypothetical,  counterfactual  situations  and  that  they  are  telling  the  truth  to  the  best  of 
their knowledge. However, as respondents have an interest in the continuation of public 
support,  they  might  be  tempted  to  over-emphasize  the  merits  thereof  by  answering 
strategically28.  To  address  the  positive  bias  in  answers  by  beneficiaries  and  the  self-
selection bias in answers from applicants who have not received funding, a similar set of 
questions was asked to applicants having had their proposal rejected. As the features of 
all  respondents  (both  successful  and  unsuccessful  applicants)  are  known,  this  helped 
triangulate findings gathered from beneficiaries.  

Concerning the indicators, those set in the Regulation represent a mixture of output and 
result indicators which conceptually are separate. In general, each project establishes its 
own  outputs  indicators.  This  means  that  output  data  reported  is  based  on  applicants' 
indications  in  the  application  form  and  not  on  beneficiaries’  actual  achievements. 
Furthermore, since as no targets are set for these indicators, their performance (measured 
as  the distance between the target and actuals  achieved) cannot  be assessed. Moreover, 
no  targets  are  set  in  several  specific  objectives,  such  as  the  specific  objective  on 
“preventing  and  combating  racism,  xenophobia,  homophobia  and  other  forms  of 
intolerance”, on “promoting and protecting the rights of people with disabilities” (apart 
from the employment rate of persons with disability) and on “promoting and protecting 
the rights of the child.  

27 European Commission, Better Regulation Toolbox, p. 344, https://ec.europa.eu/info/sites/info/files/better-regulation-toolbox_0.pdf.  
28 In interpreting the results, it should be considered that there  may be two sources of bias to the answers: 1.  Strategic behaviour 
(beneficiaries may tend to overestimate the effect of the treatment to further legitimise their receipt of funding, while applicants may 
tend to say to be well off also in absence of funding – also to qualify for the next round); 2. Estimation bias: both beneficiaries and 
applicants  don’t  know  exactly  what  the  counterfactual  situation  would  be  like,  so  answers  are  based  on  their  educated  guess.  This 
applies especially to applicants who can base their judgment only on the knowledge they acquired in the application phase with no 
actual exposure to the implementation of the measures. 

22 

 
 
 
                                                            
In  addition,  even  though  the  9  case  studies  proved  to  be  a  useful  tool  to  get  a  better 
picture  of  the  Programme’s  activities,  their  representativeness  of  the  Programme  as  a 
whole was somewhat limited. 

Despite  these  limitations,  the  evaluation  carried  out  is  supported  by  a  robust  evidence 
base. 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS 

The  Rights,  Equality  and  Citizenship  Programme  is  performing  generally  well  at  mid-
term  with  regard  to  its  specific  objectives,  in  terms  of  effectiveness,  efficiency, 
relevance,  coherence/complementarity/synergies  and  EU  added  value,  but  should  do 
better, in particular, in terms of equity29.  

5.1 Effectiveness 

As the progress in  achieving  the Programme’s  outputs and outcomes  has  been  good so 
far, it is reasonable to expect that the Programme is making good progress towards 
achieving also its general objective (namely “the further development of an area where 
equality and the rights of persons as enshrined in the TEU, in the TFEU, in the Charter 
and  in  the international  human  rights  conventions  to  which  the  Union has  acceded,  are 
promoted, protected and effectively implemented).  
However, sometimes is challenging to trace and attribute the changes in the global 
indicators to the interventions of the Programme. Its contribution to the achievement 
of the outcomes on awareness of EU citizenship rights is likely to be substantial, as there 
is  the direct impact of the Programme through its focus on awareness-raising measures 
aimed at enabling change.  In contrast, in the achievement of targets in relation to female 
employment rates, the representation of women in the higher decision-making levels and 
the  employment  rate  of  people  with  disabilities,  the  contribution  of  the  Programme 
activities is more indirect as change in these areas requires a more structural change in 
the  labour  markets  and  economic  structures  which  are  outside  the  direct  sphere  of 
influence of the Programme. The programme can be expected to play an important role in 
supporting  such  structural  changes,  but  the  actual  change  will  depend  on  many  factors 
external to the Programme, including the business cycle. 
Concerning  its  specific  objectives,  the  progress  in  the  achievement  of  targets  in 
relation  to  the  nine  objectives  is  uneven.  With  regard  to  three  specific  objectives 
(namely  the  ones  on  promoting  non-discrimination,  promoting  the  rights  deriving  from 
the  citizenship  of  the  Union  and  promoting  consumer  rights),  the  Programme’s 
contribution is direct and substantial and their targets are likely to be achieved.  
The  evidence  on  the  impact  of  the  Programme  on  the  achievement  of  other  specific 
objectives showed that the Programme activities are contributing to the achievement of 
their targets, but this impact is moderate, due both to the limited amount of Programme 
resources (especially in the areas of fighting and preventing violence, anti-discrimination 
and  combating  racism),  as  well  as  change  being  influenced  by  many  other  different 
factors  and  policies  (such  as  the  differences  across  national  contexts,  the  national 
unsupportive policy and legislative framework, the lack of cooperation with government 

29 For more information and evidence, see section 7 "Responses to evaluation questions" in the Interim report, p. 84 et seq. 

23 

 
 
 
 
 
                                                            
bodies  and  institutions,  the  public  budget  cuts  following  the  economic  crises  and  the 
consequent austerity measures).  
In  any  case,  the  key  result  achieved  across  the  specific  objectives  is  the  improved 
skills and competences of professionals. This is also the key aim of the Programme: to 
build  capacity  of  the  key  stakeholder  groups  to  instigate  change  in  the  specific  policy 
areas.  
Another  important  achievement  of  the  Programme  is  the  contribution  to  systemic 
change  by  project  results,  including  better  tools,  procedures,  services  and  policies 
developed across the specific objectives. 
The monitoring indicators selected for measuring the achievements of the Programme are 
overall  adequate,  but  some  improvements  could  be  implemented,  for  example  data 
collection  and  processing  procedures  for  the  indicators  set  in  the  Regulation  could  be 
clearer. 
Furthermore,  the  evaluation  carried  out  suggests  that  the  system  of  indicators  could  be 
further  enriched  to  capture  the  whole  range  of  expected  results  of  the  Programme.  For 
instance,  indicators  capturing  the  improvement  brought  about  by  capacity  building 
activities  of  public  institutions’  staff  and  non-governmental  organisations  in  preventing 
non-discrimination  or  in  tackling  the  cases  of  violence  could  be  added,  as  well  as 
indicators measuring the level of awareness of the changes introduced by the 2016 Data 
Protection Reform. Moreover, there are different specific objectives that do not have set 
targets or measurement indicators and this should be an area for further improvement. 
The  indicators  relating to  the  physical  and  financial  progress  in  the  achievement  of  the 
outputs of the Programme show improvements. In particular, the reach of participants by 
the awareness-raising activities can be expected to be above the figures achieved in the 
predecessor  programmes.  Similarly,  the  number  of  stakeholders  participating  in  the 
funded  activities  can  be  expected  to  exceed  the  achievements  of  the  predecessor 
programmes. This is due to the fact that the communication activities undertaken by the 
Commission to promote the Programme have been largely successful and the popularity 
of  the  Programme  has,  therefore,  increased  compared  to  the  predecessor  programmes. 
According  to  the  evaluation,  the  majority  of  the  beneficiaries  considered  that  the 
communication  activities  were  clear  and  understandable,  information  was  relevant  and 
complete and the right audience was targeted to a very high and high extent30. 
However, efforts need to be maintained to ensure all potential applicants in all Member 
States are reached. Indeed, one area where the Programme did not improve compared to 
the  predecessor  programmes  is  the  geographical  coverage  of  its  activities.  As  in  the 
previous  period,  the  Rights,  Equality  and  Citizenship  Programme  has  been  so  far 
dominated  by  the  participants  from  few  Member  States.  However,  a  positive 
development in this respect is the increased participation of Romania and Bulgaria. 
The  benefit  of  the  Programme  has  been  high  compared  to  the  predecessor 
programmes also in terms of applications received and awards granted. The award 
rate has also increased slightly compared to the predecessor programmes (from average 
6-7 applications per award to average 8 applications per award). 
An  additional  positive  development  has  been 
financial 
implementation of  the Programme  since the level of funding requested by applicants 
and granted in relation to each specific objective in years 2014-2015 has been around 90-
95% and is slightly higher than in the predecessor programmes. However, stakeholders’ 
view was less positive about the capacity to target all potential applicants. This confirms 
that the geographical coverage of the different typologies of beneficiaries of the different 

the  high 

level  of 

30 See Interim report, ibid. 

24 

 
                                                            
Member States is very uneven in the Programme and also the communication activities 
have not been equally successful from a geographical point of view. 
Moreover,  based  on  several  indicators  and  according  to  the  surveyed  and  interviewed 
stakeholders31,  the  evaluation  showed  positive  progress  in  delivering  improvements  in 
the level of knowledge of EU law and EU policies, rights and values (given the number 
and  the  extent  of  awareness  activities  carried  out  at  project  level),  as  well  as  in  the 
organisational  structures,  skills  and  knowledge  amongst  the  groups  participating  in  the 
activities funded by the Programme.  
As mentioned above, the partnerships formed during the Programme activities have had 
positive  impacts  on  beneficiaries  and  Programme  participants’  skills  and  capacities. 
According  to  the  results  of  the  evaluation,  the  most  frequently  reported  result  was  the 
improved skills and competences of professionals, which is highly appropriate given the 
objective of the Rights, Equality and Citizenship Programme. Important and appropriate 
have been also the frequent results reported in projects promoting the policy development 
and quality of services. In addition, 74% of the responding beneficiaries reported having 
benefitted  from  the  increased  capacity  of  the  organisation  and  41%  from  creating  new 
infrastructures/departments  within  the  organisation  as  a  result  of  participation  in  the 
Programme. 
Concerning the sustainability of the Programme’s activities after the end of the funding, 
the majority of surveyed stakeholders considered that actions are very highly likely to be 
sustained in terms of acquiring new skills and knowledge, increased awareness and the 
creation  of  tools  and  procedures  outlasting  the  projects.  Slightly  lower  was  the 
expectation of the training activities lasting after the project completion. However, only a 
third  of respondents expected the partnerships  which were  funded during the project  to 
last  beyond  the  funding  period  (despite  the  majority  view  that  they  have  improved  the 
organisation’s  capacity  building  and  contributed  to  the  sustainability  of  the  project 
results).  Similarly,  low  is  the  proportion  of  the  stakeholders  who  considered  that  the 
partnerships formed during the projects financed by the Programme have increased their 
fundraising  capabilities  which  is  one  of  the  factors  to  ensure  the  organisational 
sustainability. In general, there is very little evidence about the sustainability of project 
results after the end of funding. There seems to be little systematic planning, monitoring 
and implementation of activities which would support the sustainability of project results. 
Finally,  more  than  80  %  of  survey  respondents  state  that  the  Programme  is  (very) 
highly effective in meeting the needs of the relevant target groups. Over 70% of the 
Programme beneficiaries surveyed in the evaluation also considered that the Programme 
has  been  highly  and  very  effective  in  targeting  the  right  policy  areas  and  the  most 
relevant  target  groups.  This  is  an  important  improvement  perceived  amongst  these 
beneficiaries, particularly as they have been active in both programming periods and are 
therefore in a good situation to provide a direct comparative analysis. 

5.2 Relevance 

The Programme is highly relevant to the needs of its stakeholders and beneficiaries, 
e.g.  in  terms  of  knowledge  development,  training,  awareness  raising  and  structural 
support. Indeed, the stakeholders surveyed in the evaluation were very positive about the 
relevance of the Programme actions in meeting their needs.  
However,  the  evaluation  also  found  some  gaps  in  relation  to  the  needs  of  some 
stakeholders and citizens that could receive greater attention by the Programme, such as 

31 See Interim report, ibid. 

25 

 
 
 
                                                            
women subject to multiple discrimination (e.g. migrant, disabled, older women), children 
with  disabilities,  family  carers  of  elderly  with  degenerative  diseases,  refugees  with 
psychological  problems,  children  left  behind  by  migrant  families  in  eastern  European 
countries, EU citizens outside the EU. 
The  needs  identified  at  the  time  of  the  Programme’s  adoption  are  still  relevant, 
since, as demonstrated by the evaluation carried out, the achievement of several targets, 
relating,  in  particular,  to  gender  equality  or  rights  of  disabled  people  is  unlikely  as 
progress  has  been  slow.  This  means  that  the  overall  support  needs  as  identified  at  the 
start  of  the  Programme  have  not  been  met  and  continue  to  be  relevant.  Also  all 
beneficiaries  interviewed  agreed  that  the  Programme’s  objectives  and  funded  activities 
are still relevant for the current needs of the groups they target.  
The  Programme  is  also  flexible  enough  to  incorporate  emerging  needs,  thanks  to 
some calls responding to the real and urgent stakeholders’ and citizens’ needs, other calls 
formulated in a sufficiently open way to allow a margin of definition of specific activities 
and others targeting emerging new needs. 
Overall,  the  Programme  is  appropriately  addressing  the  relevant  groups  it  is 
expected  to  target.  Indeed,  given  that  the  Programme  is  expected  to  engage  the  civil 
society organisations, its focus on providing funding to non-governmental organisations 
was  appropriate,  as  well  as  the  involvement  of  other  types  of  stakeholders  to  provide 
analytical activities (especially relevant to the universities and research organisations) or 
direct awareness raising activities (pertinent to the public bodies).  
Another indication of the continuing relevance of the Programme to the current needs is 
the  increase  of  demand  for  grants  funded  compared  to  previous  programmes, 
especially  concerning  the  specific  objectives  relating  to  the  violence  against  children, 
non-discrimination  and  combating  racism.  On  average  the  Rights,  Equality  and 
Citizenship Programme is receiving around 900 applications per year. In comparison, on 
average the predecessor programmes have received around 400 applications per year. For 
example,  the  number  of  applications  to  the  specific  objective  relating  to  the  violence 
against  children  was  around  380,  whereas  the  average  number  of  applications  to  the 
predecessor Daphne III programme was around 270.  

5.3 Efficiency 

The  analysis  of  the  efficiency  is  based  on  a  qualitative  evaluation  of  the 
beneficiaries’ perception of the cost-effectiveness of the actions since the limited data 
available  in  relation  to  the  costs  and  benefits  of  the  Programme’s  intervention  do  not 
allow  for  the  application  of  standard  quantitative  methodologies  for  a  cost-benefit 
assessment. 
The  costs  relate  to  the  management  of  the  programme  at  EU  level,  as  well  as  costs 
incurred by beneficiaries when applying for funding and when implementing the actions 
for which funding was received. The costs include direct costs (such as costs of labour, 
travel  costs,  subsistence  costs,  subcontracting)  and  indirect  costs  (such  as  water, 
electricity/gas and heating, telephone, fax, internet and e-mail). 
The information obtained from the survey of beneficiaries was not sufficient to calculate 
the  average  amount  for  each  cost  category,  due  to  the  heterogeneity  of  the  answers 
provided. For 60% of respondents the main typology of costs relates to staff costs (i.e. 
accounting and reporting, including financial reporting, and proposal preparation).   
The benefits of the Programme relate to development of a European area where equality 
and the rights of persons are promoted and protected. At this stage, an assessment of the 
benefits  in  terms  of  the  result  of  the  Programme  can  be  derived  from  the  results  that 

26 

 
 
 
beneficiaries expect for the specific objectives of the Programme32. Examples of concrete 
benefits  derive  from  the  results  of  the  evaluation:  74%  of  the  responding  beneficiaries 
reported having benefitted from the increased capacity of the organisation and 41% from 
creating  new  infrastructures/departments  within  the  organisation  as  a  result  of 
participation  in  the  Programme.  Moreover,  the  performance  of  the  projects  in  terms  of 
training activities is beyond expectations for several specific objectives. In particular, this 
is  true  for  the  specific  objective  on  preventing  violence  against  children,  young  people 
and  women  and  this  is  also  more  remarkable  in  the  light  of  the  reduced  incurred 
expenditure (at around 60% of the estimated).  
In  conclusion,  according  to  the  majority  of  beneficiaries  and  Member  States 
representatives,  the  costs  of  the  Programme  are  either  proportionate  to  benefits  or 
outweighing  them.  The  interviews  furthermore  show  that  an  important  reason  for  this 
positive  assessment  is  that  the  Programme  allows  to  experiment  innovative  approaches 
and to show national governments the return of the activities performed.  
A  key  achievement  of  the  Programme,  compared  to  its  predecessors,  has  been  also  the 
lower  demand  on  beneficiaries  in  terms  of  time  and  financial  resources.  More 
specifically,  62%  of  the  responding  beneficiaries  saved  time  in  preparing  proposals 
because  of  simpler  procedures,  fewer  overlaps  and  duplications  in  procedures, 
requirements and necessary tasks; 44% reported saving material resources thanks to such 
provisions in the Programme. In addition, 52% of the responding beneficiaries reported 
having  spent  less  resources  in  implementing  activities  and  54%  reported  spending  less 
time. 
Moreover,  according  to  stakeholders,  the  changes  in  the  design  of  the  Programme, 
compared to 2007-2013, are the most important factor for the efficiency of projects 
in  achieving  the  results.  According  to  the  32%  beneficiaries,  these  changes  influence 
the efficiency of projects to a moderate or high extent. 
Beneficiaries  consider  current 
the 
Programme33.  However,  they  consider  that  there  can  be  scope  for  using  alternative 
measures, such as microcredit and small loans, as a solution to increase the capacity of 
participating  organisations  and  potential  participants  (which  is  considered  as  the  most 
influencing factors for the efficiency of projects in achieving their results).  
Finally,  several  beneficiaries  pointed  out  to  the  issue  of  the  relative  long  time  span 
between  the  application  and  the  start  of  project  activities  as  problematic  for  their 
efficiency,  since,  in  this  long  period,  contextual  factors  may  change,  requesting  an 
additional effort in adapting the project to the new context34.  

instruments  as  adequate 

funding 

for 

5.4 Coherence, Complementarity, Synergies 

According  to  beneficiaries,  the  Programme  presents  a  high  level  of  internal 
coherence  (between  the  specific  objectives  and  interventions)  and  a  high  level  of 
external  coherence  and  complementarity  with  other  EU  instruments,  programmes 
and actions, such as the Charter of Fundamental rights, the European Agenda for Justice 
and the programmes/funds “Europe for citizens”, Horizon 2020, the European Structural 
and  Investment  Funds,  the  Employment  and  Social  Innovation  Programme,  Creative 
Europe  Programme  (in  particular  MEDIA)  and  Erasmus+.  Beneficiaries  especially 
highlighted  a  high  degree  of  coherence  between  the  Programme  and  the  European 
Structural and Investment Funds in the area of anti-violence, rights of the child and non-
discrimination. This is probably explained by the fact that the European Social Fund and 

32 For more info on the specific objectives and expected results, see Interim report, ibid. 
33 See Interim report, ibid. 
34 See Interim report, ibid. 

27 

 
 
 
                                                            
the  European  Regional  Development  Fund  (which  are  part  of  the  European  Structural 
and Investment Funds) can finance projects aimed at the social integration of people at 
risk of discrimination and of most vulnerable people, such as victims of violence. 73% of 
respondents reported having established synergies with projects within the Employment 
and  Social  Innovation  Programme,  the  European  Social  Fund,  including  Youth 
Guarantee,  and  Erasmus+.  The  evaluation  showed  also  that,  according  to  70%  of 
beneficiaries  surveyed,  these  synergies  were  established  on  the  occasions  of  meetings 
organised  by  the  European  Commission.  In  addition,  the  webinar  confirmed  that  EU 
level  events  represent  useful  opportunities  for  beneficiaries  to  meet  EU  level  policy 
makers and to network with peer organisations operating in different countries on similar 
issues.  Furthermore,  it  seems  that  synergies  between  beneficiaries  of  action  grants  and 
beneficiaries  of  operating  grants  are  developed,  especially  in  the  field  of  rights  of  the 
child  and  rights  of  people  with  disabilities35.  Still,  coherence  with  other  EU  initiatives 
and  a  clearer  streamlining  of  concepts  and  messages  could  have  been  strengthened 
through  a  more  intense  consultation  and  cooperation  among  the  Commission's 
Directorates-General in the preparation of the calls. 
No significant overlaps with other EU funded programmes have been found in the 
evaluation.  This  would  seem  to  confirm  that  specific  objectives  are  more  streamlined 
and clearly defined if compared to the predecessor programmes.  
The Programme is highly coherent with the internationally agreed principles, such 
as  the  2030  Agenda  for  Sustainable  Development.  The  United  Nations  launched  the 
2030  Agenda  for  Sustainable  Development  (“Transforming  Our  World”)  in  2015.  It 
presents  the  main  international  obligations  as  regards  sustainable  development.  Three 
goals  in  the  Agenda,  in  particular,  correspond  to  policy  priorities  in  the  2014-2020 
Rights,  Equality  and  Citizenship  Programme:  1.  Achieve  gender equality  and  empower 
all  women  and  girls  (Goal  5);  2.  Reduce  inequality  within  and  among  countries  (Goal 
10);  3.  Promote  peaceful  and  inclusive  societies  for  sustainable  development,  provide 
access to  justice for all and build effective, accountable and inclusive institutions at  all 
levels (Goal 16). 60% of the applicants and 65% of the beneficiaries surveyed consider 
the  Rights,  Equality  and  Citizenship  Programme  coherent  with  the  Agenda,  mainly 
through  three  of  its  specific  objectives,  namely  on  gender  equality,  non-discrimination, 
prevention of violence against children, young people and women . 
According  to  almost  90%  of  beneficiaries,  the  Rights,  Equality  and  Citizenship 
Programme is also overall consistent, in its objectives, targets and types of actions, 
with  national  policies  in  the  same  field.  Unsuccessful  applicants  and  Programme 
Committee Members report that there is an overall consistency between the Programme 
and national policies, especially in the area of gender equality.  
The reason for inconsistencies, specifically reported in relation to the areas of hate crime 
and  hate  of  speech  and  non-discrimination,  seem  to  relate  to  lower  Member  States’ 
commitment  in  these  areas  covered  by  the  Programme  compared  to  the  European 
Commission’s commitment.  

5.5 EU added value 

The  inherent  “European”  and  transnational  dimension  is  at  the  core  of  the  EU 
added  value  of  the  Programme.  According  to  the  majority  of  beneficiaries,  in  the 
absence of the Programme activities, similar projects would not have taken place or not 
be  possible  with  the  same  coverage  in  terms  of  beneficiaries  and  target  groups,  due  to 

35 For more detailed information on the complementarities of the specific objectives of the Programme with other EU funding, see the 
Interim report, ibid. 

28 

 
 
 
                                                            
lack  of  available  resources  at  Member  State  level.  This  is  particularly  relevant  for  the 
training actions in the field of prevention of all forms of violence and in the field of data 
protection.  Similarly,  EU  funding  for  EU  level  organisations  is  particularly  vital  as 
national funding for a  transnational network would be even  more difficult to  obtain. In 
this  regard,  the  Programme  supports  Member  States  in  progressing  towards  common 
standards and practices by helping to enforce EU legislation and policies (such as in the 
field of data protection). 
Analytical activities funded through procurement actions are key to the production of EU 
added  value  as  these  serve  to  identify  gaps  in  Member  States  capacities  and  inspire 
capacity  building  activities  to  fill  in  those  gaps.  Moreover,  projects  financed  by  the 
Programme, according to surveyed stakeholders36, seem to have higher quality and to 
be  more  innovative  than  those  funded  at  the  national  level.  In  particular,  these 
projects,  by  allowing  innovative  approaches  to  be  experimented  with,  can  be  used  to 
show  national  governments  the  benefits  of  the  activities  undertaken.  For  example, 
trainings at schools in the field of bullying would not apply innovative methods, such as 
those  involving  both  perpetrators  and  victims  and  another  example  are  the  projects 
aiming to pilot the EU disability card. These were supported under the specific objective 
on  the  rights  of  people  with  disabilities  and  aimed  at  setting  up  the  respective  EU 
Disability  Card  national  organisations  (governmental  or  nongovernmental  entities)  and 
establishing the respective national package of benefits that Member States are ready to 
mutualise.  In  addition  to  funding  the  piloting  of  this  innovative  action,  the  national 
engagement in these projects brought more attention and strength to the issue, triggering 
the initiative, in terms of impact and visibility. Another example is funding the National 
Roma Platforms that contribute to increase their number and to strengthen their capacity 
to  play  active  roles  in  the  coordination  and  monitoring  activities  of  the  National  Roma 
Coordination points. They shall be empowered to participate in the dialogue, cooperation 
and coordination of stakeholders.37 
Finally,  the  demand  for  Programme  funding  has  continued  to  be  high  throughout 
the implementation of the Programme. All specific objectives have a higher number of 
applications  than  the  number  of  grants  awarded,  especially  in  the  areas  of  fighting  and 
preventing  violence,  anti-discrimination  and  combating  racism.  Indeed,  as  already 
mentioned in the section on the "state of play", the most competitive specific objectives 
are the ones related to the promotion of non-discrimination and prevention of violence.  
All these findings suggest that there is sustained demand for EU-level action in the fields 
covered by the Programme. 

5.6 Equity 

The Programme has provided specific support to the promotion of the cross-cutting 
priorities of gender equality, rights of child and rights of people with disabilities.  
The  issue  of  the  gender  mainstreaming  is  promoted  through  a  dedicated  specific 
objective. In total, the actions under this specific objective received around 12%  of  the 
Programme  resources.  This  makes  it  the  third  largest  specific  objective  in  terms  of 
allocated  funding.  Furthermore,  gender  equality  specific  actions  have  also  been  funded 
under  other  specific  objectives,  including  the  ones  on  non-discrimination  and  on 
combatting  violence  against  children,  young  people  and  women.  This  means  that 
significant  financial  resources  have been  dedicated  to  the promotion of gender equality 

36 See Interim report, ibid. 
37 There are: national, regional, local authorities, Roma and non-Roma communities including Roma youth and Roma women, Roma 
and  pro-Roma  non-governmental  organisations  (NGOs),  private  entrepreneurs,  trade  unions,  professional  associations,  academia, 
equality bodies, etc. 

29 

 
 
 
                                                            
and  gender  mainstreaming  directly.  Gender  mainstreaming  principles  and  mechanisms 
are reflected in all Programme phases of programming, implementation and monitoring 
but  still  there  is  scope  for  strengthening  the  protection  of  women  facing  multiple 
disadvantages  (since  gaps  in  addressing  their  needs  have  been  detected)  and  also  for 
involving more the equality bodies in the setting of Programme priorities. 
Concerning the protection of the rights of the child, the Programme has two specific 
objectives  directly  targeting  the  promotion  of  the  rights  of  the  child  (the  more  general 
one on the promotion and protection of the rights of child and the other to prevent and 
combat  violence  against  children  and  other  vulnerable  groups).  Furthermore,  also other 
specific  objectives  could  fund  actions  potentially  relevant  for  children  (e.g.  concerning 
children belonging to ethnic minorities at risk of discrimination or to vulnerable groups 
such  as  children  with  disabilities  or  protect  the  personal  data  of  children).  However, 
formally  the  child-centred  perspective  is  not  reflected  in  other  relevant  specific 
Programme  objectives  and  the  Regulation  also  did  not  establish  a  formal  need  to 
mainstream  the  rights  of  the  child  across  all  Programme  activities  (in  contrast  to  the 
requirement  for  gender  mainstreaming).  Nevertheless,  the  respect  of  the  rights  of  the 
child  is  enhanced  by  the  fact  that  organisations  applying  for  funding  (and  any  of  their 
partners), which will work directly with children during the project implementation, must 
provide  the  Commission  with  a  description  of  their  child  protection  policy.  In  general, 
the  mainstreaming  of  rights  of  child  across  all  Programme  phases  could  be  further 
improved.  In  particular,  through  the  involvement  of  bodies  representing  the  interest  of 
children  in  the  setting  of  Programme  priorities  and,  according  to  the  majority  of  the 
stakeholders, the rights of child  should  be more present  in the Programme calls  and be 
promoted (around 70% of respondents considering this to a very high and high extent).  
The  Programme  envisages  a  specific  objective  for  the  promotion  of  the  rights  of 
persons with disabilities (that accounts for 10% of the Programme funding) and one on 
the  promotion  of  the  effective  implementation  of  the  principle  of  non-discrimination, 
including  based  on  disability.  However,  formally  the  disability  perspective  is  not 
reflected  in  other  relevant  specific  objectives  and  the  Regulation  did  not  establish  a 
formal  need  to  mainstream  rights  of  people  with  disabilities  across  all  Programme 
activities (in contrast to the requirement for gender mainstreaming). 
All applications submitted under the specific objective dedicated to the rights of people 
with  disabilities  have  been  awarded,  although  it  should  be  noted  that  the  number  of 
applications  was  relatively  small.  The  mainstreaming  of  rights  of  rights  of  disabled 
people across all Programme phases needs to be increased, in particular through a direct 
link  with  the  EU  Disability  Strategy  2010-202038  and  a  more  involvement  of  bodies 
representing  the  interests  of  disabled  people  in  the  setting  of  Programme  priorities. 
According to stakeholders, the rights of people with disability should be more present in 
the Programme calls and be promoted (around 70% of respondents considering this to a 
very high and high extent).  
Always  in  relation  to  equity,  the  Programme  does  not  seem  to  capture  the  needs  of 
people suffering from multiple disadvantages, as the specific objectives do not support 
synergies and actions for groups falling under two or more of them.  
In  addition,  co-financing  seems  to  be  a  barrier  for  small  non-governmental 
organisations wishing to participate in the Programme. 
Finally,  as  already  mentioned,  in  the  future,  the  Programme  should  try  to  distribute  its 
resources  in a  more balanced manner across  the  different target groups of beneficiaries 
and  Member  States.  The  unequal  geographical  representation  of  Member  States  was 
detected  also  in  relation  to  the  predecessor  programmes.  This  means  that  further  effort 

38 See http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2010:0636:FIN:en:PDF . 

30 

 
                                                            
should be placed in promoting a more equal participation to the Programme funding from 
partners  in  Member  States  that  are  less  likely  to  participate:  such  effort  shall  take  the 
form of increased communication and information  activities, as  well as  better guidance 
and support to potential partners’ organisations in applying for EU funding and managing 
EU-funded programmes. 
Moreover, the Programme has concentrated its resources in two specific objectives (non-
discrimination  and  preventing  violence  against  children),  both  of  which,  so  far,  have 
taken  up  around  50%  of  overall  Programme  resources.  Comparatively,  other  specific 
objectives have received fewer resources (for example, the specific objectives on Union 
citizenship, data protection and consumer rights have received allocations below 5%). In 
the absence of the detailed needs assessments by specific objective at the Member State 
level, it is challenging to draw conclusions on the equity of such allocations. However, 
the  evaluation  already  shows  that  the  funding  does  reach  Member  States  with  a  high 
concentration of the groups in need of support, but also that various Member States with 
similar needs are receiving no funding. 
To understand better how the Programme promotes equity through the funded activities, 
participants'  data  broken  down  by  sex,  disability  status  or  age,  as  required  by  the 
Regulation, shall be collected. This is however not yet done. 

5.7 Scope for simplification 

According to the evaluation, the current direct management mode is adequate given 
the size and objectives of the Programme.  
However,  according  to  the  majority  of  beneficiaries,  there  is  scope  for  further 
simplification  of  the  implementation,  management  and  design  of  the  Programme. 
Among  the  reasons  provided,  beneficiaries  indicated,  in  particular,  that  the  financial 
reporting  is  too  detailed,  especially  compared  to  the  ones  applied  within  other  EU 
Programmes (e.g. Horizon 2020 and Erasmus+) and underlined the possibility to reduce 
the  current  administrative  burden  in  terms  of  proposal  drafting  and  monitoring  and 
reporting requirements. 
Another current key suggestion from beneficiaries concerns the operating grants, whose 
duration in their view could be extended to cover at least two years (instead of one) in 
order to reduce the administrative burden in terms of project application and management 
procedures.  However,  the  annual  operating  grants  allow  the  Commission  to  have 
oversight of the activities of the funded organisations and allow more flexibility to adapt 
the work of beneficiaries to the emerging needs in their respective fields of expertise. 
Furthermore,  the  absence  of  standard  costs  implies  sometimes  lengthy  negotiations 
among  project  partners  on  the  costs  of  the  different  activities.  The  introduction  of 
standard costs, at least for some typologies of activities (e.g. training), could significantly 
reduce  the  administrative  and  accounting  duties  on  beneficiaries  and  on  Commission 
officials involved in management and control.  
On  the  positive  side,  beneficiaries  and  Commission  officials  agreed  that  the  recently 
introduced  system  for  applications  (Participant  Portal)  goes  in  the  direction  of 
simplifying the application process and improving the collection and aggregation of 
monitoring data. However, there could be still room for further simplifying the process, 
particularly in relation to the administrative documentation to be provided and details on 
work packages.  
Moreover, as mentioned above, another  key issue according to the beneficiaries, is  that 
most of the funding available in  the Programme is relatively inaccessible to small civil 
society non-governmental organisations, due to the co-financing amounts required for the 
set size of grants. For small non-governmental organisations providing such amounts is 

31 

 
 
 
very  challenging  and,  as  a  consequence,  most  Programme  funding  goes  to  larger  non-
governmental organisations, even though, within the EU, relevant grass-root activities are 
also delivered by smaller  non-governmental organisations. It is  to be noticed, however, 
that  the average amount  of grants  has  been  increased to  the current level following the 
ex-post  evaluation  of  the  predecessor  programmes  that  found  that  the  dilution  of  funds 
amongst many small-scale projects had had limited impact and EU dimension. 
Finally, as mentioned above, an element of administrative burden, as widely shared by all 
beneficiaries, is the excessive length of the period from the application until the start of 
project activities (which is reported to range between six months and one year). 

 6.   CONCLUSIONS AND WAY FORWARD 

The  Rights,  Equality  and  Citizenship  Programme  2014-2020  was  launched  when  the 
effects  of  the  economic  crisis  had  resulted  in  a  general  reduction  in  the  amount  of 
national resources and funding available for social and human rights issues.  

In this critical political and economic context, the Programme has proved its EU added 
value and crucial role in the development of a European area of equality and rights. 

The Programme's interim evaluation confirms its relevance for: 

  contributing  to  the  promotion  of  relevant  EU  values,  such  as  the  respect  for  human 
dignity,  fundamental  rights,  non-discrimination,  tolerance,  solidarity  and  gender 
equality; 

  promoting the exchange of information at EU level as well as mutual learning and the 
dissemination  of  practices  in  the  area  of  protection  of  rights,  citizenship  and  anti-
discrimination; 

  reinforcing and supporting the development of a stronger EU citizenship. 

The activities and projects financed effectively contributed to its nine specific objectives, 
as set out in the Regulation. The problems underlying the general and specific objectives 
of the Programme are still relevant and its intervention strategy appears still adequate to 
achieving the desired objectives and the needs of the beneficiaries. However, some new 
priorities emerged and improvement could be made in specific areas of the Programme.  

Moreover,  the  geographic  balance  is  still  not  good  overall,  since  the  Programme  is 
dominated  by  beneficiaries  from  only  a  small  number  of  Member  States.  This  balance 
could  be  improved  by  strengthening  the  Programme's  visibility  and  improving 
communication activities in those Member States with relatively low participation rates. 
Dissemination could equally be enhanced through mutual collaboration between funded 
projects. 

Regarding the efficiency, the current Programme has reduced the administrative burden 
on beneficiaries compared to its predecessors. Both action grants and operating grants are 
used  efficiently  for  the  separate  functions  they  fulfil  and  in  pursuing  the  Programme’s 
objectives.  Nevertheless,  there  is  still  room  for  improvements  to  make  the  Programme 
even more efficient in its implementation.  

The simplification carried out, thanks to the merger of the three previous programmes, 
has  produced  improvements  in  the  quality  of  the  evaluation  and  monitoring  process. 
However,  the  revision  of  the  monitoring  indicators  has  been  identified  as  an  area  for 
improvement.  

32 

 
 
 
At  EU  level,  the  Rights,  Equality  and  Citizenship  Programme  is  coherent  and 
complementary with other EU instruments, programmes and actions, with limited risks 
of duplications or incoherence39. 

Lessons learned and way forward for the next programming period40:  

Effectiveness 

  The  Commission  will  consider  actions  to  increase  its  geographical  coverage,  to  introduce  more 
systematic  needs  assessments  and  to  consider  promoting more  opportunities  for networking  and 
information  exchange  within  the  Programme  participant  networks  (e.g.  in  relation  to  the 
sustainability  actions).  Moreover,  the  Commission  will  consider  the  possibility  to  undertake  a 
systematic  and  comprehensive  needs  assessments  at  the  Member  State  level  (above  all  in  the 
under-represented Member States) and to develop a comprehensive communication and outreach 
strategy.  

Relevance 

  The same recommendations illustrated under "Effectiveness" should be taken into consideration 
to reinforce also the relevance of the Programme. Moreover, there is the need to focus more on 
the  target  groups which  are  under-represented  in  the national  contexts (such  as  lesbian  women, 
people in multiple discrimination situations or children in migration situations). 

Efficiency 

  The Commission will consider the possibility to use more restricted calls, open to partnerships led 
by  public  authorities,  in  order  to  engage  key  national  stakeholders  in  designing  action  plans, 
strategies,  protocols  and  coordination  mechanisms  among  relevant  actors.  Moreover,  the 
Commission will consider to what extent alternative funding mechanisms and procedures, such as 
microcredit  and  small  loans,  could  be  introduced  side  by  side  with  existing  ones,  which  are 
reputed to be adequate to meeting Programme goals. 

Coherence, Complementarity, Synergies 

  The  limited  risk  of  overlaps  and  duplication  of  activities  can  be  further  reduced  by  organising 
more  regularly  EU  level  networking  events.  Moreover,  the  Commission  will  consider  the 
possibility to ensure a more intense consultation and cooperation among the Directorates-General 
in the preparation of the calls.  

EU added value 

  The Commission will consider the possibility to apply the approach followed by the Programme 
with regard to the European Disability Card or to the National Roma Platforms also to other areas 
of the Programme.  

Equity 

  The  Commission  will  consider  to  undertake  needs  assessments  of  beneficiaries  suffering  from 
multiple  disadvantages  and  have  monitoring  data  available  on  the  participant  status  by  sex, 
disability  status  or  age.  Moreover,  the  same  recommendations  illustrated  under  "Effectiveness" 
should be taken into consideration to reinforce also the equity of the Programme.  

39 For additional information on the conclusions, see section 8 "Conclusions" in the Interim report, p. 156 et seq. 
40 For more information, see section 9 "Recommendations" in the Interim report, p. 161 et seq. 

33 

 
 
 
 
 
 
 
                                                            
Scope for simplification 

  The Commission will consider to provide more guidance to the applicants in the template Part B. 
Each item should be better explained and complemented with examples. In addition, the Guide for 
Evaluators could be published, to make the applicants aware of the evaluation methodology. The 
Commission will consider the possibility to introduce project standard costs, lump sums and flat 
rates and will consider the introduction of  a chat tool in the Participant Portal to help applicants 
and  grant  beneficiaries  to  receive  a  quick  feedback  from  the  Commission.  In  addition,  the 
monitoring system should envisage the possibility for the beneficiary to update in an easier way 
and more frequently the values of the indicators. 

34 

 
 
ANNEX 1: PROCEDURAL INFORMATION 

1.  LEAD DG, DeCIDE PLANNING/CWP REFERENCES 

Lead Directorate-General: Directorate-General for Justice and Consumers  

Decide planning reference:   

2.  ORGANISATION AND TIMING 

The evaluation was carried out between October 2017 and April 2018. 

An external contractor, Ernst & Young Financial-Business Advisors, was commissioned 
under the Framework Contract for evaluation, Impact Assessment and Related Services 
(JUST/2016/JCOO/FW/JU04/0178)  to  carry  out  a  interim  evaluation  of  the  Rights, 
Equality  and  Citizenship  Programme  (2014-2020).  The  final  report  was  completed  in 
April  2018.  An  inter  service  Steering  group  led  by  Directorate-General  for  Justice  and 
Consumers  was  set  up  specifically  for  this  evaluation.  The  representatives  of  the 
following Commission services were members of the group:  

Secretariat  General,  Directorate-General  for  Budget,  Directorate-General  for  Regional 
and Urban Policy, Directorate General for Education, Youth, Sport and Culture, Eurostat, 
Directorate-General 
for  Communications  Networks,  Content  and  Technology, 
Directorate-General  for  Employment,  Social  Affairs  and  Inclusion,  Directorate-General 
for  Migration  and  Home  Affairs,  Directorate-General  for  Health  and  Food  Safety,  and 
European Anti-Fraud Office.  

Chronology of the interim evaluation:  

Date 

 Meetings of Inter-service Steering Group 

January 2017 

Kick-off  meeting  of  the  mid-term  evaluation  Inter-Service 
Steering group 

December 2017 

Second  meeting  of  the  mid-term  evaluation  Inter-Service 
Steering group to discuss the interim report 

April 2018 

Third  meeting  of  the  mid-term  evaluation  Inter-Service 
Steering  group  to  discuss  the  draft  final  report  and  ask  for 
several improvements 

3.  EXCEPTIONS TO THE BETTER REGULATION GUIDELINES 

A public consultation was not carried out specifically under this evaluation, but under the 
Impact Assessment for the post-2020 Multiannual Financial Framework.  

4.  CONSULTATION OF THE REGULATORY SCRUTINY BOARD (IF APPLICABLE) 

N/A 

5.  EVIDENCE, SOURCES AND QUALITY 

The  evaluation  was  based  on  evidence  from  different  sources.  The  complete  set  of 
documents that were consulted for this evaluation is listed below: 

Legal basis of the Programme: 

35 

 
 
 
 
  Arts. 19(2), 21(2), 114, 168, 169, 197 TFUE; 

  Regulation (EU) No 1381/2013 of 17 December 2013 establishing a Rights, Equality 

and Citizenship Programme for the period 2014 to 2020 (OJ L 354, 28.12.2013). 

Commission Implementing Decisions: 

  Commission  Implementing  Decision  C(2014)2556  concerning  the  adoption    of  the 
work  programme  for  2014  and  the  financing  for  the  implementation  of  a  Rights, 
Equality and Citizenship Programme; 

  Commission  Implementing  Decision  C(2015)1997  concerning  the  adoption    of  the 
work  programme  for  2015  and  the  financing  for  the  implementation  of  a  Rights, 
Equality and Citizenship Programme; 

  Commission  Implementing  Decision  C(2016)1677  concerning  the  adoption    of  the 
work  programme  for  2016  and  the  financing  for  the  implementation  of  a  Rights, 
Equality and Citizenship Programme; 

  Commission  Implementing  Decision  C(2017)1316  concerning  the  adoption    of  the 
work  programme  for  2017  and  the  financing  for  the  implementation  of  a  Rights, 
Equality and Citizenship Programme. 

EU documents : 

  Policy-specific  reports  published  by  Directorate-General  for  Justice  and  Consumers, 
such  as  on  Roma,  gender  equality,  non-discrimination,  Fundamental  Rights  Agency 
and European Institute for Gender Equality 

  Policy-specific reports and resolutions by the European Parliament 

  Political and policy documents, including Council Conclusions by the Council of the 

EU, and by the EU Member States 

  Policy-specific reports published by other stakeholders. 

  Ex-post evaluation report of Fundamental Rights Programme (2007-2013)  

http://eur-lex.europa.eu/legal-
content/EN/TXT/?qid=1488141558521&uri=CELEX:52017DC0069  

  Ex-post evaluation report of Daphne Programme (2007-2013)  

http://eur-lex.europa.eu/legal-
content/EN/TXT/?qid=1488141666866&uri=CELEX:52017DC0055  

  Ex-post  evaluation  of  the  Programme  for  employment  and  social  solidarity  – 
PROGRESS  2007-2013  and  recommendations  for  the  successor  programmes  to 
PROGRESS 2014-2020 

 http://ec.europa.eu/social/BlobServlet?docId=12610&langId=en 

  The Commission Impact Assessment which accompanied the Commission proposal to 
establish  the  Rights,  Equality  and  Citizenship  Programme  2014-2020,  SEC(2011) 
1364 final 

http://eur-
lex.europa.eu/LexUriServ/LexUriServ.do?uri=SEC:2011:1364:FIN:EN:PDF  

  The  interim  evaluation  report  of  Fundamental  Rights  Programme,  2011  http://eur-

lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2011:0249:FIN:EN:PDF 

36 

 
  The interim evaluation report of Daphne Programme, 2011  

http://eur-lex.europa.eu/legal-
content/EN/TXT/PDF/?uri=CELEX:52011DC0254&from=EN 

  Policy-specific reports published by other DGs of the Commission, specifically by DG 

Regional and Urban Policy, and DG Employment, Social Affairs and Inclusion  

  Monitoring of and reporting by selected beneficiaries receiving operating grants, such 
as the European Judicial Training Network, the European network of equality bodies 
and Framework Partners.  

  European Citizenship Reports 

RIGHTS EQUALITY AND CITIZENSHIP   Programme documentation: 

  Programming  documents:  annual  monitoring  reports  (2014,  2015),  annual  work 
programmes  (2014-2017),  programme  statements  (2014-2018),  annual  management 
reports (2014-2017), annual activity reports (2014-2016) 

  Project  documents  (all  action  grants  and  operating  grants):  proposals,  grant 
agreements, interim and, if available, final project reports, project websites, proposals 
evaluations, projects evaluations, Guide for Applicants, Guide for Evaluators 

  Procurements: signed contracts, delivered goods and services 

  Responses to the surveys, interviews, and all other consultation activities  

  Analysis of public consultation  

  The Commission Impact Assessment which accompanied the Commission proposal to 

establish the Justice Programme 2014-2020 and the Rights, Equality and Citizenship   
Programme 2014-2020 

  EU Justice Agenda for 2020 (COM(2014) 144final 

   http://ec.europa.eu/justice/effective-justice/files/com_2014_144_en.pdf. 

37 

 
ANNEX 2: STAKEHOLDERS CONSULTATION 

Consultations  with  the  main  stakeholders  of  the  Rights,  Equality  and  Citizenship 
Programme were conducted over several steps: 

  A web-based survey targeting the following stakeholder groups:  

o  projects applicants; 

o  projects beneficiaries;  

o  unsuccessful applicants; 

o  EU28 Member States Programme Committee Members; 

o  Other relevant stakeholders involved directly by the Commission. 

This consultation was launched by the external consultant via the EU Survey tool and 
was open from 23 December 2017 until 6 March 2018. It received 173 responses41; 

  33 interviews with 9 Commission officials and 24 selected projects beneficiaries and 
representatives of target groups were performed between the 8th of January 2018 and 
the 26th  of  March  2018. Interviews had a twofold objective: 1. Collecting additional 
information (from the survey and the desk research) to feed the analysis alongside the 
evaluation  criteria,  filling  gaps  (especially  in  terms  of  quantification  of  costs  and 
benefits) and going more in-depth with specific aspects, such as the identification of 
best  practices;  3.  Gathering 
to  draw  conclusions  and 
recommendations on how to improve the design and functioning of the Programmes. 
Interviews consisted of semi-structured phone interviews. Questions were customised 
according to the different categories of stakeholders targeted, taking into account their 
different contribution to the evaluation questions; 

insight  and 

input 

  1  webinar,  organized  in  March  2018,  to  discuss  preliminary  findings  with  few 
selected project beneficiaries. The webinar aimed at assessing at a higher level results 
achieved  by  the  Programme,  testing  and  discussing  preliminary  findings  and 
conclusions, also identifying key success factors and areas for improvement in view of 
the design of operational recommendations. It was organised using Zoom (an online 
tool); 

  A  broad  public  consultation  undertook  by  the  Commission  in  the  area  of  "values 
and  mobility"  was  available  online  in  23  official  EU  languages  for  a  mandatory 
period between 10 January 2018 and 9 March 2018. The purpose of this consultation 
was  to  collect  the  views  of  all  interested  parties  on  how  to  make  the  most  of  every 
euro of the EU budget. Consultations have taken place in the context of evaluations of 
existing EU financial programmes covering several policy areas, including the Rights, 
Equality  and  Citizenship  Programme.  In  total,  the  public  consultation  received  1839 
replies from all over Europe. The respondents had experience with the following EU 
programmes:  1.  Europe  for  Citizens  Programme  and  /or  2.  Rights,  Equality  and 
Citizenship  Programme  and  /or  3.  Creative  Europe  Programme  and  /or  4.  Justice 
Programme. The results of this public consultation have shown that, according to the 
majority of respondents, "promote European identity and common values", as well 
as  "promote  rights  and  equality",  are  important  common  policy  challenges  to  be 

41  For  more  information  on  the  outcomes  of  the  web  survey,  see  Annex  2  "Analysis  of  surveys  and  quantitative  analysis"  in  the 
Interim report, ibid, p. 168 et seq. 

38 

 
 
                                                            
addressed in each of the four programmes. "Support active citizenship, democratic 
participation  in  society  and  the  rule  of  law"  and  "promote  social  inclusion  and 
fairness"  appear  to  be  important  challenges  to  be  addressed  in  the  concerned 
programmes  as  well.  Instead,  "support  innovation",  "foster  European  cultural 
diversity  and  cultural  heritage",  "promote  European  identity  and  common 
values"  are considered as  policies  that  fully  or  fairly  well  address  the  challenges  by 
half  or  more  of  the  respondents.  Around  80%  the  respondents  agree  that  these 
programmes  add  value  to  a  large  extent  or  to  a  fairly  good  extent  to  what  Member 
States  could  achieve  at  national,  regional  and/or  local  levels.  The  main  obstacles 
identified by the respondents, that could prevent the current programmes/funds from 
achieving  their  objectives,  are  very  similar  regardless  of  the  programme  concerned: 
"lack  of  budget  of  the  programmes  to  satisfy  demand",  "insufficient  support 
provided  to  small-scale  stakeholders"  and  "lack  of  support  to  first-time 
applicants" are identified as the main three obstacles. Finally, the respondents agree 
that  "the  use  of  more  simplified  application  forms",  "facilitating  structured 
network  and  partnerships",  "facilitating  funding  for  actions  cutting  across  the 
sectors  of  action",  as  well  as  "better  coordination  between  different 
programmes/funds",  are  the  main  steps  to  be  taken  to  simplify  and  reduce  the 
administrative burdens for beneficiaries. 

39 

 
 
 
 
 
  
ANNEX 3: METHODS 

The following methodological tools have been used:   

 1. Data collection42; 

  desk research of relevant documents (at programme, policy and project level); 

  field  research  relied  on  a  web-based  survey  with  relevant  stakeholders,  i.e. 
(unsuccessful)  project  applicants,  project  beneficiaries,  Programme  Committee 
members, Commission staff and additional relevant stakeholders, 33 interviews (with 
24  selected  project  beneficiaries  and  representatives  of  target  groups  and  9  with 
Commission officials) and one webinar to test and discuss preliminary findings and 
conclusions with few selected project beneficiaries. 

  9 case studies of projects connected to the specific objectives of the Programme43; 

  analysis of the results of a broad public consultation undertook by the Commission 
between 10 January and 8 March 2018 and done in the framework of the proposal for 
the post 2020 Multiannual Financial Framework in the area of "European values and 
mobility".  

2. Data analysis 

The contribution of the Rights, Equality and Citizenship Programme to its objectives was 
assessed in quantitative and qualitative terms.  

  quantitative data analysis44:  in  addition to monitoring and survey data,  descriptive 
statistics have been used to analyse the effectiveness, efficiency and EU added value 
of  the  Programme;  self-reported  counterfactual  analysis  have  been  used  for  the 
analysis  of  its  effectiveness  and  added  value45;  exploratory  data  analysis  and 
econometric  modelling  have  been  used  to  analyse  its  effectiveness  and  EU  added 
value46; cost-benefit analysis to analyse its efficiency47 and Social Network Analysis 
to analyse its effectiveness and equity48; 

  qualitative data analysis49: a "text mining" methodology50 and machine learning 

algorithms. 

Based  on  this  methodology,  output  indicators  (as  monitoring  data  at  project  level)  and 
results indicators (based on Eurobarometer and other sources) were used. The analysis of 
the performance at the project level has mainly relied on the answers to the survey.  

42 For more information, see section 5.3 "Data collection tools and activities" in the Interim report, p. 47 et seq. 
43 For more information, see the Annexe 4 of the Interim report, ibid. 
44 For more information, see section 5.3.1 "Quantitative data analysis" in the Interim report, p. 50 et seq. 
45 A counterfactual analysis is a tool that can be used to distil changes that are caused by a specific policy intervention from changes 
that would have occurred even in absence of the given intervention (see Interim report, ibid). 
46  Different  tools  of  econometric  analysis  have  been  employed  in  order  to  examine  the  results  of  the  self-reported  counterfactual 
questions, as well as to test the responses of different stakeholders to the main question of the survey linked to effectiveness/relevance 
(see Interim report, ibid). 
47  The  Cost-Benefit  analysis  is  as  an  economic  tool  applied  to  public  decision−making  to  quantify  the  advantages  (in  terms  of 
benefits) and disadvantages (in terms of costs) associated with a particular project or policy (see Interim report, ibid). 
48  The  Social  Network  analysis  enabled  to  better  understand  the  explicit  and  implicit  links  and  interrelations  among  the  different 
beneficiaries and hence the value of the partnership approach that is a distinguishing feature of the Programme (see Interim report, 
ibid). 
49 For more information, see section 5.3.2 "Qualitative data analysis through text mining" in the Interim report, p. 57 et seq. 
50  Text  mining  is  an  analytical  methodology  that  has  the  aim  of  structuring  large  amounts  of  unstructured  texts  into  categories  or 
clusters of information by broad themes (e.g. project objectives, weaknesses) (see Interim report, ibid). 

40