COMMISSION STAFF WORKING DOCUMENT EVALUATION Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS assessing the implementation and achievements of the 2014-2020 rights, equality and citizenship programme
Date: 2022-03-22
Institutions: Directorate-General for Justice and Consumers, European Commission
Subjects: EU programme, European citizenship, area of freedom, security and justice, children's rights, civil society, consumer protection, data protection, equal treatment, gender equality, human rights

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 22.3.2022  SWD(2022) 58 final  COMMISSION STAFF WORKING DOCUMENT  EVALUATION  Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS assessing the implementation and achievements of the 2014-2020 rights, equality and citizenship programme {COM(2022) 118 final} - {SWD(2022) 59 final}  
 
 
 Table of contents 

1. 

INTRODUCTION ....................................................................................................... 3 

Purpose and scope .............................................................................................................. 3 

2.  BACKGROUND TO THE INTERVENTION ........................................................... 5 

Description of the intervention and its objectives .............................................................. 5 

Baseline and points of comparison ................................................................................... 11 

3. 

IMPLEMENTATION / STATE OF PLAY .............................................................. 13 

Description of the current situation .................................................................................. 13 

Progress in indicators ....................................................................................................... 22 

4.  METHOD .................................................................................................................. 25 

Short description of methodology .................................................................................... 25 

Limitations and robustness of findings............................................................................. 28 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS ................ 29 

6.  CONCLUSIONS ....................................................................................................... 37 

ANNEX 1: PROCEDURAL INFORMATION ................................................................ 40 

ANNEX 2: STAKEHOLDER CONSULTATION ........................................................... 45 

ANNEX 3: METHODS AND ANALYTICAL MODELS ............................................... 47 

ANNEX 4: PROGRESS ON INDICATORS .................................................................... 54 

ANNEX 5: RESULTS OF THE CASE STUDIES ........................................................... 57 

1 

 
 
 
Glossary 

Term or acronym  Meaning or definition 

AG 

AWP 

CERV 

Action grant 

Annual work programme 

The citizens, equality, rights and values programme 

DG EMPL 

Directorate-General for Employment, Social Affairs and Inclusion 

DG JUST 

Directorate-General for Justice and Consumers 

LGBTI 

OG 

RCHI 

RCIT 

RCON 

RDAP 

RDAT 

RDIS 

REC 

RRAC 

RDIB 

RGEN 

UNICEF 

UNHCR 

lesbian, gay, bisexual, transgender and intersex 

Operating grant 

Specific objective - promoting the rights of the child 

Specific objective - exercise of rights deriving from EU citizenship 

Specific objective - consumers’ or entrepreneurs’ rights 

Specific objective - preventing and combating violence against children, young 
people and women, and other groups at risk 

Specific objective - protection of privacy and personal data 

Specific objective - promoting non-discrimination 

Rights, equality and citizenship programme 

Specific objective - preventing and combating racism, xenophobia, homophobia and 
other forms of intolerance 

Promoting the rights of persons with disabilities 

Promoting equality between women and men including through gender 
mainstreaming 

United Nations Children's Fund 

United Nations High Commissioner for Refugees 

2 

 
 
 
1.  INTRODUCTION 

Purpose and scope 

This  staff  working  document  accompanies  the  report  from  the  European  Commission  to  the 
European Parliament and the Council, in line with the obligation specified in Article 13.2(c) of the 
legal base of the rights, equality and citizenship programme (hereinafter, 'REC programme')1.  

The  Regulation  requires  the  European  Commission  to  provide  the  European  Parliament  and  the 
Council  with  an  ex  post  evaluation  report  for  the  programme  by  31 December 2021.  The  report 
must  assess  the  long-term  impact  and  the  sustainability  of  the  programme’s  effects,  in  order  to 
inform a decision on a subsequent programme. In 2021, however, a considerable number of projects 
(more  than  40%)  remained  ongoing  due  to  their  multiannual  life-span.  This  situation  was  also 
affected by the COVID pandemic, which resulted in the duration of several projects being extended. 
Therefore, it may only be possible to comprehensively evaluate the programme’s long-term results 
or impacts at a later stage. 

For these reasons, the ex post evaluation will be carried out in two parts. This report represents the 
first  part.  It  is  based  on  the  data  currently  available,  provides  an  overview  of  how  the  funding  is 
distributed  and  assesses  the  programme’s  achievements  to  date.  While  the  new  citizens,  equality, 
rights  and  values  programme  for  2021-2027  (hereinafter  ‘CERV  programme’)  has  already  been 
adopted2, based on, among other things, the results of the interim evaluation of the 2014-2020 REC 
programme,  this  first  part  of  the  ex  post  evaluation  of  the  previous  programme  will  be  of  added 
value in shedding light on potential areas for improvement as regards implementing the new CERV 
programme. 

The second part of the evaluation will be carried out at a later stage once all final data are available 
and in  conjunction with  the interim evaluation of the succeeding CERV  programme. This  second 
part  will  assess  the  long-term  impacts  and  the  sustainability  of  the  programme’s  effects.  It  will 
provide recommendations for the new multiannual financial framework after 2027, as appropriate. 

The  document  provides  an  overview  of  the  programme’s  objectives  and  the  needs  addressed,  the 
intervention  logic,  and  the  programme’s  implementation,  providing  a  backdrop  to  the  evaluative 
work.  It  then  provides  the  replies  to  the  evaluation  questions  on  the  effectiveness,  efficiency, 
coherence,  EU  added  value,  equity  and  scope  for  simplifying  the  programme.  Finally,  it  also 
explains  the  conclusions  of  the  analysis  of  eight  case  studies,  in  which  a  number  of  specific 
questions have been studied in detail. 

Although the analysis covered the call for proposals and procurement activities financed through all 
annual  work  programmes  from  2014  to  2020,  not  all  grant  agreements  and  contracts  had  already 

1  REGULATION  (EU)  No  1381/2013  OF  THE  EUROPEAN  PARLIAMENT  AND  OF  THE  COUNCIL  of  17 
December 2013 establishing a Rights, Equality and Citizenship Programme for the period 2014 to 2020. 
2 REGULATION (EU) 2021/692 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 28 April 2021 
establishing the Citizens, Equality, Rights and Values Programme and repealing Regulation (EU) No 1381/2013 of the 
European Parliament and of the Council and Council Regulation (EU) No 390/2014. 

3 

 
                                                           
been signed by the time the evaluation began. Therefore, a cut-off date of 31 December 2020 was 
set, so to define the scope of this evaluation. The second part of the ex post evaluation will complete 
the overview with the contracts and grant agreements signed after the cut-off date. 

In terms of geographical scope, the evaluation covers all the participating countries, namely all EU 
Member States and, when relevant, Liechtenstein, Iceland, and Serbia. 

4 

 
 
 
2.  BACKGROUND TO THE INTERVENTION 

Description of the intervention and its objectives 

The REC programme aimed at further developing a Europe of rights and equality in  line with the 
Europe 2020 strategy. It promoted the key values upon which the European Union is founded and 
helped further develop of an area where equality and the rights of people, as enshrined in the Treaty 
of  the  European  Union,  the  EU  Charter  of  Fundamental  Rights  and  international  human  rights 
conventions,  are  promoted,  protected  and  implemented  effectively.  The  EU’s  overall  financial 
contribution the programme over its duration (2014 to 2020) was EUR 439 473 000. 

When  the  REC  programme’s  focus  and  priorities  were  being  determined,  the  following  key 
problems and needs were identified:  

 

 

limited  effectiveness  in  the  implementation  of  actions  against  discrimination  on  the 
grounds  of  sex,  racial  or  ethnic  origin,  religion  or  belief,  disability,  age  or  sexual 
orientation, and more Roma integration and gender equality;  

insufficient  promotion  and  protection  of  the  rights  enshrined  in  the  EU  Charter  of 
Fundamental Rights, including those of the child and persons with disabilities, and in the 
UN Convention on the Rights of Persons with Disabilities;  

 

too  high  a  level  of  gender-based  violence  and  too  little  support  provided  to  victims  of 
violence;  

 

insufficient protection of EU citizens’ personal data;  

 

limited  knowledge  and  implementation  of  the  rights  deriving  from  EU  citizenship  and 
EU law. 

Specific objectives 

The  REC  programme  aimed  to  tackle  the  problems  explored  above.  It  attempted  this  challenging 
task by providing funding to help achieve the overall strategic objective of developing a Europe of 
freedom,  security,  and  justice  in  conjunction  with  the  Europe  2020  strategy.  In  particular,  it 
contributed to i) the EU justice agenda for 20203; ii) the 2016-2020 strategic engagement for gender 
equality4; iii) the European disability strategy5; iv) the list of actions by the Commission to advance 

3  COM(2014) 144 Communication from the Commission to the European Parliament, the Council and the European 

Economic and Social Committee and the Committee of the Regions, The EU Justice Agenda for 2020 - Strengthening 
Trust, Mobility and Growth within the Union:  
http://ec.europa.eu/justice/effective-justice/files/com_2014_144_en.pdf  
4  SWD(2015) 278 Strategic Engagement for Gender Equality 2016-2019:  

https://ec.europa.eu/anti-trafficking/sites/antitrafficking/files/strategic_engagement_for_gender_equality_en.pdf  

5  COM(2010) 636 European Disability Strategy 2010-2020: A Renewed Commitment to a Barrier-Free Europe: 

https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=LEGISSUM%3Aem0047  

5 

 
                                                           
LGBTI equality6; and v) the EU framework for national Roma integration strategies up to 20207. It 
also strengthened the implementation of EU law, such as the General Data Protection Regulation, 
the  Racial  Equality  Directive,  Equal  Treatment  Directives,  the  Work-life  Balance  Directive  and 
Victims’ Rights Directives.  

The  REC  programme  focused  on  the  following  nine  specific  objectives  addressing  fundamental 
rights. 

  Specific objective 1  –  ‘RDIS’: Promote the effective implementation of the principle of non-
discrimination on grounds of sex, racial or ethnic origin, religion or belief, disability, age or 
sexual orientation, and to respect the principle of non-discrimination on the grounds provided 
for in Article 21 of the Charter  

The programme supported projects aiming to prevent and combat discrimination on the grounds of 
sex,  racial  or  ethnic  origin,  religion  or  belief,  disability,  age  or  sexual  orientation.  In  particular, 
promoting  Roma  integration  is  a  key  aspect  of  this  specific  objective.  In  this  regard,  the  REC 
programme supported the national Roma platforms and worked closely with EU Member States and 
their  national  Roma  contact  points,  and with  civil  society organisations active  in the area of  non-
discrimination and Roma integration. The programme also financed the activities of the European 
network of equality bodies. This network’s core task is strengthening the cooperation and exchange 
of  information  on  topics  related  to  non-discrimination  between  the  national  equality  bodies 
designated by the Member States. 

  Specific  objective  2  –  ‘RRAC’:  Prevent  and  combat  racism,  xenophobia,  homophobia  and 

other forms of intolerance  

Projects financed in this area helped improve the implementation of existing EU legislation in the 
Member  States  and  assist  the  victims  of  hate  crime  and  hate  speech.  In  particular,  the  activities 
funded  supported  the  protection  of  minorities  against  a  surge  of  populism,  extremism  and 
intolerance. 

  Specific objective 3 – ‘RDIB’: Promote and protect the rights of persons with disabilities  

The  aim  of  this  specific  objective  was  to  increase  awareness  and  improve  implementation  of  the 
rights of people with disabilities leading to a reduction of barriers, their full participation in society 
and  enjoyment  of  their  rights.  This  was  mostly  done  through  operating  grants  supporting  the 
activities  of  European  level  networks  advocating  for  the  rights  of  people  with  disabilities  and 
through procurement activities focusing on data collection, training and awareness-raising activities. 
Under the 2014 annual work programme, the European Disability Card8 was launched with support 
from the programme to facilitate cross-border travelling for persons with disabilities. 

6  List of actions by the Commission to advance LGBTI equality (2015-2019): 

https://ec.europa.eu/info/sites/info/files/lgbti-actionlist-dg-just_en.pdf  

7  COM/2011/0173 An EU Framework for National Roma Integration Strategies up to 2020: 

https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1444910104414&uri=CELEX:52011DC0173  

8 For more info on the European Disability Card, see http://ec.europa.eu/social/main.jsp?catId=1139  

6 

 
                                                           
  Specific  objective  4  –  ‘RGEN’:  Promote  equality  between  women  and  men  and  to  advance 

gender mainstreaming 

The  programme  promoted  projects  to  support  participating  countries  and  relevant  stakeholders  in 
improving  gender  balance  in  all  spheres  of  life,  especially  in  economic  decision-making,  and  in 
promoting  equal  economic  independence  for  women  and  men.  Funding  under  this  objective  was 
also focused on reducing the gender gaps in pay, earnings and pensions.  

  Specific  objective  5  –  ‘RDAP’:  prevent  and  combat  all  forms  of  violence  against  children, 
young people and women, as well as violence against other groups at risk, in particular groups 
at risk of violence in close relationships, and to protect victims of such violence (DAPHNE) 

The  programme  primarily  financed  projects  which  helped  to  protect  and  support  victims  of 
violence, such as women and children, as well as activities to raise awareness and prevent violence. 
This  involved  engaging  with  civil  society  organisations  working  on  the  ground.  It  also  supported 
the  treatment  of  perpetrators  of  violence.  Strengthening  child  protection  systems  is  one  of  the 
Commission's  priorities  and  the  programme  supported  the  rights  of  migrant  children  by  funding 
international organisations working directly for and with these children.  

  Specific objective 6 – ‘RCHI’: Promote and protect the rights of the child  

The  rights  of  the  child  are  promoted  through  projects  focusing  on  child-friendly  justice  and  on 
protecting vulnerable children (such as those leaving foster care systems and those in conflict with 
the law), by supporting, in particular, training activities of civil servants, civil society organisations 
and lawyers dealing with vulnerable children. 

  Specific objective 7 – ‘RDAT’: contribute to ensuring the highest level of protection of privacy 

and personal data  

The  programme  is  the  main  source  of  EU  funding  for  data  protection.  The  activities  carried  out 
within  this  specific  objective  were  strongly  linked  with  the  data  protection  reform9,  adopted  in 
2016.  Under  this  objective,  the  programme  aimed  to  finance  activities  to  support  Member  States, 
specifically in transposing and implementing the new EU data protection legislation and in training 
national data protection authorities and data protection officers. 

  Specific  objective  8  –  ‘RCIT’:  promote  and  enhance  the  exercise  of  rights  deriving  from 

citizenship of the Union 

The  programme  financed  projects,  mostly  awareness-raising  activities,  aimed  at  including  EU 
citizens in the civic and political life of the EU.  It supported projects aimed at helping citizens to 
become  more  aware  of  their  rights  deriving  from  EU  citizenship.  Information  campaigns  on  EU 
citizenship  rights  were  financed  under  this  specific  objective,  with  a  particular  focus  on  their 
electoral rights in view of the 2019 European elections. 

  Specific  objective  9  –  ‘RCON’:  enable  individuals  in  their  capacity  as  consumers  or 
entrepreneurs in the internal market to enforce their rights deriving from Union law, having 
regard to the projects funded under the consumer programme 

9 For more info on the Data Protection Reform, see https://ec.europa.eu/info/law/law-topic/data-protection_en.  

7 

 
                                                           
The programme also financed activities which support individuals in their capacity as consumers or 
entrepreneurs in the internal market in order to enforce their rights deriving from EU law. EU funds 
in this area aim to help increase the knowledge and awareness of consumers' rights, especially in the 
digital market. 

Funding delivery mechanisms  

The  REC  programme  has  three  main  funding  mechanisms:  action  grants,  operating  grants  and 
procurement actions.  

  Action  grants  are  addressed  mainly  to  civil  society  organisations  and  Member  States’ 

authorities.  

  Operating grants (‘support to networks’) fund mainly European networks active in: i) non-
discrimination;  ii)  preventing  racism,  xenophobia,  homophobia  or  other  forms  of 
intolerance;  iii) disabilities;  iv) equality  between  women  and  men;  v)  preventing  and 
combating all forms of violence; and vi) promoting the rights of the child. 

  Procurement  actions  (‘Commission  initiatives’)  mostly  fund  conferences,  seminars, 

studies, surveys and awareness-raising activities. 

Expected outputs, results and impact 

The implementation of these activities was meant, in the short term, to lead to the following specific 
outputs: (i) training and capacity-building activities, conferences and seminars, mutual learning and 
exchange  of  good  practices,  which  can  also  be  measured  in  terms  of  the  respective  number  of 
participants;  (ii)  tools  being  developed  such  as  training  kits/material/curricula,  brochures, 
publications,  information  and  dissemination  material;  (iii)  analytical  studies;  (iv)  databases;  (v) 
policy recommendations. 

These outputs were expected to lead in the longer term to results, which can be linked back to the 
specific objectives identified above, namely: 

 

the  design,  development  and  monitoring  of  policies  aimed  at  tackling  discrimination  and 
promoting  equality;  and  the  legislation  on  non-discrimination  being  implemented  more 
effectively,  including  an  increased  number  of  national  Roma  platforms  which  are  actively 
engaged in activities of the national Roma contact points; 

  national and local authorities as well as civil society organisations and communities having 
increased  capacity  to  address  issues  related  to  racism,  xenophobia  and  other  forms  of 
intolerance, including hate speech over the internet; 

 

increased  awareness  and  better  implementation  of  the  rights  of  people  with  disabilities 
leading to barriers being reduced and these people being able to fully participate in society 
and enjoy their rights; 

8 

 
 

 

 

 

 

 

increased awareness of gender equality gaps  and their underlying mechanisms, and related 
policies  towards  equal  economic  independence  of  women  and  men,  also  in  regard  to 
tackling the pensions gap; 

improved support being provided to children, especially the most vulnerable, when they are 
preparing to leave care, and better protection being provided including support for the rights 
of all children in migration on EU territory; 

increased awareness and reporting of violence, with appropriate mechanisms of response in 
place; and improved protection and support standards for victims of gender-based violence 
between 
including 
against 
and 
European/national/regional/local authorities; 

strengthened 

cooperation 

children, 

violence 

increased awareness of data protection and increased capacity for national experts and civil 
society organisations; 

increased capacity  for national,  regional and local  experts to  enable mobile EU citizens to 
participate in their host Member States, including the exercise of their democratic rights; 

increased knowledge and awareness  of consumers’ rights,  especially in the digital  internal 
market. 

These results are instrumental  to  achieving the  specific objectives and  were expected to  promote, 
protect, and implement effectively the long-term equality and the rights of people. 

The intervention logic of the REC programme displays the causal links between the different levels. 

Figure 1. Intervention logic of the REC programme 

9 

 
 
Source: Interim evaluation of the REC programme  

Stakeholders and target groups  

The stakeholders eligible for support were public or private organisations (usually non-profit), duly 
established in one or more countries participating in the programme, or international organisations 
(such as the UN High Commissioner for Refugees, UNICEF and the International Organization for 
Migration).  

In  terms  of  target  groups,  intended  as  the  groups  that  can  benefit  directly  or  indirectly  from  the 
programme,  these  were  mainly  public  authorities  and  civil  society  organisations.  Target  groups 
were also  the  general population  of participating  countries, since  the  programme’s objectives  and 
initiatives aimed at promoting, protecting and implementing effectively the equality and the rights 
of  people,  in  particular  those  subjected  to  discrimination,  intolerance  or  violence,  including 
migrants and minorities. 

10 

 
Baseline and points of comparison  

The  2018  interim  evaluation  used  the  2011  impact  assessment10  and  the  2015  ex  post 
evaluation  of  the  three  predecessor  programmes  (Daphne  III,  fundamental  rights  and 
citizenship  and  progress)11  as  a  baseline  for  measuring  the  achievements  of  the  2014-
2020  REC  programme.  These  documents  will  also  remain  the  baseline  for  the  ex  post 
evaluation of this programme.  

The  impact  assessment  outlined  both  a  ‘baseline  scenario’  against  which  the  policy 
options for the REC programme were benchmarked, and a series of expected outcomes 
connected  to  the  programme’s  development.  The  baseline  analysis  showed  that,  if  no 
changes  had  been  made  between  the  2007-2013  and  2014-2020  programming  periods, 
the predecessor programmes would have continued to be successfully implemented, but 
with  reduced  potential.  The  following  issues  were  reported  as  critical  regarding 
continuation of the predecessor programmes: 

 

 

 

the  lack  of  flexibility  in  the  funding  instruments  would  have  represented  a 
difficulty given the pace of change and reform in the policy areas;  

fragmentation of funding, especially in an area where funding would be needed 
to  support  cross-cutting  issues,  could  have  reduced  the  capacity  of  the 
programmes to deliver results; 

the  high  number  of  different  funding  instruments,  in  the  context  of  rising 
the  administrative  burden, 
numbers  of  applicants,  could  have 
potentially resulting in delays in procedures for the beneficiaries.  

increased 

Given  these  difficulties,  the  2011  impact  assessment  suggested  the  option  of 
consolidating  the  three  programmes  in  the  new  comprehensive  REC  programme, 
outlining the following expected outcomes. 

  For  relevance  and  scope,  a  reduction  in  fragmentation  and  overlaps  was 
expected, as policies with close links to each other would be addressed under the 
same  programme.  In  addition,  the  broader  scope  of  the  REC  programme  was 
expected  to  bring  more  flexibility  in  supporting  policies  in  an  ever-evolving 
policy context.  

  For effectiveness, it was expected that the new comprehensive REC programme 
would  foster  concentration  of  funds  and  development  of  stronger  partnerships. 
The  applicants  would  coordinate  their  efforts  in  wider  partnerships  with  more 

10 SEC(2011) 1364 Final, ibid. 
11  European  Commission  (2015),  Ex-post  evaluation  of  five  programmes  implemented  under  the  2007-
2013 financial perspective – Final Report, ibid.  

11 

 
 
                                                           
funding available. A more balanced geographical spread among beneficiaries was 
also expected thanks to larger partnerships.  

  For  efficiency,  the  REC  programme  was  expected  to  improve    compared  with 
preceding  funding  instruments,  chiefly  by  simplifying  the  procedures  for 
applicants  and  beneficiaries.  In  particular,  merging  several  predecessor 
programmes  under  the  umbrella  of  the  REC  programme  was  expected  to  end 
‘programme-shopping’  among  applicants,  who  previously  had  to  identify  the 
most appropriate calls among six different funding instruments with diverse rules 
and procedures. Increases in efficiency were also expected thanks to the number 
of  calls  being  reduced,  which  was  expected  to  enable  them  to  become  more 
focused.  

  For  complementarity,  it  was  expected  that  merging  several  predecessor 
programmes into one would ensure a coordinated approach in identifying annual 
priorities,  to  be  applied  also  vis-à-vis  programmes  in  other  EU  policy  areas 
related  to  justice  and  rights.  Coordination  at  the  level  of  annual  programming 
would ensure that there is no duplication of funding and that funds are used in a 
complementary way, to the best interest of the public. 

 

In  terms  of  administrative  burden,  considerable  improvements  were  expected, 
in terms of reducing the number of annual work programmes and the number of 
calls. Harmonisation of the management procedures was also expected to reduce 
the administrative burden faced by applicants and beneficiaries. 

All the 2015 ex post evaluation reports on the previous funding programmes confirmed 
their  overall  effectiveness  and  highlighted  that  their  specific  objectives  and  priorities 
were largely specific, attainable and realistic, but not always measurable. The problems 
identified in 2011 were also confirmed for the three programmes. 

12 

 
 
 
3.  IMPLEMENTATION / STATE OF PLAY 

Description of the current situation  

Programme management 

The  REC  programme  was  implemented  via  direct  centralised  management  by  the 
European  Commission.  This  ensured  a  close  relationship  between  the  programme 
management and EU policy-making. This type of management allowed the Commission 
to tailor funded activities to policy priorities and needs and to directly target the relevant 
groups of stakeholders. It also enabled close contact with the programmes' beneficiaries 
and better knowledge of the needs on the ground.  

State of play  

The  purpose  of  this  section  is  to  present  the  state  of  play  on  the  programme’s 
implementation based on the information available at the time of writing. As explained in 
the introduction, a considerable number of projects are still ongoing, and therefore, the ex 
post evaluation will be carried out in two parts. The first part was being prepared in 2021, 
based  on  the  available  data  at  the  time,  and  it  will  assess  the  REC  programme’s 
achievements  so  far.  In  2024,  the  second  part  will  complete  the  evaluation  of  all  the 
projects  and  will  assess  the  long-term  impact  and  sustainability  of  the  programme’s 
effects.  

Annual work programmes 

Annual  work  programmes  over  the  programming  period  have  varied  their  focus  on 
specific  objectives  depending  on  the  wider  social,  political  and  economic  context.  This 
ensured the programme continued to be relevant so as to meet the challenges in the year 
that  followed,  such  as  the  migrant  crisis,  rising  hate  crime  and  hate  speech,  European 
elections or data protection reform.  

The first annual work programme of the programming period covered 2014 and, among 
other  priorities,  it  gave  special  focus  to  protecting  children  through  the  call  to  combat 
bullying  of  minors  in  public  institutions.  Furthermore,  following  the  Council’s 
Recommendation  2013/C 378/01  on effective Roma integration, a call  was launched to 
promote  Roma  integration,  with  continuous  focus  on  this  throughout  recent  years.  On 
combating  racism  and  intolerance,  activities  to  improve  monitoring  and  criminal 
proceedings were highlighted. Following the proposal for an EU disability card to make 
travel  for  people  with  disabilities  easier,  funds  were  provided  to  help  Member  States 
introduce  the  card.  Calls  to  fund  projects  that  involve  training  public  officials  were 
launched  in  the  context  of  the  tail  end  of  the  European  strategy  for  equality  between 
women and men.  

In  2015, specific  attention was  given to  preventing violence  against  children. The calls 
focused on preventing sexual violence against girls and young women in the workplace, 

13 

 
schools, and universities through new technologies. Under the same objective, a call was 
also  launched  to  help  fund  national  missing  children’s  hotlines.  In  addition,  calls  for 
proposals  were  launched  to  set  up  and  strengthen  a  national  consultation  process  by 
national  Roma  contact  points  following  the  Council’s  Recommendation  on  effective 
Roma integration mentioned above. A conference in the first half of the  year under the 
Latvian Presidency was funded to help implement the UN Convention on the Rights of 
Persons with Disabilities, in order to help achieve the specific objective of promoting and 
protecting the rights of people with disabilities.  

In  2016,  the  programme  focused  in  particular  on  the  ongoing  migration  crisis.  This 
involved  providing  direct  grants  to  the  International  Organization  for  Migration, 
UNICEF, UNHCR and Save the Children to finance capacity-building for professionals. 
Among  other  priorities,  the  annual  work  programme  aimed  at  raising  awareness  and 
training  professionals  in  light  of  the  Data  Protection  Reform  taking  effect  in  2018. 
Following the first Annual Fundamental Rights Colloquium, two calls for proposals were 
launched  to  address  hate  crime,  hate  speech  and  radicalisation,  with  a  focus  on 
antisemitism and islamophobia.  

The  2017  annual  work  programme  supported  the  national  data  protection  authorities  in 
their  efforts  to  raise  awareness  among  individuals  and  businesses,  in  particular  SMEs, 
about  the  entry  into  application  of  the  General  Data  Protection  Regulation  (GDPR)  in 
2018. It also provided for an EU communication campaign, carried out in 2017-2018, to 
raise  overall  awareness  of  the  GDPR.  In  2017,  there  was  also  increasing  concern  over 
hate crime and hate speech with particular attention being paid to antisemitism and anti-
Muslim  hatred  and  discrimination.  That  same  year,  there  was  a  call  for  action  grants 
aimed  at  supporting  the  Commission’s  LGBTI  list  of  actions  as  well  as  diversity 
promotion and Roma inclusivity. Two calls specifically sought to address gender-based 
violence  and  violence  against  children.  With  European  elections  taking  place  in  2019, 
there  were  also  calls  for  proposals  to  strengthen  democratic  participation  and  electoral 
rights.   

In  2018,  when  the  new  data  protection  legislative  framework  entered  in  force,  the 
funding  focused  on  helping  Member  States  to  successfully  implement  it.  With  the 
European elections due the following year, funding was allocated to projects promoting 
democratic participation  and awareness  of electoral  rights.  In 2018,  the following areas 
were  also  specifically  targeted:  protecting  minorities  and  countering  extremism, 
populism and intolerance. As a result, child protection funding was aimed specifically at 
migrant  children.  Countering  hate  speech  online  was  also  targeted.  Equal  opportunities 
for  women  and  men  were  promoted  in  public  roles  (politics,  corporate  sector,  or  civil 
society),  in  line  with  the  'New  Start  to  Support  Work-Life  Balance  for  Parents  and 
Carers'  initiative.  Particular  attention  was  paid  to  overseeing  the  national  Roma 
integration strategies through a restricted call for a national Roma platform.   

The  2019  annual  work  programme  highlighted  the  need  to  promote  the  Commission’s 
priority of ‘an area of Justice and Fundamental Rights based on mutual trust’. Data policy 

14 

 
was  promoted  through  the  framing  of  a  ‘Connected  Digital  Single  Market’  and  gender 
equality was promoted in  line with  boosting jobs and investment.  Support to  the Roma 
community was also highlighted, in particular through building national platforms. 2019 
was the first year in which the work programme offered a monetary prize for the winners 
of the Access City Award, which rewards efforts to increase urban accessibility to people 
with  disabilities.  The  application  of  the  new  General  Data  Protection  Regulation  since 
2018 was accompanied by a new wave of grants to national data protection authorities to 
help them reach out to individuals and businesses, in particular SMEs. Due to the 2019 
European  Parliament  elections,  special  attention  was  paid  to  ensuring  the  successful 
inclusion and political participation of all EU citizens. This involved funding a study on 
the elections and data gathering for the next EU citizenship report.   

In  2020,  particular  attention  was  paid  to  ensuring  children  were  included  in  decision-
making processes as well as on the ‘worrying increase of intolerance in our societies’. On 
the latter, the work programme sought to develop a common methodology on collecting 
data on hate crimes with continued special focus on antisemitism and anti-Muslim hatred 
and discrimination. Due to the Digital Content Directive being adopted in 2019, the 2020 
work programme focused on funding activities which helped Member States implement 
the  required  changes.  Focus  was  also  concentrated  on  promoting  the  rights  of  EU 
citizens. This included a new wave of grants to support the activities of the national data 
protection authorities in raising awareness on the GDPR. 

The  table  below  shows  the  budget  amount  per  annual  work  programme  amounting  to 
EUR 426.8 m for the entire programming period. The yearly planned amounts increased 
every year from EUR 54.2 m in 2014 to EUR 67.9 m in 2020.  

Table 1. Budget amount planned by year 

Budget year 

Amount planned (EUR) 

Annual increase (%) 

- 

4.0% 

4.5% 

6.2% 

-0.4% 

4.0% 

4.9% 

2014 

2015 

2016 

2017 

2018 

2019 

2020 

Total 

Source: Annual work programmes 2014-2020 

54 158 000 

56 323 637 

58 852 000 

62 515 000 

62 282 000 

64 771 000 

67 913 000 

426 814 637 

15 

 
 
 
 
Calls for proposals 

A  total  of  81  calls  for  proposals12  were  planned  in  the  2014-2020  annual  work 
programmes. The yearly number of calls peaked in the initial years of the period before 
dropping after 2017 as a result of being merged in the consolidation effort to increase the 
efficiency  of  the  programme’s  management.  Most  of  these  calls  covered  the  specific 
objective RDAP  -  ‘prevent  violence (Daphne)’,  with  18  calls.  As the  specific objective 
RCON - ‘Consumers' rights’ was funded through procurement only, there were no calls 
for proposals related to it. 

Table 2. Calls for proposals by specific objective and year 

Year 

2014 
2015 
2016 
2017 
2018 
2019 
2020 
Total 

RDIS 
Discrimi
nation 
2 
2 
2 
3 
2 
1 
2 
14 

RRAC 
Racism 

RDIB 
Disability 

3 
1 
2 
4 
3 
1 
1 
15 

2 
- 
1 
1 
- 
1 
- 
5 

Specific objectives 
RDAP 
Violence 

RCHI 
Children 

RGEN 
Gender 

RDAT 
Data 

RCIT 
Citizens 

2 
1 
1 
2 
1 
1 
1 
9 

5 
5 
3 
2 
1 
1 
1 
18 

3 
1 
1 
2 
1 
1 
1 
10 

- 
- 
1 
1 
- 
1 
1 
4 

1 
- 
1 
1 
1 
1 
1 
6 

RCON 
Consume
rs 
- 
- 
- 
- 
- 
- 
- 
- 

Tota
l 

18 
10 
12 
16 
9 
8 
8 
81 

Source: Annual work programmes 2014-2020 

The biggest share of the programme (25%) was planned for the specific objective RDAP 
-  ‘prevent  violence  (Daphne)’.  The  specific  objective  with  the  second  largest  budget 
allocation was that dedicated to promoting non-discrimination (22%).  

Table 3. Specific objectives and the amounts planned 

Specific objective  

Budget (EUR) 

Share  

Prevent violence against children, young people, women 
and other groups at risk (Daphne)   
To promote non-discrimination  

To prevent and combat racism, xenophobia, homophobia 
and other forms of intolerance  
Promote equality between women and men and gender 
mainstreaming  
Promote rights of persons with disabilities  

Promote the rights of the child  

109 835 157 

93 294 280 

56 321 000 

50 066 000 

44 034 000 

31 740 200 

25.73% 

21.86% 

13.20% 

11.73% 

10.32% 

7.44% 

12 This includes restricted all explicit ‘Call for proposals’ and ‘Calls for fixed-year Framework Partnership 
Agreements’ (operating grants), but excludes the direct grants, yearly calls for operating grants as well as 
procurement, prizes, and awards. 

16 

 
 
 
 
                                                           
To promote the rights deriving from EU citizenship  

Ensure the highest level of data protection  

Enforce consumer rights  

Total  

Source: annual work programmes 2014-2020 

19 571 000 

11 783 000 

10 170 000 

426 814 637 

4.59% 

2.76% 

2.38% 

100% 

For  the  specific  objectives  on  ‘rights  of  the  child’,  ‘prevent  violence  (Daphne)’,  ‘non-
discrimination’,  ‘racism  and  other  forms  of  intolerance’  and  ‘gender  equality’,  the 
biggest share of the funding was earmarked for action grants. For the remaining specific 
objectives (data protection, rights of persons with disabilities, EU citizenship, consumer 
rights)  the  activities  were  implemented  mainly  through  procurement.  Prizes  were 
introduced  in  the  final  2  years  of  the  REC  programme  and  were  used  for  the  specific 
objective on ‘rights of persons with disabilities’. 

Action grants and operating grants 

The  table  below  shows  the  number  of  action  grants  (AGs)  and  operating  grants  (OGs) 
awarded  by  specific  objective  and  year.  Overall,  942  AGs  and  OGs  were  awarded 
throughout the programme’s duration.  

Table 4. Number of AGs and OGs awarded by specific objective and by year 

Year 
2014 
2015 
2016 
2017 
2018 
2019 
2020 
Total 

RCON 
Consumers 
- 
- 
- 
- 
- 
- 
- 
- 

RDIB 
Disability 
8 
16 
8 
7 
7 
10 
7 
63 

RDAT 
Data 
- 
- 
6 
11 

4 
7 
28 

Specific objective 

RCIT 
Citizens 
6 
- 
9 
3 
5 
4 
4 
31 

RCHI 
Children 
13 
11 
11 
16 
7 
13 
11 
82 

RGEN 
Gender 
11 
10 
14 
15 
14 
10 
12 
86 

RRAC 
Racism 
9 
10 
16 
28 
24 
27 
25 
139 

RDIS 
Discrimination 
17 
31 
41 
38 
38 
45 
38 
248 

RDAP 
Violence 
31 
52 
27 
43 
43 
32 
37 
265 

Total 
95 
130 

132 
161 
138 
145 
141 
942 

Source: Data for 2014-2015 are retrieved from the interim evaluation of the REC programme. Data for 2016-2020 are 
retrieved from the project database (AG and OG excluding direct grants). 

Types of project activities 

Figure 2 provides an overview of the different types of activities implemented by funded 
projects.  Up  to  three  main  types  of  activities  could  be  flagged  for  each  project  2016-
2020.  Most  of  activities  funded  by  REC  projects  between  2016  and  2020  can  be 
classified as ‘training activities’ (35%).  

17 

 
 
Figure 2. Types of activities funded by year (2016-2020) 

120
100
80
60
40
20
0

2016

2017

2018

2019

2020

Others

ICT tools

Mutual learning

Analytical and monitoring activities

Awareness rising and dissemination

Training

Note: Projects can be assigned to more than one type of activity. 

Source: Project documents by DG JUST (AG and OG). 

Procurement 

Procurement  was  an  important  funding  mechanism  for  a  number  of  specific  objectives 
and amounted to 345 contracts over the entire period. The specific objective which most 
heavily relied on this funding option  was ‘Racism and other forms of intolerance’  with 
112 procurement contracts over the period.  

In  total,  approximately  EUR 81  million  was  allocated  to  procurement  contracts  and 
services.  Contracts  under  the  specific  objective  on  ‘non-discrimination’  accounted  for 
38% of the total amount. 

Table 5. Number of procurement contracts awarded by year and specific objective 

2014 

2015 

2016 

2017 

2018 

2019 

2020 

Grand total 

3 

1 

4 

3 

2 

RCHI 
Children 
RCIT 
Citizens 
RCON 
Consumers 
RDAP 
Violence 
RDAT 
Data 
RDIB 
Disability 
RDIS 
Discrimination 
RGEN 
Gender 
RRAC 
Racism 
Blank 

Grand total 

13 

1 

7 

10 

2 

6 

20 

6 

4 

56 

1 

1 

7 

1 

2 

5 

15 

5 

2 

3 

42 

5 

8 

3 

8 

7 

19 

4 

15 

1 

70 

2 

6 

5 

4 

9 

23 

5 

6 

1 

61 

4 

3 

3 

2 

5 

5 

16 

4 

3 

2 

47 

1 

7 

6 

1 

8 

16 

5 

12 

17 

32 

29 

9 

22 

44 

112 

31 

42 

7 

56 

345 

Source: Procurement contracts by DG JUST (n=345). 

18 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Type of procurement activities 

The  graph  below  details  the  distribution  of  procurement  activities  by  type.  The  most 
common  activity  was  categorised  as  ‘Analytical  and  monitoring’.  Many  contracts 
incorporated  more  than  one  of  these  components;  for  example,  many  would  have  had 
both ‘Analytical and monitoring’ and ‘Awareness-raising’ activities.  

Figure 3. Number of procurement contract signed by type (2014-2020)  

Analytical and monitoring
Awareness raising
Mutual learning
ICT
Training

31

19

20

0

98

80

77

40

60

80

100

120

Source: Procurement data by DG JUST (n=305). For 40 procurement contracts managed by DG EMPL, no 
information is available on activities and are therefore not included. 

Applications by Member States 

The  country  with  the  most  grant  applications  was  Italy  with  975  applications 
(coordinator)  over  the  programming  period.  This  was  by  far  the  largest  number  of 
applications from any country, with Spain in second place (353 applications).  

Figure 4. Number of applications by participating states (2014-2020) 

Source: Interim evaluation (2014-2015) and project database (2016-2020) 

However,  once  the  population  size  of  participating  countries  is  taken  into  account  (by 
looking  at  applications  per  100 000  people),  some  smaller  countries  do  actually  have  a 
large  number  of  applications  compared  to  their  populations.  Cyprus  stands  out  in  this 
regard  with  the  highest  ratio  of  applications  to  population  (6.31  per  100  000  people). 
Malta and Slovenia are second and third with 4.28 and 3.48, respectively.  

Distribution of resources by type of beneficiary  

19 

 
 
 
Non-profit  organisations  accounted  for  the  majority  (57%)  of  beneficiaries  throughout 
2016-2020  (Figure  5  –  OTH).  In  this  period,  the  share  of  beneficiaries  in  the  research 
sector  (REC)  fell  from  12%  in  2016  to  9%  in  2020.  The  share  of  higher  or  secondary 
education sector (HES) also fell from 14% in 2016 to 8% in 2020. This was mirrored by 
a slight increase of the share of private for-profit companies (PRC). Overall, the annual 
changes remained insignificant and did not affect the general trend.  

Figure 5 - Share of resources by type of beneficiary (2016-2020) 

100%

80%

60%

40%

20%

0%

2016

2017

2018

2019

2020

OTH

HES

PUB

REC

PRC

Source: monitoring data by DG JUST   

Target groups  

In  2014-2015  the  most  mentioned  target  group  was  young  people  (including  children), 
followed by women and students. If we look at the distribution of target groups in 2016-
2020,  their  granularity  increases  compared  to  2014-2015.  In  2016-2020  overall, 
professionals  are  the  most  mentioned  target  group,  followed  by  women  and  children. 
Citizens, young people and communities appear also to be important target groups for the 
REC programme in those same years. 
Figure 6.  Distribution of target groups by specific objective 

2016-2020 

2014-2015 

Source: Interim evaluation and text mining on project documents. 

20 

 
 
 
 
 
 
 
 
 
 
 
Note: Two specific objectives, RCIT and RDAT, were excluded from the representation as their representation is too 
disaggregated. 

21 

 
 
Progress in indicators 

Legal base indicators  

Article  14  of  the  REC  Regulation  lists  indicators  which  will  serve  as  a  basis  for 
monitoring and evaluating. Assessing the extent to which the legal basis indicators have 
been met is hindered by the difficulties in calculating them and by the lack of a baseline 
or benchmark against which to measure progress. In many cases, the conclusion on these 
indicators has been reached indirectly and based on partial information. The informative 
value of these figures, therefore, should be taken with caution.  

(a) The  number  and  percentage  of  persons  in  a  target  group  reached  by  the 

awareness-raising activities funded by the programme 

This  indicator  provides  information  on  the  number  of  people  reached  by  awareness-
raising activities funded by the programme. It should also provide information about the 
coverage  of  such  activities  with  respect  to  the  reference  population.  Awareness-raising 
activities are a key output of the REC programme. They play a key role in empowering 
target groups as they help them gain a better knowledge of their own rights and provide 
them with basic tools to report and fight any violation of such rights. Awareness-raising 
activities are also key to promoting awareness of fundamental EU values and changes of 
attitudes in society, particularly when directed at tackling stereotypes. 

Analysis  of  211  project  documents  (out  of  more  than  900  projects  funded  by  the 
programme) shows that, between 2016 and 2020, at least 80 million people were reached 
by  information  and  dissemination  campaigns  or  took  part  in  awareness-raising  events. 
This figure includes various target  groups, of which the most numerous are the  general 
public (ca. 46 million); followed by EU citizens (ca. 19 million); women (ca. 4 million); 
and  the  scientific  community,  civil  society,  policymakers,  media  (ca.  3  million); 
individuals  belonging  to  the  LGBTI  community  (ca.  1  million)  and  who  represent 
businesses/companies (ca. 1 million). 

(b) The  number  of  stakeholders  participating  in,  inter  alia,  training  activities, 
exchanges, study visits, workshops and seminars funded by the programme 

This  indicator  provides  an  overview  of  how  the  programme  helped  increase  the 
capacities  of  stakeholders,  including  private  and  public  authorities  and  professionals 
working with the target groups and advancing their rights. As such, it provides important 
information  about  the  programme’s  capacity  to  help  improve  the  quality  of  support 
services aimed at target groups. 

The analysis of 211 projects shows that between 2016 and 202013 approximately 85 000 
individuals  participated  in  exchange  and  mutual  learning  events.  In  addition,  the 

13  The  analysis  counts  only  finalised  projects  and  thus  presumably  covers  projects  which  started,  at  the 
latest, in the first quarter of 2019. 

22 

 
                                                           
documents  report  that  approximately  1.5  million  individuals  took  part  in  training 
activities.  

(c)  The  improvement  in  the  level  of  knowledge  of  Union  law  and  policies  and, 
where applicable, of rights, values and principles underpinning the Union, in 
the groups participating in activities funded by the programme compared to 
with the entire target group 

This  indicator  assesses  how  the  REC  programme  has  helped  increase  the  level  of 
knowledge of EU law and policies, another key goal of the REC programme. 

Based on the evidence at hand, when looking at the key types of results reported by the 
projects,  stakeholders  and  end  beneficiaries  increasing  their  knowledge  of  EU  law 
and policies is by far the most reported result, mentioned by 78% of projects for which 
a  technical  report  was  available.  This  goes  in  line  with  the  programme’s  intervention 
logic: it intends to trigger a behavioural change in society, for which awareness-raising 
activities are crucial.   

(d) The number of cases, activities, and outputs of cross-border cooperation 

Cross-border cooperation is a distinctive feature of the REC programme, with most calls 
and projects being directed at transnational consortia and actions. Only a limited number 
of calls did not require transnational partnerships.  

A  total  of  645  cross-border  tools  and  mechanisms  were  sustained  after  the  projects 
ended, which include formal advisory groups, memorandums of cooperation/cooperation 
agreements, and networks. The results obtained in this area and the strengths reported in 
the REC programme are similar to those obtained from predecessor programmes. 

(e)  Participants'  assessment  of  the  activities  in  which  they  participated  and  of 

their (expected) sustainability 

It is not possible to  calculate this indicator due to  the lack of a systematic reporting by 
projects on this theme.  

(f)  The geographical coverage of the activities funded by the programme, (g) the 
number of applications and grants related to each specific objective, and (h) 
the level of funding requested by applicants and granted in relation to each 
specific objective 

During  its  implementation  period,  the  REC  programme  received  4903  applications 
corresponding to 5.4 applications per grant  awarded. Overall, the average success  rate 
of  receiving  a  grant  in  the  REC  programme  was  17%.  Comparing  relevant  specific 
objectives  with  their  predecessors,  the  success  rate  of  ‘prevent  violence’  (Daphne)  was 
14.3%,  lower  than  the  success  rate  of  the  Daphne  III  programme  (16%).  On  the  other 
hand,  the  fundamental  rights  and  citizenship  programme  had  a  much  higher  number  of 
applications per project granted. Consequently, its success rate ranged between 0.4% and 
2%  and  was  much  lower  than  the  success  range  of  relevant  specific  objectives  of  the 

23 

 
REC  programme:  rights  of  the  child,  racism  and  other  forms  of  intolerance,  EU 
citizenship, and data protection.  

The specific objectives ‘rights of persons with disabilities’ and ‘data protection’ recorded 
the highest success rates (40% and 37%, respectively). ‘Gender equality’ followed with 
28%.  

Applications were not equally distributed across the Member States. The balance of the 
REC  programme’s  resources  and  activities  were  skewed  towards  a  few  countries  that 
made many applications. Nevertheless, the operating grants, which mainly fund EU-wide 
networks, guarantee fair geographic coverage at EU level.   

Indicators  for  the  REC  programme’s  general  objective  and  the  specific 
objectives  

Along  with  the  indicators  of  the  legal  base,  a  number  of  additional  indicators  were 
established,  which  aimed  to  measure  the  progress  of  the  programme  from  the  baseline 
situation.  Annex  4  presents  the  2020  target,  the  baseline,  the  milestone  and  the  actual 
results for each indicator, where available.   

Although the data is only partially available at this point in time, achieving most of the 
indicators’  2020  targets  seems  challenging  as  regards  the  programme’s  general 
objective14, given that Europe is exiting a long financial recession, exacerbated in 2020 
by  the  COVID-19  crisis.  Only  the  target  for  the  indicator  on  ‘the  percentage  of 
Europeans who consider themselves as “well or “very well” informed of the rights they 
enjoy as citizens of the Union’ is presumed to be achieved, well before 2020. 

Several indicators have been used to evaluate the programme’s performance at the level 
of individual specific objectives. Indicators are a valuable source of information, despite 
the bias and the external factors influencing the results. The data available so far indicate 
that the targets for several indicators have been achieved or exceeded. However, over 
the period of the programme it was proven difficult to collect  data for some indicators. 
These shortcomings were addressed when the performance monitoring framework for the 
new CERV programme was designed. 

14  ‘To  contribute  to  the  further  development  of  an  area,  where  equality  and  the  rights  of  persons  as 
enshrined in the Treaty on European Union, in the Treaty on the Functioning of the European Union, in the 
Charter of Fundamental Rights of the European Union and in the international human rights conventions to 
which the Union has acceded, are promoted, protected and effectively implemented’.  

24 

 
 
 
                                                           
 
4.  METHOD 

Short description of methodology 

During  the  evaluation  process  for  this  first  part  of  the  ex  post  evaluation  of  the  REC 
programme,  the  Commission  was  supported  by  a  consortium  of  contractors  (Ernst  & 
Young Consulting BV/SRL and Fondazione Giacomo Brodolini Foundation S.r.l. SB). 

This  assessment  is  based  on  a  mixed  methods  approach.  To  this  end,  the  evaluation 
integrates  and  compares  quantitative  and  qualitative  approaches,  data  collection,  data 
analysis and interpretation. Its purpose is to strengthen the reliability of data, and validity 
of  the  findings  and  recommendations.  It  does  this  by  triangulating  multiple  sources  of 
information, to broaden and deepen the understanding of the processes through which the 
programme’s  outcomes  are  achieved,  and  how  these  are  affected  by  the  context  within 
which the programme is implemented. 

Ultimately  it  provides  answers  to  the  evaluation  questions  on  effectiveness,  efficiency, 
relevance,  coherence,  EU  added  value,  equity  and  simplification  as  well  as  provides 
results on the eight case studies.  

The evaluation relied on both desk15 and field research. 

The field research aimed at collecting relevant data and input from project beneficiaries 
and  stakeholders,  filling  gaps  and  validating  information  collected  during  the  desk 
research. 

The field research focused on different categories of stakeholders, namely: 

  Beneficiaries of the support provided under the programme, including i) national, 
regional  and  local  authorities  in  all  participating  countries,  ii) international 
organisations, iii) EU-level umbrella organisations, iv) civil society organisations, 
v) academic and research institutions, vi) networks, vii) representatives of target 
groups  and  other  organisations;  in  some  instances,  potential  beneficiaries  were 
also consulted; 

  Commission staff (such as programme managers and policy officers); 

  The REC programme committee members representing participating countries. 

Field research relied on the following techniques:  

  semi-structured interviews (in total, 53 interviews were carried out); 

 

focus groups (4 focus groups were organised); 

15 For the list of documents used during the desk research, see Annex 1. 

25 

 
                                                           
  E-Survey  to  programme  committee  members  (17  responses  in  total).Additional 

survey to beneficiaries to feed the case studies. 

Table 6. Distribution of stakeholders consulted 

Coverage/stakeholders 

Number 

Type of activity 

Scoping interviews 

Interviews 

Interviews with beneficiaries 

Programme managers and other Commission officers 

Policy officers (covering the different specific objectives) 

Action grants 
Operating grants 
Contractors 

Focus groups with (potential) 
beneficiaries 

(potential) Beneficiaries 

Focus group with DG JUST 
project officers 

DG JUST project officers 

Web-based survey 

Programme committee members 

Projects  sampled  that  benefited  from  non-EU  funding  (Case 
study 4) and that experienced under-spending (Case study 5) 

5 

7 

37 

21 

4 

17 

This  evaluation  covered  the  call  for  proposals  and  procurement  activities  financed  in 
2014-2020 with a cut-off date of 31 December 2020 for signature of the grant agreements 
and the contracts. The second part of the ex post evaluation will complete the overview 
with the contracts and grant agreements signed after the cut-off date. 

The collected data were used to carry out a series of quantitative and qualitative analyses. 
On  the  quantitative  side,  the  analysis  helped  to  provide  perspective  both  at  the 
programme  level,  assessing,  for  example,  how  and  where  the  programme  deployed  the 
available  financial  resources,  and  at  the  project  level,  giving  a  granular  view  of 
quantitative  indicators  provided  by  individual  projects.  Concerning  the  qualitative 
analysis,  automated  text  mining  and  mixed  techniques  were  deployed  to  help  sift 
through the large amounts of data on individual projects. This was done because of the 
need to  give structure and find meaning  within a large and mostly unstructured dataset 
(namely project applications and final reports). 

Following the collection, extraction, and systematisation of the project-related data and 
documents,  additional  data  collection  activities  took  place  to  feed  the  response  to  the 
evaluation questions. This included field research on a sample of projects.  

A  total  of  36  projects,  beneficiaries  both  action  (28)  and  operating  grants  (8),  plus  six  
procurement contracts, were sampled in order to carry out a qualitative analysis based on 
comments  and  insights  from  key  stakeholders  concerning  the  REC  programme’s 
implementation.  The  projects  were  sampled 
into  account  geographical 
distribution, specific objective, and financial amount.  

taking 

26 

 
 
 
 
Table 7. Projects sampled by specific objective and form of support 

  Specific objective 

Action 
grants 

Operating 
grants 

Procur ement  To tal 

Combat racism, xenophobia, 
homophobia and  other forms of 
intolerance 

Enforce consumer rights 

Ensure the highest level of data 
protection 

Prevent violence against children, 
young people, women and other 
groups at risk 

Promote equality between women 
and men, and gender mainstreaming 

Promote non-discrimination 

Promote rights of persons with 
disabilities 

Promote the rights 
deriving from Union 
citizenship 

Promote the rights of the child 

TOTAL 

Case studies 

4 

- 

2 

8 

3 

6 

- 

2 

3 

28 

1 

- 

- 

1 

1 

2 

2 

- 

1 

8 

- 

1 

- 

0 

1 

2 

1 

1 

- 

6 

5 

1 

2 

9 

5 

10 

3 

3 

4 

42 

The main purpose of the case studies was to provide solid, triangulated evidence, in order 
to  support  preparation  of  replies  to  evaluation  questions  as  well  as  suggestions  for 
improvements, with a focus on specific issues mentioned in the tender specifications. For 
each case study a customised methodological approach was developed. Please see Annex 
3 for an overview of the focus of the eight case studies and Annex 5 for the results.  

27 

 
 
 
 
Limitations and robustness of findings 

The  evaluation’s  findings  and  conclusions  are  deemed  to  be  credible  and  consistent 
according  to  the  methodology.  There  were,  however,  some  challenges  which  could  not 
always be mitigated.  

  The  change  in  the  project  information  base  and  migration  to  eGrants  in  the 
middle  of  the  programming  period  led  to  differences  in  completeness  and 
consistency  of  the  data  used  for  the  monitoring  of  the  programme.  While  a 
detailed  and  extensive  database  has  existed  for  the  projects  since  2016, 
information on projects that ran before 2016 is  fragmented.  In certain cases, the 
indicators could be computed only  for projects granted from 2016 onwards, due 
to  unavailability  of  information  before  2016.  To  overcome  these  hurdles,  the 
contractors  worked  on  linking  the  different  data  sources  (for  funded  projects), 
assessing  what  represents  the  common  ‘nucleus’  of  information  that  is  shared 
between them, and creating a unique dataset of projects. 

  Weaknesses in the existing monitoring framework have been identified as the 
main  limitation  in  relation  to  assessing  the  programme’s  effectiveness.  To 
overcome  this  problem,  in  many  cases,  the  conclusion  on  indicators  had  to  be 
reached  indirectly  and  with  additional  input  from  interviews  and  project 
document analysis. 

  There was a limited response rate to the e-surveys launched as part of the data 
collection activities. To address low response rates in surveys, the response time 
was prolonged. 

  There  were  issues  surrounding  the  feasibility  of  using  readily  available 
information.  When  automatic  data  extraction  techniques  were  applied  (to  semi-
structured information, e.g. in the case of monitoring information reported in the 
annexes  to  the  technical  reports),  the  format  in  which  data  were  reported 
proved sometimes to be problematic for the extraction. 

28 

 
 
 
 
5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS 

Effectiveness  

Based  on  qualitative  evidence  collected,  the  REC  programme’s  implementation  was 
effective  and  this  helped  to  achieve  its  overall  objectives.  In  particular,  this  success 
was  thanks  to  the  quality  of  most  of  the  projects  implemented  with  REC  funding, 
enabling  targeted  actions  to  be  implemented  such  as  awareness  campaigns,  training 
activities, and publication of materials, research and relevant data.  

Progress  was  also  recorded  in  terms  of  the  legal  base  indicators  (identified  by  the 
Regulation), concerning in particular: 

 

 

the  number  of  people  that  were  reached  by  information  and  dissemination 
campaigns  or  that  took  part  in  awareness-raising  events  (at  least  80  million 
people);  

the number of stakeholders participating in training activities (at least 1.5 million 
individuals)  and  in  exchanges,  study  visits,  workshops  and  seminars  (at  least 
85 000 individuals) funded by the programme. 

Overall,  the  evidence  gathered  suggests  that  the  REC  programme  has  helped  increase 
the knowledge of EU soft and hard law.  

Key activities implemented match to a large extent the specific objectives. Furthermore, 
the activities and achievements of all projects sampled were directly relevant to the calls’ 
objectives. The calls’ priorities helped the beneficiary organisations to align their actions 
with  the  REC  programme’s  objectives.  At  the  same  time,  difficulties  were  also 
highlighted in identifying projects with the most promising impact, as some organisations 
were  adept  at  writing  proposals  and  passing  the  selection  process  without  necessarily 
being best placed to deliver on the expected results. 

Overall, both desk research and fieldwork show that the projects were largely successful 
in  implementing  the  planned  activities.  Qualitative  information  collected  shows  that 
almost all finalised projects have achieved the expected results.  

29 

 
Figure  7.  Comparison  between  expected  and  achieved  results  by  specific  objective

Increased compliance with EU treaties and charters

Creation of transnational networks/increased transnational cooperation

Development of policy recommendations, incl. monitoring and evaluation of policies

Exchange and transfer of good practices and mutual learning

Development of research tools and analyses (eg indexes, ranking, comparative research)

Increased support/empowerment to target groups and stakeholder engagement

Increased effectiveness and quality of services

Increased knowledge and awareness of stakeholders and end beneficiaries

0

20

40

60

80

100 120 140 160 180 200

Reported

Expected

Source: Project documents 2016-2020. 

Note: The result categories are based on the main typologies of results reported and on the main expected results of 
the REC programme16. 

A  complex  interplay  of  internal  and  external  factors  supported  and  hindered  the 
programme’s  performance.  Internal  factors  mainly  relate  to  the  difficulties  linked  with 
the application process, internal capacity of project partners, and the type and quality of 
consortium. Common external factors relate to the impact of external shocks such as the 
COVID-19 pandemic and the challenge of involving public entities and stakeholders in 
the activities planned.  

Figure 8. External factors that influenced the implementation of the funded projects 
(sample) 

50

40

30

20

10

0

Covid-19

Political instability / changes
occurred at political level

Cultural problems / language
barrier

 Lack of trust of end
beneficiary

Source: Project documents 2016-2020.  

16 Art 4 (2 a, b, c, d) of the REC Regulation. 

30 

 
  
 
 
                                                           
Figure 9. Internal factors that influenced the implementation of the funded projects 
(sample) 

Lack of management skills or other organisational difficulties

19

Lack of cooperation among one or more partners

13

Insufficient resources

Lack of interest/participation from target groups/stakeholders

Lack of political commitment/ commitment of public institutions /…

Administrative burden (eg technical and financial reporting,…

3

3

7

5

Other

0

2

4

6

8

10

12

14

15

16

18

20

Source: Project documents 2016-2020.  

The  evaluation  found  that  the  success  factors  in  REC  projects  include  the  quality  of 
partnerships,  bringing  together  organisations  with  complementary  expertise,  covering 
managerial and coordination skills with experience of working with target groups on the 
ground. Another element of success is the capacity of the projects to respond to the aims 
of  the  calls  while  also  pursuing  the  longer-term  strategies  of  the  implementing 
organisations.  

According to the data collected, the application process was communicated effectively 
to  applicants.  It  was  complex  but  fair,  even  though  several  factors  hindering  the 
participation of certain types of beneficiaries (mainly public institutions) were detected, 
such as low administrative capacity, lack of experience in applying for and implementing 
EU-funded projects, and difficulties in covering the resources required for co-financing.  

Funding a second (continuation) project with the same beneficiary organisations (case 
study  1)17  does  create  significant  synergies  in  terms  of  ensuring  sustainability, 
geographical spread and improving the way REC funded activities are implemented. At 
the same time, the programme needs to address the potential risk of closing the door to 
new  applicants  and  restricting  the  space  for  new  inputs  and  perspectives.  This  is 
particularly important since the application process may lead to the creation of barriers to 
participation in the REC programme as the process could be perceived as being resource 
intensive, competitive, and skewed by the capacity to ‘tick the boxes’. 

Efficiency 

Existing  qualitative  evidence  demonstrates 
the  benefits  of  projects’ 
implementation  outweigh  the  costs  and,  in  this  sense,  efficiency  was  achieved  (case 

that 

17 For more details, see Annex 5. 

31 

 
 
 
                                                           
study 4)18. However, the costs linked to the administrative burden are still considered an 
issue by beneficiaries, especially in relation to the application and reporting procedures. 
While  most  beneficiaries  identified  the  administrative  burden  as  a  significant  cost 
component,  they  also  recognised  that  burdensome  administrative  requirements  are 
legitimate when transparency and fairness in the use of public funding is at stake.  

REC administrative costs are perceived on average to be higher compared with other 
non-EU  alternatives  or  with  national  programmes.  National  and  international  funding 
instruments  are  considered  to  be  less  demanding  when  it  comes  to  the  application  and 
reporting requirements. 

In  terms  of  benefits,  the  REC  programme  offers  the  possibility  of  working  on  a 
transnational  level,  sharing  best  practices  and  building  relationships  with  consortium 
partners,  and  strengthening  internal  capacity  and  expertise.  There  are  also  intangible 
gains such as a better reputation and increased trustworthiness of their organisation as 
a result of participation in EU-funded projects.  

Key aspects that contributed to the projects’ efficiency included: 

  good planning and organisation, especially financial planning; 

  anticipation  of  any  over  or  underspending  in  order  to  take  prompt  corrective 

steps; 

 

 

the in-house experience and capacity of participating organisations; 

the  support  received  from  the  Commission,  including  the  possibility  for 
reallocating funds.  

The evaluation identified underspending in  82% of the projects sampled (case study 
519).  While  underspending  is  widespread  in  the  sample  analysed,  most  projects 
interviewed  reported  underspending  of  around  10-12%,  which  was  mainly  due  to  the 
COVID-19  outbreak  and  the  consequent  halt  in  physical  meetings  and  activities  which 
resulted  in  savings  in  terms  of  travel  and  subsistence  costs  for  physical  meetings, 
workshops and other events. 

Compared  to  its  predecessors  (case  study  620),  there  has  been  an  improvement  in  the 
prompt  adoption/publication  of  calls  published  under  the  REC  programme  and  the  
predictability  of  these.  Information  collected  through  interviews  with  recurring 
beneficiaries  shows  that  the  merging  of  the  predecessor  programmes  considerably 
simplified the management and implementation procedures. 

18 For more details, see Annex 5. 
19 For more details, see Annex 5. 
20 For more details, see Annex 5. 

32 

 
 
                                                           
Relevance 

The needs  identified at  the time the REC  programme  was  adopted are  still  relevant  to 
ensuring that equality and the rights enshrined in the EU Charter of Fundamental Rights 
and other legally binding instruments continue to being promoted across the EU. This is 
supported  by  the  fact  that  the  programme  receives  many  more  applications  than  it  can 
fund, showing its continued relevance to the needs of its target groups. Furthermore, the 
REC  programme  throughout  the  years  has  continued  to  evolve  in  order  to  adapt  to  the 
emergence  of  new  needs,  such  as,  for  example,  the  need  to  tackle  the  increase  in  hate 
speech. 

The analysis also confirmed that the actions implemented under the REC programme are 
relevant to the needs of its stakeholders, such as improving the knowledge base, sharing 
best  practices  through  mutual  learning,  capacity  building  and  structural  support  for 
specific  organisational  structures.  Furthermore,  the  REC  programme  was  relevant  in 
targeting the intended end beneficiaries – ultimately EU public. This has been confirmed 
through  several  strands  of  evaluation  evidence,  such  as  the  text  mining  analysis,  a 
sampling of projects and stakeholder consultations.  

The analysis of the overview of the main target groups according to needs addressed is 
presented  in  the  Figure  below.  More  than  one  target  group  and  need  can  be  associated 
with each project. Figures below show the links between target groups and needs covered 
by projects.  

Figure 10. Matching needs and target groups (2016-2020) 

Source: Text mining on 295 projects with available documentation.  

33 

 
 
In terms of the programme’s ability to select the most relevant beneficiaries, two types of 
beneficiaries were identified as being particularly relevant but less likely to be involved: 
(i) public authorities, for whom obstacles to participation can be mainly traced to a lack 
of  administrative  capacity  and  dedicated  human  resources;  (ii)  small  grassroots 
organisations, for whom the application process is very complex and the requirement to 
secure a co-financing is challenging (case studies 3 and 7)21.  

Coherence  

In  the  REC  programme,  the  calls  for  proposals  are  guided  by  the  specific  strategic 
developments  and  the  EU  priorities.  In  this  way,  the  REC  programme  acted  as  a  link 
across  the  EU  priorities,  in  line  and  consistent  with  other  EU  actions.  The  REC 
programme is aligned with several EU policies, such as the EU justice agenda for 2020, 
the  2016-2020  strategic  engagement  for  gender  equality,  the  2010-2020  European 
disability strategy, the list of actions by the Commission to advance LGBTI equality, the 
EU  framework  for  national  Roma  integration  strategies  up  to  2020.  The  analysis  also 
indicates  a  certain  level  of  consistency  with  national  policies  and  developments  at  the 
project level.  

EU Added Value 

The  EU  added  value  of  the  REC  programme  was  assessed  according  to  the  following 
four criteria:  

1)  Supporting  implementation  of  EU  law  and  awareness  about  the  rights  deriving 

from it;  

2)  increasing  mutual 

trust  among  participating  countries  and  cross-border 

cooperation;  

3)  developing and disseminating best practices;  
4)  Creating standards, practical tools and solutions that address EU-wide challenges.  

The programme was successful in all of these aspects, thanks to its inherent transnational 
nature.  The  evaluation  identified  additional  aspects  of  the  EU  added  value  of  the 
programme.  

 

It represents an important source of financing for the beneficiaries, enabling them 
to  implement  additional  projects  addressing the challenges  the REC  programme 
was created to tackle. 

  Receiving EU funding and implementing activities under the umbrella of the REC 
programme  had  positive  consequences  for  the  projects  in  terms  of  improved 
reputation and increased visibility that no national input could have developed. 

21 For more details, see Annex 5. 

34 

 
                                                           
  The  REC  programme  is  an  important  driver  for  innovation  since  little  national 

funding is available for testing innovative and transnational actions.  

  The  REC  programme  generated  a  scope  effect,  by  broadening  the  actions 
implemented  by  the  beneficiaries  in  a  way  that  the  national  level  interventions 
would find difficult to sustain.  

Beneficiaries  highlight  cross-border  cooperation  and  transnational  partnerships  and 
networks, and mutual learning among partners as key results of their participation in the 
programme. The transnational aspect is particularly evident for beneficiaries of operating 
grants  due  to  their  European  remit;  their  activities  could  not  have  been  similarly 
developed through national or international funding.  

Equity  

The  REC  programme’s  resources  and  support  were  not  distributed  in  a  balanced 
manner across participating states, and the distribution of REC resources was focused 
on a few countries – namely Italy, Belgium, Greece and Spain, from where the applicants 
apply  more  often.  However,  this  picture  changes  once  the  population  of  participating 
countries  is  taken  into  account,  as  some  smaller  countries  have  a  large  number  of 
beneficiaries  compared  to  their  population  (e.g.  Malta  and  Cyprus).  Furthermore,  there 
was  quite  a  high  level  of  participation  in  Romania  and  Bulgaria.  In  addition,  the 
countries,  where  the  actual  activities  take  place,  are  not  always  the  same  as  the 
‘nationality’ of the beneficiary. 

Figure 11. Number of beneficiaries by country (2016-2020) 

Source: 2016-2020 projects 

35 

 
 
Overall,  programmed  funding  reaches  different  target  groups  in  an  equitable  way, 
despite  the  concentration  of  REC  programme  resources  on  three  specific  objectives. 
Often the same target groups were targeted across different specific objectives. The REC 
programme  does  not  fully  capture  the  needs  of  people  suffering  from  multiple 
disadvantages.  

Significant financial resources were dedicated to promoting the cross-cutting priorities of 
gender  equality,  children’s  rights  and  the  rights  of  people  with  disabilities.  In  the 
implementation phase, the calls which focused on children’s rights, gender rights and the 
rights  of  people  with  disabilities  were  those  organised  under  the  dedicated  specific 
objectives. While there is a good awareness of the importance of gender mainstreaming, 
there is a general lack of knowledge of good practice among the approved REC projects 
on the best approaches to address gender mainstreaming in the activities funded outside 
dedicated projects. Beneficiaries have difficulty in formulating a gender approach when 
developing their application (case study 822). 

Simplification  

The  recently  introduced  system  for  submitting  applications  (the  funding  &  tenders 
opportunities portal) simplified the application process and improved the collection and 
aggregation of monitoring data. In general, the beneficiaries positively assess the new IT 
tools  developed  and  find  them  effective  and  easy  to  use.  The  possibility  of  directly 
communicating  with  project  officers  has  been  particularly  appreciated.  Some 
beneficiaries still believe the system is not sufficiently user-friendly, e.g. slow uploading 
of documents, unclear navigation and the lack of information on the number of characters 
allowed. 

A case study carried out in the context of this evaluation on the ‘Changes in management 
of  the  programmes  by  DG  JUST  (case  study  623)’  concluded  that  the  trend  towards 
increased digitalisation is clearly viewed as a positive development, although there could 
still  be  room  for  further  simplifying  the  process  particularly  in  relation  to  the 
administrative documentation required from applicants. 

22 For more details, see Annex 5. 
23 For more details, see Annex 5. 

36 

 
 
 
                                                           
6.  CONCLUSIONS 

The 2014-2020  REC  programme was launched when the effects  of the  economic crisis 
had led to a general reduction in the amount of national resources and funding available 
for social and fundamental rights issues.  

In this critical political and economic context, the programme has proven its EU added 
value and its crucial role in developing a European area of equality and rights. 

The REC programme continues to be relevant to the (changing) needs of stakeholders 
and the EU public. This was ensured via its flexible programming nature, including the 
annual  prioritisation  of  emerging  policy  needs  and  issues,  and  the  dialogue  with  civil 
society organisations.  

Overall,  based  on  the  qualitative  evidence  collected,  the  implementation  of  the  REC 
programme was successful and helped achieve the overall objectives. In particular, this 
success was due to the quality of most of the projects implemented with REC funding, 
enabling the implementation of concrete actions, directly relevant to the evolving needs 
on the ground.  

Both  desk  research  and  fieldwork  show  that  the  projects  were  largely  successful  in 
implementing the planned activities and almost all finalised projects have achieved the 
expected results.  

Qualitative  evidence  collected  by  the  study  suggests  that  the  benefits  of  the  REC 
programme outweigh the costs. 

As regards the baseline  situation, as  described in  the 2011  impact  assessment24 and the 
2015 ex post evaluation  of the three predecessor programmes (Daphne III,  fundamental 
rights,  and  citizenship  and  progress)25,  the  evaluation  evidence  shows  that  almost  all 
difficulties identified at that time have been overcome in the REC programme, except for 
the geographical imbalance which still persists. 

The  analysis  also  identified  a  number  of  areas  for  improvement  which  are  being 
addressed  for  the  REC  programme’s  successor,  the  CERV  programme,  which  are 
described below.   

The  weaknesses  in  the  existing  monitoring  framework  have  been  identified  as  the 
main  limitation  in  relation  to  assessing  the  programme’s  effectiveness.  Therefore,  an 
important  lesson  learned  for  the  next  programming  period  is  the  need  to  improve  the 
monitoring  system  for  the  successor  programme  in  order  to  systematically  collect 
information  on  achieved  outputs,  results  and  achieved  targets.  Such  a  comprehensive 

24 SEC(2011) 1364 Final, ibid. 
25  European  Commission  (2015),  Ex-post  evaluation  of  five  programmes  implemented  under  the  2007-
2013 financial perspective – Final Report, ibid.  

37 

 
                                                           
monitoring  system  would  be  key  to  assessing  the  programme’s  effectiveness  and 
efficiency  (and  more  generally  its  performance).  The  system  should  be  proportional  to 
the  investment  of  the  programme  and  not  be  too  burdensome  for  beneficiaries  and  the 
Commission. It should identify a limited number of output and result indicators against 
which performance will be assessed and measured.  

The  geographical  coverage  and  allocation  of  programme  resources  across  participating 
states are significantly concentrated on a limited number of (larger) countries. While 
such concentration is partly linked to the size of recipients’ countries, it may highlight a 
potential  problem  from  an  equity  perspective.  The  following  lessons  learned  were 
highlighted during the evaluation:   

1)  To  encourage  participation  in  calls  for  proposals  and  to  increase  the  geographical 
coverage  of  participants,  the  need  to  increase  the  capacity  of  organisations  in  some 
Member States  could be addressed. This  could  be achieved through the  Civil  Dialogue 
Group  (CDG)  set  up  under  the  new  CERV  programme.  The  CDG  is  an  open  and 
informal  forum  aiming  to  provide  information  about  the  programme  to  representatives 
from  civil  society  organisations,  promote  the  exchange  of  experience,  and  provide  a 
platform for sharing information and networking between projects beneficiaries. In this 
context, information sessions could be continued for work programmes and could target 
Member States where the level of participation is lower than expected. Capacity-building 
actions targeting civil society organisations in countries where the level of participation 
is typically weaker could also lead – in the medium to long term  - to a more proactive 
CSO sector, willing to invest in EU-funded projects. 

2)  The  national  contact  points  under  the  new  CERV  programme  can  contribute  to  a 
more balanced participation, especially from smaller, grassroots organisations. As they 
work in national languages, they can play a positive role particularly in less-represented 
Member States to guide and support organisations interested in the programme through 
the application process. These contact points should receive sufficient resources to carry 
out these tasks.  

3) To encourage a wide range of beneficiaries to participate, particularly organisations 
that are new to the programme, simplifying the application process and the IT tools used 
for  applying  could  motivate  potential  beneficiaries  that  are  reluctant  to  apply  due  to 
their low administrative capacity.  

The  tendency  to  implement  different  types  of  activities  (training,  research,  awareness 
raising)  under  the  same  project,  while  not  being  a  negative  element  in  itself,  makes 
projects  less  focused  and  increases  the  managerial  burden  for  coordinating  different 
tasks. In future, consideration could be given to encouraging projects in particular policy 
areas  to  focus  on  a  limited  number  of  key  activities.  The  possibility  of  funding  the 
continuation of projects through recurring or multiple grants could help beneficiaries to 
focus on a limited set  of activities, as  well as  provide opportunities to  implement  more 
systematic and continuous actions. 

38 

 
Concerning  the  evaluation  process,  no  significant  differences  in  the  characteristics  of 
projects when clustered according to their evaluation score have been detected. Projects 
sampled  that  obtained  a  score  below  80  still  managed  to  implement  all  activities  and 
achieved outputs and results that help fulfil the objectives of the REC programmes (case 
study  226).  Therefore,  the  70-points  threshold27  can  also  be  maintained  in  future 
programmes.  Meanwhile,  the  process  for  integrating  the  observations  received  during 
the negotiation phase for grant agreements could be simplified: only the major comments 
from  evaluators  could  be  taken  into  account  at  the  preparation  stage  for  grant 
agreements in order to streamline and shorten the process.  

On the administrative burden, feedback from beneficiaries suggests that there is scope for 
decreasing the administrative burden or at least for increasing its perceived usefulness. 
This  could  be  achieved  by  shifting  the  focus  of  the  requirements  from  a  compliance 
perspective to a more result-oriented one. 

At the programme level, the REC programme’s management has improved over time and 
compared with its predecessors, notably due to the new portal, improved communication 
with the Commission, offered more flexibility in budget modifications and clearer access 
to information.  

To improve gender equality mainstreaming in the future programmes, the importance of 
‘gender mainstreaming’  could be communicated more extensively, and support in  the 
form  of  guidance  could  be  provided  to  beneficiaries.  Furthermore,  the  future 
programmes could lead by example and continue promoting gender mainstreaming as a 
key tenet of their intervention. 

The  promotion  of  mainstreaming  of  other  equality  grounds  (‘equality  mainstreaming’) 
could be strengthened by requesting applicants to carry out a needs assessment from the 
perspective  of  equality,  which  could  also  cover  the  intersection  with  other  grounds  of 
discrimination. The monitoring data must be broken down by participant sex. 

Finally  mainstreaming  at  the  level  of  the  future  programmes  could  be  strengthened  by 
adopting  a  ‘gender  budgeting  approach’  to  the  programming  of  resources  by  keeping 
track of  how  resources  are allocated and spent  on actions that not  only  directly target 
gender equality, but also indirectly promote this cross-cutting objective. 

26 For more details, see Annex 5. 
27 Currently, project proposals that receive at evaluation stage 70/100 points or more can be funded. 

39 

 
 
                                                           
ANNEX 1: PROCEDURAL INFORMATION 

1.  LEAD DG, DeCIDE PLANNING/CWP REFERENCES 

Lead Directorate-General: Directorate-General for Justice and Consumers  

Decide planning reference: PLAN/2020/9646 - JUST - Report to the EP and the Council 
assessing the 2014-2020 REC Programme 

2.  ORGANISATION AND TIMING 

The evaluation was carried out between March and September 2021. The Commission 
was supported by a consortium of contractors (Ernst & Young Consulting BV/SRL and 
Fondazione Giacomo Brodolini Foundation S.r.l. SB), commissioned under the 
framework contract No JUST/2020/PR/03/0001-01 Lot 1 with a specific contract 
JUST/2021/JADM/FW/JU04/0007. 

As this is not a fully-fledged ex post evaluation, no Inter-service Steering Group was 
established. However, several DG JUST policy units (JUST 01, JUST 03, JUST A1, 
JUST B1, JUST B2, JUST C3, JUST D2) as well as colleagues from DG EMPL were 
involved and contributed to the evaluation. 

Chronology of the evaluation: 

Date 

 Meetings of Inter-service Steering Group 

November 2020  Appointment of colleagues in policy units to follow evaluation 

Preparation of the terms of reference for the request for services 

March 2021 

Contract with external evaluators begins 

Kick-off meeting of the evaluation 

April 2021 

Inception report 

June 2021 

Interim report and the second meeting  

September 2021  Final report and the third meeting 

October 2021 

Preparation of the Report to the European Parliament and the 
Council and of the accompanying staff working document 

3.  EXCEPTIONS TO THE BETTER REGULATION GUIDELINES 

As this ‘first-step’ report only provides a first assessment of the REC programme’s 
achievements, it is not fully aligned with better regulation guidelines. In particular, while 
taking guidance from the ‘better regulation’ regarding the evaluation methodology to be 

40 

 
 
 
applied, no roadmap, no public consultation and no formal inter-service steering group 
were created at this stage. These ‘better regulation’ requirements will be fully followed 
for the ‘second-step’ reports as the benefits from these would be more pertinent when 
preparing the fully-fledged ex post evaluation. 

4.  CONSULTATION OF THE RSB (IF APPLICABLE) 

Not applicable 

5.  EVIDENCE, SOURCES AND QUALITY 

The evaluation was based on evidence from different sources. The complete set of 
documents that were consulted for this evaluation is listed below: 

Legal base of the programme: 

  Regulation (EU) No 1381/2013 of 17 December 2013 establishing a Rights, Equality 

and Citizenship Programme for the period 2014 to 2020 (OJ L 354, 28.12.2013). 

Commission Implementing Decisions: 

  Commission  Implementing  Decision  C(2014)2556  concerning  the  adoption  of  the 
work  programme  for  2014  and  the  financing  for  the  implementation  of  a  Rights, 
Equality and Citizenship Programme 

  Commission  Implementing  Decision  C(2015)1997  concerning  the  adoption  of  the 
work  programme  for  2015  and  the  financing  for  the  implementation  of  a  Rights, 
Equality and Citizenship Programme 

  Commission  Implementing  Decision  C(2016)1677  concerning  the  adoption  of  the 
work  programme  for  2016  and  the  financing  for  the  implementation  of  a  Rights, 
Equality and Citizenship Programme 

  Commission  Implementing  Decision  C(2017)1316  concerning  the  adoption  of  the 
work  programme  for  2017  and  the  financing  for  the  implementation  of  a  Rights, 
Equality and Citizenship Programme 

  Commission  Implementing  Decision  C(2017)  8518  concerning  the  adoption  of  the 
work  programme  for  2018  and  on  the  financing  of  the  Rights,  Equality  and 
Citizenship Programme 

  Commission  Implementing  Decision  C(2018)  7916  concerning  the  adoption  of  the 
work  programme  for  2019  and  on  the  financing  of  the  Rights,  Equality  and 
Citizenship Programme 

  Commission  Implementing  Decision  C(2019)  7824  concerning  the  adoption  of  the 
work  programme  for  2020  and  on  the  financing  of  the  Rights,  Equality  and 
Citizenship Programme 

Desk  research  focused  on  available  documents,  both  at  programme,  policy  and  project 
level. The documents at programme level included the REC programme’s programming 
and  implementation  documents  (such  as  annual  implementation  reports,  programme 
statements, impact assessment), as well as the interim evaluation of the REC programme. 

41 

 
 
Documents at project level comprised: 

 

files with extracts of key monitoring data; 

  monitoring data derived from the interim evaluation; 

  grant  agreements  and  project  proposals  (covering  projects  funded  from  2016 

onwards); 

 

technical  periodic  reports  (final  reports),  including  monitoring  annexes  and 
financial reports (only for finalised projects funded from 2016 onwards); 

  amendment  of  grant  agreement  (‘amendment  notarised’),  providing  information 

about possible changes made after the signature of the agreement; 

  proposal  evaluation  form  (‘evaluation  summary  –  ESR’):  providing  information 
the  proposal  (scores,  ranking,  qualitative 

the  evaluation  process  of 

on 
assessment); 

 

technical report and annexes; 

  evaluation  of  the  final  technical  reports,  providing  a  final  assessment  of  the 

project by policy officers; 

 

final financial statements; 

 

list of final deliverables; 

 

list of milestones. 

The documents at policy level included the 2016-2020 strategic engagement for  gender 
equality,  the  Commission’s  list  of  actions  to  advance  LGBTI  equality,  the  EU 
Framework for national Roma integration strategies up to 2020.  

In addition, the following documents were used for this evaluation: 

CATEGORY 
GENERAL DOCUMENTS 

GENERAL DOCUMENTS 

GENERAL DOCUMENTS 

AUTHOR 

- 

- 

Prof. Julia 
Laffranque, 
Judge, 
European 
Court of 
Human Rights 

YEAR 
2012 

2012 

2014 

DOCUMENT 

Consolidated Version of the Treaty on 
European Union (TEU) 
Consolidated Version of the Treaty on the 
Functioning of the European Union (TFEU) 
Article 6 of the European Convention on 
Human Rights (Convention) and Article 47 of 
the European Union (EU) Charter on 
Fundamental Rights (Charter): Mutual 
Relation, Scope, And Interpretation 

GENERAL DOCUMENTS  EU Agency 

2020 

Fundamental Rights Report 2020 

for 
Fundamental 
Rights  

42 

 
 
 
GENERAL DOCUMENTS  EU Agency 

2020 

for 
Fundamental 
Rights  

Ten years on: Unlocking the Charter’s Full 
Potential 

GENERAL DOCUMENTS  EU Agency 

2017 

Fundamental Rights Report 2017 

for 
Fundamental 
Rights  

GENERAL DOCUMENTS  European 

2020 

institute for 
Gender 
Equality  
GENERAL DOCUMENTS  European 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS  

POLICY DOCUMENTS  

POLICY DOCUMENTS 

POLICY DOCUMENTS  

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS  

POLICY DOCUMENTS  

institute for 
Gender 
Equality  
European 
Commission 
and the 
Parliament 
European 
Commission  
United 
Nations 
European 
Commission 
European 
Commission 
European 
Commission 
European 
Commission 
European 
Commission 
European 
Commission 

European 
Commission 

European 
Commission 
European 
Commission 
European 
Commission 
European 
Commission 
European 
Commission 
 European 
institute for 
Gender 
Equality 
European 
Commission  

Gender Equality Index 2020: Key Findings for 
the European Union - Report 

2017 

2014 

Gender Equality Index 2017: Measuring 
gender equality in the European Union 2005-
2015 – Report 

The EU Justice Agenda for 2020 - 
Strengthening Trust, Mobility and Growth 
within the Union 

2014 

10 Priorities for 2015-19 

2015 

Agenda 2030 for Sustainable Development  

2016 

DG Justice strategic plan 2016-2020 

2017-
2019 
2014-
2016 
2014-
2019 
2014-
2018 
2011 

2011 

2020 

2016 

2016 

DG Justice annual activity reports  

DG Justice annual activity reports  

DG Justice management plans 

DG Justice programme statements 

Commission Staff Working Paper – Impact 
Assessment – ‘Accompanying the document 
Proposal for a Regulation of the European 
Parliament and of the Council establishing for 
the Period to 2020 the Justice Programme’. 
SEC(2011) 1364 Final 
Commission Staff Working Paper on 
Consumer Empowerment in the EU, 
SEC(2011)469 final 
EU Citizenship Report 2020 

Effective Roma integration measures in the 
Member States 
Strategic Engagement for Gender Equality 

2019 

Countering Racism and Xenophobia in the EU 

2010 

European disability strategy 2010-2020 

European Institute for Gender Equality, 
Gender Equality Index 2017: Measuring 
gender equality in the European Union 

Commission staff working document on the 
Application of the EU Charter of Fundamental 
Rights in 2016, accompanying the document 

2017 

2017 

43 

 
POLICY DOCUMENTS  

PROJECT DOCUMENTS 

PROJECT DOCUMENTS 

REGULATIONS 

European 
Union Agency 
for 
Fundamental 
Rights  
European 
Commission 
European 
Commission 
European 
Parliament and 
the Council 

Several 
years 

2014-
2016 
2014-
2017 
2013 

‘Communication from the Commission to the 
European Parliament, the Council, the 
European Economic and Social Committee 
and the Committee of the Regions on the 2016 
Report on the Application of the EU Charter 
of Fundamental Rights’, {COM(2017) 239 
final}, 18 May 2017 
FRA, Fundamental Rights Report 

Annual monitoring reports REC  

Annual work programmes REC 

Regulation (EU) 1381/2013 of the European 
Parliament and of the Council of 
17 December 2013 establishing a Rights, 
Equality and Citizenship Programme for the 
period 2014 to 2020 

44 

 
ANNEX 2: STAKEHOLDER CONSULTATION 

Consultations with the main stakeholders of the REC programme were conducted over 
several stages: 

Interviews 

Semi-structured  interviews  were  organised  with  both  project  beneficiaries/contractors 
and  Commission  staff.  Interviews  consisted  of  semi-structured  video  interviews. 
Questions were customised according to the different categories of stakeholders targeted, 
taking  into  account  their  different  answers  for  preparing  replies  to  the  evaluation 
questions. 

In-depth interviews had a two-fold objective: 

-  collecting information to feed the analysis alongside the evaluation criteria, filling 
gaps (e.g. in terms of quantifying costs and benefits) and going more in-depth with 
specific aspects,     such as identifying good practices; 

-  gathering insight and input to draw conclusions and formulate recommendations 

on how to improve the programme’s design and operation. 

A total of 53 interviews were carried out. These comprised: 

-  5 scoping interviews (April – May 2021); 

-  7 interviews with policy officers (May – July 2021); 

-  41 interviews with project beneficiaries (28 related to action grants, 8 related to 
operating grants, and 5 related to procurement contracts) (May – July 2021). 

Focus groups 

To help  draft the case studies,  in  addition to  monitoring data and information collected 
through  interviews  with  project  beneficiaries  and  Commission  staff,  three  focus  groups 
were organised with a sub-sample of project beneficiaries selected for in-depth analysis. 
Additional potential applicants were also invited to the focus groups. 

-  Focus  Group  1  –  Case  study  1:  ‘Recurring  beneficiaries/continuations  of 

projects’. 

-  Focus  Group  2  –  Case  study  3:  ‘Why  some  institutional  bodies are  reluctant  to 
apply?’  Case  study  7:  ‘Are  the  beneficiaries  selected  best  fit  to  help  the  target 
groups?’ 

-  Focus  Group  3  –  Case  study  8:  ‘How  has  the  gender  mainstreaming  been 
promoted  within  the  funded  intervention  and  what  were  the  strengths  of  your 
approach?’ 

One additional focus group was organised with DG JUST project officers. Its aim was to 
collect  additional  and  complementary  information,  focusing  on  management  and 
procedural aspect of project implementation. 

Programme Committee Members (PCM) E-Survey  

The  goal  of  this  survey  was  to  collect  comprehensive  and  specific  information  on  the 
application processes, main features and results of projects  and activities funded by the 

45 

 
 
programmes,  as  well  as  to  capture  insights  and  expert  views.  Survey  results  made  it 
possible  to  gain  a  preliminary  understanding  of  how  the  REC  programme  performed 
according to the evaluation criteria. 

A first survey was sent to PCMs. Its goal was to collect information that could feed the 
analysis  and  the  answers  to  the  evaluation  questions.  Questions  were  tailored  to  PCMs 
experience and knowledge.  

The  survey  was  launched  though  an  online  tool  (Qualtrics)  on  23 June  2021  and 
remained  open  until  14 July.  The  survey  was  kept  open  4  weeks  longer  than  originally 
planned to allow PCMs more time to answer the questions.  

A total of 17 responses were collected for the REC programme, covering the following 
countries – Austria,  Bulgaria, Czechia,  France,  GR,  Croatia,  Latvia, Romania, Sweden, 
Slovenia, and Serbia). 

Additional survey to feed the case study 4 

A second survey was launched aimed at collecting information to feed Case Study 4. The 
survey was sent to 200 project beneficiaries, and 11 relevant responses were collected.  

The  different  field  research  activities  together  with  the  main  stakeholders  involved  and 
their relationship with the methodological steps are shown in the table below. 

Distribution of stakeholders consulted 

Type of activity 

Coverage/stakeholders 

Scoping interviews 

Programme managers and 
other Commission staff 

Interviews 

Interviews with 
beneficiaries 

Focus groups with 
(potential) beneficiaries 
Focus group with DG 
JUST project officers 
Web-based survey 

Policy officers (covering the 
different specific objectives) 
Action grants 
operating grants 
contractors 
(potential) Beneficiaries 

DG JUST project officers 

Programme committee 
members 
Projects sampled that benefited 
from non-EU funding (Case 
study 4) and that experienced 
underspending (Case study 5) 

N. 

5 

7 

37 

21 

4 

17 

Meth steps 
involved 
Task 3 case studies 
Task 3 Evaluation 
Questions (EQ) 
Task 3 case studies 
Task 3 EQ 
Task 3 case studies 
Task 3 EQ 

Task 3 case studies 

Task 3 EQ 

Task 3 EQ 

46 

 
 
 
 
ANNEX 3: METHODS AND ANALYTICAL MODELS 

Starting  from  a  deep  understanding  of  the  programme,  its  intervention  logic  and 
underlying  theory  of  change,  the  evaluation  is  built  upon  three  main  interconnected 
methodological steps. 

-  Task  1  –  Extraction,  collection  and  analysis  of  the  information  monitored 
providing  an  overview  of  the  state  of  play  on  implementation  and  of  the 
programme’s performance and main achievements. 

-  Task 2 – Collection of additional information, through: 

o 

o 

research 

field 
stakeholders  and 
Commission  officials,  focus  groups,  e-survey  to  programme  committee 
members and additional documentary review); and 

(interviews  with  beneficiaries, 

in-depth analysis of a sample of projects and services funded through the 
programme,  providing  a  more  nuanced  and  detailed  picture  of  the  key 
features and achievements of a limited set of projects:  

 

types of actions implemented; 

  how the actions are linked with the specific objectives; 

  outcomes and results of the actions. 

-  Task 3 – Drafting  case studies and answers to the evaluation questions. Based 
on the information and findings derived from the two tasks mentionned above and 
their triangulation: 

a.  eight  case  studies  were  drafted,  focusing  on  a  set  of  operational  and 
procedural  components  covering  both 
justice 
programmes  and  ultimately  contributing  to  feeding  the  answers  to  the 
evaluation questions 

the  REC  and 

the 

b.  the evaluation questions were answered and suggestions were identified 

as regards improvement. 

Data collection tools and activities 

The study relied on both desk and field research. 

Desk research focused on available documents, both at programme, policy and project 
level. It was mostly carried out under Task 1, but additional desk research was carried out 
for projects selected for in-depth analysis under Task 2 and for answering the evaluation 
questions. 

These documents were mainly used for the following steps: 

  building a  retrospective  overview of the programme against which the  achievement 

and performance of the projects were assessed; 

47 

 
 
 

reviewing and integrating, where relevant, the Intervention Logic and the underlying 
theory of change28;  

  collecting additional information on programmes’ performance. 

Information collected was fed into the evidence grids, the case studies and ultimately the 
evaluation report. Part of the information was also used to finalise Task 1 data collection 
(e.g.  Block  4  indicators  covering  projects  sampled  and  Block  2  indicators  covering  the 
‘legal base indicators’). 

Distribution and coverage of project documents reviewed 

Type of document 
Monitoring  files  from 
the interim evaluation 
Monitoring  files  by  DG 
JUST 
Grant 
agreements/proposals 
Technical reports 

Additional project docs 
(e.g.  amendments  and 
evaluation summary) 

from 

funded 

Coverage/stakeholders 
Projects funded in 2014-
2016 
Projects 
2016 onwards 
Projects 
2016 onwards 
Finalised 
funded 
onwards 

projects, 
2016 

funded 

from 

from 

N. 
1 

7 

599 

215 

Meth steps involved 
Task 1 Monitoring 

Task 1 Monitoring 

Task 1 Monitoring 
Task 2 In-depth analysis 
Task 1 Monitoring 
Task 2 In-depth analysis 

Sampled projects 

AGs and OGs: 298 
Procurements: 23 

Task 2 In-depth analysis 
Task 3 Case studies 

The field research aimed at collecting relevant data and input from project beneficiaries 
and  stakeholders,  filling  gaps  and  validating  information  collected  during  the  desk 
research. It was mostly carried out as part of the in-depth project analysis under Task 2 
and during the inception phase, but additional field information was collected under Task 
3  for  drafting  the  case  studies  and  answering  the  evaluation  questions.  For  more 
information, please also see Annex 2: Stakeholder consultation. 

Task 1 - Extraction, categorisation, and aggregation of monitoring data 

For this task, the following steps were undertaken.  

A.  Creation of the dataset for action grants and operating grants  

The  information  retrieved  from  monitoring  files  was  cleaned,  systematised,  and 
transferred  to  a  dataset  specifically  created  for  this  evaluation.  Since  information 
available  from  monitoring  files  and  project  documents  covered  mostly  2016-2020 
projects  and  applications,  resulting  in  the  dataset  still  having  gaps  and  missing 
information  for  older  projects,  the  dataset  was  integrated  with  information  from  the 
interim evaluation. Similarly, information on applicants and participants covered mostly 
2016  onwards.  Information  was  further  completed  from  different  sources:  e.g. 

28 Theory of change implies logical explanation of the ways and assumptions through which allocation of 
funds  will produce outputs through which intended results are to be achieved (the expected change). 
This  theory  of  change  has  to take  on  board  the  economic  and  political  context  of  the  programme  as 
well as other factors (social, cultural, institutional…) that may influence the mechanisms leading to the 
results. 

48 

 
 
 
                                                           
information  on  the  type  of  organisation  applicants  represented  –  which  was  missing  – 
was extracted from the list of participants and linked to those applicants that participated 
in the granted project too.  

Further information was obtained from the following. 

  New extractions on projects and applications – both updating the list and adding new 
types  of  information  from  proposals  (e.g.  on  the  names,  country,  and  type  of 
beneficiary)  and  reports  (e.g. on  dissemination activities carried out).  The new data 
were either added to the existing dataset or used to update it.  

  Annex 3 to the technical reports. The new data were first extracted automatically and 
then  manually  cleaned  and  systematised  to  reduce  the  ‘noise’  and  prevent  it  from 
being lost’.  

B.  Extraction of key qualitative texts/unstructured information 

To extract data on qualitative descriptors, the research team adopted a mix of qualitative 
data analysis approaches and tools (word search, categorisation, text mining and semantic 
tools). The starting base for implementing these  approaches  involved firstly  identifying 
the  document  sources  and  the  exact  text  fields  within  these  sources  to  be  screened. 
Next, the team organised the manual extraction (copy/paste) of these relevant text fields 
into  an  excel  sheet  on  which  the  various  qualitative  analysis  tools  could  then  be 
implemented.  Based  on  the  qualitative  information  extracted,  the  following  analyses 
were carried out: 

Text mining/semantic tool 

The  documents  were  analysed  using  named  entity  recognition  (NER)  tools.  NER  is  a 
subtask  of  information  extraction  that  seeks  to  locate  and  classify  named  entities 
mentioned  in  unstructured  text  into  pre-defined  categories  such  as  names  of  people, 
organisations,  locations,  medical  codes,  time  expressions,  quantities,  monetary  values, 
and  percentages.  In  the  case  of  analysing  project  proposals,  these  entities  are  words 
relevant  for  the  content  of  the  call.  In  our  case,  we  focused  on:  target  groups, 
dissemination activities and needs.  

Screening of qualitative information and categorisation 

For  a  number  of  descriptors,  the  research  team  adopted  a  manual  categorisation 
approach, involving the screening of specific text fields and categorising the information 
for each project based on a drop-down list of categories.  

Mixed methods (word search + categorisation): 

For a number of descriptors, a mixed approach to qualitative analysis was followed. First, 
the relevant information (when available) was identified through a simple word search; it 
was then manually extracted. This approach was followed for the following descriptors: 

  number  of  interactions  (likes,  shares,  retweets,  comments)  generated  by  online 

awareness-raising, information and dissemination activities; 

  Number  of  fans  and  followers  acquired 

through  online  awareness-raising, 

information and dissemination activities 

49 

 
 
 
C.  Creation of a list of indicators/descriptors 

The monitoring data that were collected as per the previous steps were then categorised 
and aggregated – to support a narrative around key topics – as follows:  

 

 

 

 

 

indicators and descriptors related to applicants/participants; 

indicators  and  descriptors  related  to  the  projects’  budget  allocation  and 
distribution; 

indicators and descriptors related to the types of activity funded; 

indicators and descriptors related to target groups/stakeholders; 

indicators  and  descriptors  related  to  other  features  (such  as  geographical 
coverage, project duration); 

indicators and descriptors related to the programme’s performance/the evaluation 
criteria. 

Task 2 - Additional collection and systematisation of information  

After the project-related data and documents were collected, extracted, and systematised 
as  described  in  the  previous  section,  the  Research  Team  carried  out  further  data 
collection activities to gather additional input and systematise all information collected to 
feed the response to the evaluation questions. 

This included both desk and field research on a sample of projects.  

A  total  of  36  projects,  involving  beneficiaries  of  both  action  (28)  and  operating  grants 
(8),  plus  six  procurement  contracts,  were  sampled  in  order  to  carry  out  a  qualitative 
analysis  based  on  comments  and  insights  from  key  stakeholders  on  the  REC 
into  account 
programme’s 
geographical distribution, specific objective, and financial amount.  

implementation.  The  projects  were  sampled 

taking 

D.  Sampling of action grants 

This  sample  of  projects  aimed  to  be  as  representative  as  possible  of  the overall  project 
‘population’ at Member State/specific objective level, which was: 

  proportional  to  the  amount  and  form  of  funding  per  Member  State/specific 

objective; 

  comprehensive, thus including projects that: 

o  covered all specific objectives; 

o  covered all forms of support; 

o  covered different years of the programme’s period’. 

These  criteria  ensured  that  the  sample  of  projects  identified  was  suitable  enough  to 
demonstrate how the programme was implemented by each of its specific objectives. 

In order for this distribution to be representative not only in terms of specific objectives 
and forms of support, but also geographically, attention was also paid to the beneficiary’s 
nationality.  Therefore,  the  sample  of  projects  ensured  that  Member  state  representation 
(in  terms  of beneficiaries’  nationalities)  was  proportionate  to  the  overall  geographical 
distribution  of projects (in terms of participation in the calls and success rate for action 
grants and in terms         of grants awarded for operating grants).  

50 

 
E.  Sampling of operating grants 

On selecting the operating grants, this was first informed by criteria such as: i) the overall 
financial  allocation  to  a  given  organisation/network  within  the  specific  objective;  or 
ii) the  relevance  of  the  organisation  to  the  programmes  objectives  and  the  specific 
objective. Sampling of procurement contracts 

The selection of the procurement contracts was based on the relevance of the contracted 
service  to  the  programmes’  objectives  and  to  the  European  Commission’s  needs  as 
regards  information.  In  selecting  contracts,  care  was  taken  to  cover  different  types  of 
procurement activities  

Task 3 Drafting case studies and answering the evaluation questions 

F.  Case studies 

The  evaluation  included  eight  case  studies.  Their  main  purpose  was  to  provide  solid, 
triangulated evidence to support preparation of replies to evaluation questions as well as 
suggestions for improvements, with a focus on  several specific issues. The case studies 
were developed, based on a customised methodological approach. 

In  addition  to  data  and  information  collected  under  Tasks  1  and  2,  the  case  studies 
involved additional field and desk research, in particular: 

further in-depth analysis of sampled projects’ documents;  

 
  an online survey regarding all projects sampled (for Case Study 4); 
 

three focus  groups, which included beneficiaries  from projects that were not part of 
the sample (for case study 1, 3 and 7, 8). 

The  table  below  provides  an  overview  of  what  the  eight  case  studies  focused  on  and 
shows the evaluation questions they refer to.  

Overview of the eight case studies 

Case study 

Description 

Case study 1 
Recurring beneficiaries/ 
continuous projects 

Case study 2 
Is the selection/evaluation 
process for the project 
proposals robust? Is the 70 
points threshold pertinent? 
Case study 3 
Why are some potential 
beneficiaries reluctant to 
apply? 

The analysis aimed at finding out what the 
benefits/disadvantages were in financing the 
continuation of projects or projects which are 
implemented by recurring beneficiaries. The 
case study will draw a clear distinction between 
the two situations covered, i.e. recurring 
beneficiaries and continuous projects.  
The objective of the analysis was to verify 
whether the selection/evaluation process allows 
for quality project proposals to be selected. 

The focus will be on public authorities, relevant 
from the perspective of programmes and calls 
objectives, that do not apply for restricted calls 
(and inversely, to know the drivers for 
participation). 
Among the drivers for participation/non-
participation, the evaluation will explore 
administrative burden as a possible reason for 
not applying, or whether the reason is related to 
awareness/communication, capacity, language 
barrier or relevance.  

51 

Relevant evaluation 
questions 
EQ 2. How fit for purpose 
were the actions funded under 
the programme to the 
objectives of the programme?  

EQ 3. What factors influenced 
the achievement observed? 
Sub-question: Has the 
selection procedure been 
effective?  
EQ 4. How effective have 
been the communication 
activities in informing the 
potential applicants about 
upcoming calls and in 
increasing the visibility of 
funded projects?  
EQ 9. How relevant were the 
actions and results achieved to 
the needs of the different 
stakeholders? 

 
Case study 

Description 

Case study 4 
How do EU projects 
compare in terms of 
costs/benefits to similar 
projects funded by other 
sources? 
Case study 5 
Understanding the scale of 
underspending in projects 

Case study 6 
Changes in management of 
the programmes by DG 
JUST 

The analysis aimed to conclude whether the EU 
projects are comparable in terms of costs and 
benefits to similar projects funded from other 
sources. If EU projects were found to be more 
costly, then the analysis aimed to identify the 
reasons behind this. 
The analysis aimed to compare the planned 
costs with the actual costs per project, to 
understand the reasons behind the recurrent 
underspending and to identify solutions to 
tackle this. 
The objective is to see how the management of 
the programmes at DG JUST level evolved over 
the programming period – 2014-2020 – and 
what was the effect on the beneficiaries. 

Case study 7 
Are the beneficiaries 
selected the best fit to help 
the target groups in terms of 
socio-economic impact? 

Case study 8 
Evaluation of gender 
mainstreaming at the level 
of programme activities 

The objective of the analysis is to understand 
whether the programmes are funding the most 
relevant organisations, what the most effective 
combination of actors within a 
partnership/consortium is, and how essential is 
the role of project management organisations. 
The analysis aims to evaluate gender 
mainstreaming at the level of project activities 
by looking at how gender has been 
mainstreamed across the different projects and 
the different stages of the process cycle 
(programming, implementation, monitoring and 
evaluation). 

G.  Answering the Evaluation questions 

Relevant evaluation 
questions 
EQ 6. To what extent has the 
intervention been cost-
effective?  

EQ 6. To what extent has the 
intervention been cost-
effective?  

EQ7. Has the management of 
the programme by the 
Commission become more 
efficient?  

EQ 9. How relevant were the 
actions and results achieved to 
the needs of the different 
stakeholders?  

EQ 15. How has the gender 
mainstreaming been promoted 
within the funded 
interventions?  

Information gathered through desk and field research was used to answer the following 
evaluation questions:  

1.  To what extent have the objectives been achieved considering the set indicators?  

2.  How  fit  for  purpose  were  the  actions  funded  under  the  programme  to  the 

objectives of the programme? 

3.  What factors influenced the achievements observed? 

4.  How  effective  were  the  communication  activities  as  to  informing  the  potential 
applicants  about  upcoming  calls  and  in  increasing  the  visibility  of  funded 
projects? 

5.  Have the projects delivered the results envisaged in the applications?  

6.  To what extent have the interventions been cost-effective? 

7.  Has  the  management  of  the  Programme  by  the  Commission  become  more 

efficient?  

8.  How relevant were the actions and results achieved to the needs of the different 

stakeholders? 

9.  How  well  do  the  (original)  objectives  (still)  correspond  to  the  needs  within  the 

EU? 

52 

 
10. How  relevant  for  achieving  the  objectives  of  the  Programme  were  the  groups 

targeted by the intervention? 

11. To  what  extent  are  these  interventions  coherent/  complementary  with  other  EU 
and/or  national  policies  and  funding  Programmes  which  have  similar  objectives 
and Union bodies' work? Are the conclusions on coherence and complementarity 
in the interim evaluation still valid? 

12. What is the added value resulting from the EU intervention(s), compared to what 

could be achieved by Member States? 

13. What would be the most likely consequences of limiting the level of the existing 

EU intervention or completely stopping/withdrawing from it? 

14. How  fairly  are  the  different  activities  distributed  across  the  different  target 

groups, and participating countries? 

15. How  has 

the  gender  mainstreaming  been  promoted  within 

the  funded 

interventions?  

16. How have the rights of the child been promoted within the funded interventions? 

17. How have the rights of people with disabilities been promoted within the funded 

interventions? 

18. How  can  the  programme  management,  with  focus  in  particular  on  the  grant 
management,  be  further  simplified  to  alleviate  administrative  burden  of  the 
Commission and of the applicants and the beneficiaries? 

53 

 
ANNEX 4: PROGRESS ON INDICATORS 

Indicators concerning the general objective 

Result indicators 

Baseline 2013 

1.  Female  employment  rate  20-64  age 
group 

62.4  % 
Source: Eurostat) 

(2012, 

Results  available  thus 
far29 

2020 targets 

71% for women 

65.5% (2016) 
66.4% (2017) 
67.6% (2018) 
67.4%(2019) 

Overall  assessment:  the  target  will  be  difficult  to  achieve  given  that  Europe  is  exiting  a  long  financial  recession, 
exacerbated in 2020 by the Covid-19 crisis. 

2.  Employment  rate  of  people  with 
disabilities 

(2013, 
48.5% 
Source:  Eurostat) 
EU-Statistics 
on 
Income and Living 
Conditions 

50.6% (2017) 

55% 

Overall assessment: the target will be difficult to achieve because of the impact from recession . 

3. The gender pay gap 

16.8% 
Source: Eurostat) 

(2013, 

16% (2017) 

14% 

Overall assessment: the gender pay gap is showing decreasing trends but the target is unlikely to be reached. Projects 
financed by the programme promote  wage  transparency and awareness-raising activities  that  can  enable  employees 
and employers to reduce the gender pay gap. 

4.  The  percentage  of  women  among 
non-executive  directors  on  boards  of 
listed companies 

16.2% 
Source: EC data) 

(2011, 

31.4% (2019) 

40% 

Overall assessment: this percentage is increasing steadily. Nevertheless, the 2017 milestone – 30% target – was only 
reached in 2019 and the 2020 target – 40% – is unlikely to be reached. 

5.  Percentage  of  Europeans  who 
consider  themselves  as  ‘well’  or  ‘very 
well’ informed of the rights they enjoy 
as EU citizens 

(2010, 

32% 
Source: 
Eurobarometer) 

42% (2015) 
54% (2017) 
55% (2018) 

51% 

Overall assessment: the target of this indicator was reached in 2017. Appropriate awareness-raising activities tailored 
to different targets could further improve the result. 

Indicators concerning the specific objectives 

Result indicators 

Baseline 2013 

Results,  as 
available30 

far  as 

2020 targets 

Specific  objective  on  non-discrimination  –  RDIS:  To promote  the  effective  implementation of  the  principle  of  non-
discrimination on the grounds of sex, racial or ethnic origin, religion or belief, disability, age or sexual  orientation, 

29 Programme statement 2021– document. 

30 Programme statement 2021– document. 

54 

 
 
 
 
                                                           
and to respect the principle of non-discrimination on the grounds provided for in Article 21 of the Charter 

1. Share of person aware of their rights 
if they fall victims of discrimination 

37%  

45% (2015) 

70% 

(2012, Source: 
Eurobarometers) 

Overall  assessment:  Since  the  interim  evaluation,  no  further  data  on  the  progress  of  the  indicator  is  available. 
Therefore, it is difficult to assess whether the target has been achieved.  

up 

structural 

2.  The  number  of  Member  States  that 
set 
coordination 
mechanisms  with  all  stakeholders, 
including 
the 
Roma, 
implementation  of  the  national  Roma 
integration  strategies  (source:  national 
reporting to the Commission) 

on 

No Member State 
(2013, Source: EC 
data, based on 
national reporting) 

21  Member 
(2016);  

22  Member 
(2019) 

States 

States 

26  (Malta  is  not  included 
as 
it  has  no  Roma 
population) 

Overall  assessment:  The  indicator  is  unlikely  to  be  achieved.  Nevertheless,  significant  improvements  have  been 
registered  thanks  to  the  considerable  contribution  from  the  REC  programme.  Ad  hoc  calls  took  place  on  the 
implementation of the national Roma integration strategies. Therefore, the achievement of the target depends largely 
on the investments from the programme in this regard.   
Specific  objective  on  preventing  violence  (Daphne)  –  RDAP:  To  prevent  and  combat  all  forms  of  violence  against 
children,  young people  and  women,  as  well as  violence against other  groups at  risk,  in  particular  groups  at  risk  of 
violence in close relationships, and to protect victims of such violence  

1. Percentage of people that consider 
domestic violence against women 
unacceptable 

84% (2010, Source: 
Special 
Eurobarometer 449 
(2016)) 

100% (2018)  

100% 

Overall assessment: the target was formally achieved in 2018, but there could be bias affecting the results. Still, it is 
clear that there has been progress and a positive trend. Until 2015, there had been no progress made in this regard, 
but  it  is  likely  that  the  awareness-raising  actions  under  the  projects  funded  by  the  programme,  which  represent  a 
significant portion of the activities funded in this area, led to positive change. This indicator is also affected by many 
external factors, such as economic growth and national policies (including ratification of the Istanbul Convention). 
Specific objective on data protection – RDAT: To contribute to ensuring the highest level of protection of privacy and 
personal data  

1.  Number  of  complaints  received  by 
data 
from 
individuals related to data protection 

authorities 

protection 

60 000 (2018) 

60  000 

68 569 (2010, 
Source: Special 
Eurobarometer 449 
(2016)) 

Overall  assessment:  As  the  Programme  Statement  (2021)  reports,  this  indicator  cannot  really  be  used,  as  data 
protection authorities in Member States have different methods for recording the number of complaints received from 
individuals,  to  the  extent  that  e.g.  Denmark  was  reporting  6 000  and  Romania  25.  For  this  reason,  since  the 
application  of  GDPR  from  25  May  2018,  a  more  relevant  indicator  would  be  the  number  of  cross-border  cases 
recorded in the internal market information (IMI) system case register (the IMI was developed by the Commission as 
the  IT  platform  to  support  cooperation  and  consistency  procedures  under  the  GDPR).  In  2018,  from  25  May  to 
31 December, there were 213 cases; in 2019: 553; and in 2020: 626. In total, there have been 1392 cases. 

Specific objective on EU citizenship – RCIT: To promote and enhance the exercise of rights deriving from citizenship 
of the Union 

1.  Share  of  the  population  that  know 
the  meaning  of  citizenship  of  the 
Union 

46%  (2013,  Source: 
Eurobarometer) 

52% (2015) 

53%   

Overall assessment: Since the interim evaluation, no additional data on the progress made in relation to this indicator 
were  available.  The  target  was  almost  achieved  in  2015,  and  it  is  likely  to  have  been  achieved  in  the  meantime. 
However,  it  is  difficult  to  gauge  the  REC programme’s  contribution to  these  results  since  this  topic  is  addressed  by 
other EU-level initiatives (i.e. Erasmus+). 

2. Share of the population that consider 
themselves 
‘very  well’ 
‘well’  or 
informed  of  the  rights  they  enjoy  as 

36%  (2013,  Source: 
Eurobarometer) 

54% (2017) 

51%   

55 

 
 
 
 
citizens of the Union 

Overall  assessment:  The  target  of  the  indicator  was  slightly  overachieved  in  2017.  As  for  the  previous  indicator 
though, it is difficult to gauge the REC programme’s contribution to these results since this topic is addressed by other 
EU-level initiatives (i.e. Erasmus+). 

3.  Awareness  of  the  right  to  vote  and 
stand  as  a  candidate  in  European 
elections  in  the  Member  State  of 
residence,  without 
the 
holding 
nationality of that Member State 

72%  (2013,  Source: 
Eurobarometer) 

67% (2015) 

80%   

Overall assessment: Since the interim evaluation, no additional data on the progress made in relation to this indicator 
were available. 

Specific  objective  on  consumers'  rights  –  RCON:  To  enable  individuals  in  their  capacity  as  consumers  or 
entrepreneurs  in  the  internal  market  to  enforce  their  rights  deriving  from  Union  law,  having  regard  to  the  projects 
funded under the Consumer Programme  

1. Consumers’ perception that they are 
being protected 

64%  (2011,  source: 
Consumer 
Conditions 
Scoreboard) 

76% (2015);  
72% (2018) 

75% 

Overall assessment: Despite the target set for this indicator being achieved in 2015, its value decreased. Awareness-
raising activities under the REC programme can help improve consumers’ perception of being protected and stabilise 
the indicator. 

2.  Percentage  of  consumers  who  are 
aware  of  their  right  to  keep  a  product 
they  did  not  order  which  was  sent  to 
them along with an invoice 

36%  (2011,  source: 
Consumer 
Conditions 
Scoreboard) 

34% (2015) 
 35% (2018) 

39% 

Overall assessment: Although the target of this indicator is not far from being achieved, no progress has been made. 
The funding provided under the REC programme to fulfil this specific objective was very limited. 

3. Percentage of retailers in the EU 
who know that including invoices with 
marketing material (for unordered 
products) is prohibited 

51%  (2011,  source: 
Consumer 
Conditions 
Scoreboard) 

56% (2015)  60% 

Overall assessment: the target is likely be achieved, but there is no fresh data available for the moment. 

34%*(2012,  source: 
Eurobarometers) 

57.8% (2016)  50% 

4. Level of consumer confidence in 
cross-border shopping, as measured by 
the percentage of consumers who have 
at least an equal level of confidence in 
sellers from their own country as they 
have in sellers from another EU 
country (source: Eurobarometers) 

Overall assessment: the target is likely be achieved, but there is no fresh data available for the moment. 

56 

 
 
 
 
 
 
 
 
 
 
ANNEX 5: RESULTS OF THE CASE STUDIES 

Case study 

Results 

1. Recurring 
beneficiaries/continuation of 
projects 

2. Is the selection/evaluation 
process for project proposals 
robust? Is the 70 points threshold 
pertinent? 

REC programme: 54.86% of partners are one-time beneficiaries.  

 

effects 

(organisational  objectives, 

Implementing  more  projects  or  continuing  projects  had 
positive 
reduced 
administrative  burden,  visibility  of 
the  organisation, 
reaching out to the target groups, strengthening partnerships 
and possibility of building on the experience of the previous 
partnerships). 

  Public  authorities  were  recurring  beneficiaries  because  the 
main  incentive  to  apply  for  funding  was  the  drive  for 
innovation. 

  Recurring  funding  increases  the  beneficiary’s  capacity  to 

apply and obtain further funding.  

  No significant differences in the characteristics of projects 

that obtained between 70 and 100 points.  

  REC  projects  obtain  on  average  higher  evaluation  scores 

than those under the justice programme. 

  Despite several aspects related to the evaluation procedures 
that could be improved in the future, overall, the evaluation 
criteria  and  appraisal  carried  out  accurately  captures  the 
quality of the applications/projects.  

  The  selection/evaluation  process  for  project  proposals  is 

robust and the 70 points threshold is pertinent.  

3. Why are some potential 
beneficiaries reluctant to apply? 

  Geographical  coverage  of  countries  in  some  participating 

countries remains low. 

  No  evidence  that  potential  applicants  were  not  being 
low 
informed 
programme 
coverage/quality  of  communication  and  dissemination 
activities.  

about 

calls 

due 

to 

  Main factors for not applying (mainly public institutions): 

1)  Low  capacity  and  lack  of  experience  in  applying  for  and 
implementing EU-funded projects (vis-à-vis  their perceived 
complexity,  including  the  technical  language  used,  and  the 
competitiveness of applications,);  

2)  The  requirement  to  secure  co-financing  can  be  an  obstacle 
for both public institutions and non-profit organisations; 

3)  Short time span between the call’s launch and the deadlines, 

especially for public institutions. 

4. How do EU projects compare 
in terms of costs/benefits to 
similar projects funded from 

  Despite  its  application  procedures  being  quite  burdensome, 
the  REC  programme  is  found  to  provide  much  greater 
benefits  than  other  non-EU  initiatives,  despite  the  initial 

57 

 
 
 
 
 
other sources? 

costs and effort required in order to participate. 

  Nonetheless,  potentially  valuable  organisations  (especially 
smaller  ones)  are  often  prevented  from  participating  in  the 
EU  programmes,  and  therefore  have  to  choose  from  other 
funding  instruments,  which  are  less  focused  on  creating  an 
EU added value, as the sole alternatives. 

5. Understanding the scale of 
underspending in projects 

  Underspending  on  EU  resources  seems  quite  limited 

under the REC programme. 

  Key  reasons  for  underspending,    seem  to  relate  to  both 
internal and external factors, mainly linked to the difficulty 
in implementing and/or completing the activities as planned, 
namely: 

1)  Occurrence  of  unexpected  events 

(e.g.  COVID-19 

pandemic); 

2)  Adoption  of  a  careful  project  management  approach  and 

expenditure choices by project beneficiaries; 

3)  Overestimation of costs in the proposal. 

  Administrative complexities or potential rigidities within the 
programmes’  procedures  do  not  seem  to  be  reasons  for 
underspending for projects. 

6. Changes to the programmes’ 
management by DG JUST 

The programme’s management cycle has been improved in recent 
years, according to recurring beneficiaries. 

  Adopting 

the  AWPs 

their 
implementation helped to launch the calls earlier and thus to 
complete the selection process during the same year. 

the  year  prior 

in 

to 

  The  length  of  the  appraisal  phase  and  the  time  taken  to 

prepare grant agreements has also been shortened. 

  There  was  an  improvement  in  the  ‘predictability’  of  the 

calls. 

  The application and reporting procedures were digitalised. 

However, the length of the application process is one of the main 
problems encountered by beneficiaries, which has not been addressed 
so far.   

Another factor that makes the application process difficult for 
beneficiaries is the very technical language used in the application 
form. 

7. Are the beneficiaries selected 
the best fit to help the target 
groups in terms of socio-
economic impact? 

  This  evaluation  could  not  find  clear  evidence  that  projects 
were being awarded to organisations that were irrelevant as 
regards  the  objectives  set  (at  programme,  call,  and  project 
level). 

  Beneficiaries  –  especially  public  authorities  –  are  more 
likely to use the services of external consultancies or partner 
with  ‘specialised’  organisations  to  help  them  submit  the 
application  and  to  get  support  in  implementing  the  project. 
However this is not a negative factor in itself. 

58 

 
 
 
 
  With  all  due  caveats,  projects  implemented  by  consortia 
comprising  two  types  of  beneficiaries,  where  one  is  a 
secondary  or  higher  education  body,  are  usually  less 
effective.  Projects 
implemented  by  three  or  more 
partners are more effective.  

  Difficulties 

in 

encountered 
and 
terms  of 
implementation  phase  by  beneficiaries 
partnership  development  and  coordination  significantly 
influence  the  project’s  potential  to  achieve  its  outputs  and 
generate positive medium and long-term effects.  

application 
in 

the 

8. Evaluation of gender 
mainstreaming at the level of 
programme activities 

  Assessments on gender-sensitive needs are seldom carried 
out. This is the key reason why there is still significant room 
for improvement as regards gender mainstreaming. Another 
reason  is  the  less  detailed  presentation  of  expectations  in 
terms of gender mainstreaming at programme level. 

  Overall,  there  is  a  high  level  of  awareness  as  regards  the 
importance  of  promoting  gender  equality,  but  this  does  not 
translate into gender-sensitive/transformative activities. 

  Gender-related measures taken are often limited to ensuring 
equal  participation  in  project  activities  (e.g.  training)  or 
teams.  

59