COMMISSION STAFF WORKING DOCUMENT EVALUATION Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL Ex post evaluation of the Structural Reform Support Programme
Date: 2023-10-05
Institutions: Directorate-General for Structural Reform Support, European Commission
Subjects: EU Member State, EU aid, EU financing, EU programme, administrative reform, economic convergence, economic reform, institutional reform, structural adjustment

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 5.10.2023  SWD(2023) 322 final  COMMISSION STAFF WORKING DOCUMENT EVALUATION Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL  Ex post evaluation of the Structural Reform Support Programme {COM(2023) 571 final}  
 
 
Table of contents 

1. 

INTRODUCTION ................................................................................................................................ 4 

2.  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? ....................................... 5 

The ex-ante evaluation of the SRSP ............................................................................ 9 

The mid-term evaluation of the SRSP ...................................................................... 10 

3.  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? ....................... 10 

Current state of play .................................................................................................. 12 

Main outputs and results achieved during the implementation of the programme ... 15 

4. 

EVALUATION FINDINGS (ANALYTICAL PART) ....................................................................... 17 

4.1.  To what extent was the intervention successful and why? ................................ 17 

4.1.1.  Effectiveness:  How  effective  has  the  SRSP  been  in  achieving  its 
objectives? ................................................................................................................. 17 

4.1.2 Efficiency: How efficient has the SRSP been in achieving its objectives? ..... 25 

4.1.3 Coherence ......................................................................................................... 35 

4.2 How did the EU intervention make a difference and to whom? ............................. 39 

4.3 Is the intervention still relevant? ............................................................................. 43 

5.  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? ................................................. 49 

5.1  CONCLUSIONS ................................................................................................................................. 49 

5.2  LESSONS LEARNED ........................................................................................................................ 54 

ANNEX I. PROCEDURAL INFORMATION ............................................................................................. 56 

1. 

2. 

LEAD DG, DECIDE PLANNING/CWP REFERENCES ............................................................. 56 

EXCEPTIONS TO THE BETTER REGULATION GUIDELINES ........................................... 56 

3.  ORGANISATION AND TIMING ................................................................................................... 56 

4.  CONSULTATION OF THE RSB (IF APPLICABLE) ................................................................. 56 

5. 

EVIDENCE, SOURCES AND QUALITY – INCLUDING EXTERNAL EXPERTISE ............ 56 

ANNEX II. METHODOLOGY AND ANALYTICAL MODELS USED ................................................... 58 

ANNEX III. EVALUATION MATRIX AND, WHERE RELEVANT, DETAILS ON ANSWERS 

TO THE EVALUATION QUESTIONS (BY CRITERION) ............................................................. 66 

1. Effectiveness (results) ............................................................................................... 66 

2. Impact ....................................................................................................................... 73 

3. Efficiency .................................................................................................................. 76 

4. Relevance .................................................................................................................. 82 

5. Coherence ................................................................................................................. 84 

6. EU added value ......................................................................................................... 87 

1 

 
 
 
ANNEX IV. OVERVIEW OF BENEFITS AND  COSTS  AND, WHERE RELEVANT, TABLE 

ON SIMPLIFICATION AND BURDEN REDUCTION. .................................................................. 90 

ANNEX V. STAKEHOLDER CONSULTATION – SYNOPSIS REPORT ............................................... 95 

1.1.  Summarised results of the Open Public Consultation (OPC) ..................... 98 

1.2.  Summarised results of the targeted survey ................................................. 103 

1.3.  Summarised results of interviews ............................................................... 111 

1.4.  Results of focus group discussions .............................................................. 115 

2.1.  Main outputs and results achieved during the implementation of the programme

123 

2 

 
 
 
 
 
 
Term or acronym 

Meaning or definition 

Glossary 

BA  

CA  

CSP  

CSRs  

EPSAS 

ERDF  

ESF  

ESIF  

JRC 

JTF  

JTM  

OPC  

SDG 

SRSP  

SRSS  

SWD 

Beneficiary authority 

Coordinating authority 

Cooperation and support plan 

Country-specific recommendations under the European Semester 

European Public Sector Accounting Standards  

European Regional Development Fund 

European Social Fund 

European structural and investment funds 

Joint Research Centre 

Just Transition Fund 

Just Transition Mechanism 

Open public consultation 

Sustainable development goal 

Structural Reform Support Programme 

Structural Reform Support Service 

Staff working document 

TAIEX  

Technical Assistance and Information Exchange instrument 

TJTP  

TSI  

Territorial Just Transition Plan 

Technical Support Instrument 

3 

 
 
 
1. 

Introduction 

Structural  reforms  are  essential  for  modernising  economies,  encouraging  investment, 
creating  jobs  and  raising  living  standards.  Reforms  are,  by  their  very  nature  complex 
processes.  To  facilitate  these  reforms  across  the  European  Union,  Regulation  (EU) 
2017/8251  on  the  establishment  of  the  Structural  Reform  Support  Programme  (“the 
SRSP Regulation”) was adopted in May 2017.  

The  objective  of  the  Structural  Reform  Support  Programme  (“the  programme”  or  “the 
SRSP”)  was  to  contribute  to  institutional,  administrative,  and  growth-sustaining 
structural  reforms  in  Member  States,  by  providing  technical  support  to  national 
authorities.  The  budget  for  the  programme  was  EUR  222.8  million  over  2017-20.  The 
programme  was  managed  by  the  Structural  Reform  Support  Service  (SRSS),  which 
subsequently  became  the  Directorate—General  for  Structural  Reform  Support  (DG 
REFORM).2 

This  staff  working  document  provides  an  ex-post  evaluation  of  the  programme.  In 
compliance  with  the  Better  Regulation  guidelines,  this  evaluation  is  based  on  the 
following five criteria:  

•  Relevance  –  do  the  objectives  of  the  SRSP  appropriately  address  the  needs  in 

Member States? 

•  Effectiveness – how successful is the SRSP in achieving or progressing towards 

its objectives? 

•  Efficiency – are the effects of the SRSP achieved at a reasonable cost? 

•  Coherence  –  to  what  extent  are  the  various  components  of  the  SRSP’s 
intervention logic coherent (internal coherence) and how does the SRSP interact 
with other EU interventions with similar objectives (external consistency)? 

•  EU added value – what is the additional value resulting from action at EU level, 
compared to  what  could be achieved by Member States  acting alone  at  national 
and/or regional level? 

This ex-post evaluation aims to help improve the provision of further technical support to 
Member  States  and  to  identify  good  practice  for  designing  a  future  technical  support 
programme in the next programming period (beyond 2027). This report is also intended 
as a tool to share the results of DG REFORM’s action with stakeholders and promote the 
use of evaluations as an evidence-based policy making tool.  

1 Regulation (EU) 2017/825 of the European Parliament and of the Council, OJ L 129, 19.5.2017, p. 1–16, 
as amended by Regulation (EU) 2018/1671, OJ L 284, 12.11.2018, p. 3–5 (the SRSP Regulation).  
2 On 1 January 2020, DG REFORM succeeded the SRSS.  For reasons of simplicity, this document refers 
only to DG REFORM.  

4 

 
 
 
In  accordance  with  Article  16  of  the  SRSP  Regulation,  an  external  contractor  was 
engaged to produce an independent evaluation study (“the study”).  

Box 1 – Study 

The  external  independent  study,  presented  in  annex  of  this  document,  was  carried  out 
over a period of 11 months, starting on 11 October 2021. It included 826 SRSP-funded 
projects from 2017 to 2020, covering beneficiaries in all Member States.  

The contractor deployed a complex methodology – in line with the EU Better Regulation 
Guidelines – to collect robust evidence, consisting of extensive desk research (including 
information  extracted  from  DG  REFORM’s  internal  monitoring  tool  with  data  up  till 
November  2021),  stakeholder  consultation  activities  (interviews  with  key  stakeholders, 
open public consultation, a survey of key stakeholders, and focus groups), and in-depth 
case  studies  (30  projects  in  8  Member  States,  and  1  multi-country  project  covering  17 
Member States). 

The contractor carried out all tasks under the scrutiny of an inter-service steering group 
and  the  guidance  of  DG  REFORM.  The  quality  and  the  representativeness  of  the 
collected  evidence  was  considered  to  be  solid  and  meaningful,  allowing  us  to  draw 
conclusions  and  provide  reliable  and  valid  answers  to  the  evaluation  questions  (see 
section 4). 

Despite the efforts and the robust methodology employed by the contractor, it should be 
stressed  that  assessing  the  long-term  impacts  of  SRSP-supported  projects  remains  a 
challenge.  The  impact  of  the  programme  is  difficult  to  measure  distinctively  and  in 
isolation from the different factors that affect its implementation, such as ownership and 
the financial capacity of Member States to implement the relevant reforms.  

This should be kept in mind when drawing any conclusions about the longer-term impact 
of the programme in terms of achieving its general objectives. 

This document presents and reflects on the main outcomes and findings of the Study, but 
also draws on other DG REFORM sources of information, including additional data from 
the  internal  IT  system,  annual  monitoring  reports,  annual  activity  reports  and  the  mid-
term and ex-ante evaluation of the SRSP.  

This document has the following structure:  

-  Section  2  describes  the  background  of  the  programme  and  summarises  its 

expected outcome.  

-  Section 3 covers the evolution of the situation during the evaluation period. 

-  Section 4 provides the main findings of the evaluation. 

-  Section 5 presents the conclusions and recommendations. 

2.  What was the expected outcome of the intervention? 

The general objective of the SRSP was to contribute to institutional, administrative and 
growth-sustaining  structural  reforms  in  the  Member  States  by  providing  support  to 

5 

 
national authorities. The support may also help Member States preparing for participation 
in the euro area.3  
To  achieve  this  general  objective,  the  SRSP  Regulation  has  set  the  following  specific 
objectives:4 

a)  To support the initiatives of national authorities to design their reforms according 
to  their  priorities,  considering  initial  conditions  and  expected  socioeconomic 
impacts. 

b)  To  support  the  national  authorities  in  enhancing  their  capacity  to  formulate, 
develop  and  implement  reform  policies  and  strategies  and  in  pursuing  an 
integrated  approach  ensuring  consistency  between  goals  and  means  across 
sectors. 

c)  To support the efforts of national authorities to define and implement appropriate 
processes  and  methodologies  by  considering  good  practices  of  and  lessons 
learned by other countries in addressing similar situations. 

d)  To assist the national authorities in enhancing the efficiency and effectiveness of 
human-resource management, inter alia, by strengthening professional knowledge 
and skills and setting out clear responsibilities.  

The Programme intended to provide technical support for the design and implementation 
of reforms that relate to a broad range of policy domains, including: 

•  Revenue administration and public financial management. 
•  Governance and public administration. 
•  Growth and business environment (including climate, energy and environment). 
•  Labour market, education, health and social policy. 
•  Financial services and access to finance. 

Designing  and  implementing  institutional  and  structural  reforms  is  complex  and 
demanding  and  requires  a  high  degree  of  knowledge  and  skills;  the  overall  successful 
implementation  of  reforms  requires  efficient  and  effective  public  administrations.  For 
this  reason,  addressing  the  structural  challenges  faced  by  public  administrations  – 
especially  by  developing  adequate  institutional  and  administrative  capacity  –  is  crucial 
for well-functioning and sustainable economies and the success of structural reforms.  

The  SRSP  ex-ante  evaluation5  identified  the  following  needs  as  more  relevant  with 
regard to Member States’ capacity to implement solid reform agendas: 

3 The general objective as laid down in Article 4 of the SRSP Regulation reads as follows: ‘to contribute to 
institutional,  administrative and growth-sustaining structural reforms in the  Member States by providing 
support  to  national  authorities  for  measures  aimed  at  reforming  and  strengthening  institutions, 
governance,  public  administration,  and  economic  and  social  sectors  in  response  to  economic  and  social 
challenges,  with  a  view  to  enhancing  cohesion,  competitiveness,  productivity,  sustainable  growth,  job 
creation, investment and social inclusion and to contributing to real convergence in the Union, which may 
also  prepare  for  participation  in  the  euro  area,  in  particular  in  the  context  of  economic  governance 
processes, including through assistance for the efficient, effective and transparent use of the Union funds’.  
4 Article 5(1) of the SRSP Regulation.  
5  Commission  staff  working  document  Ex-ante  Evaluation  Accompanying  the  document  Proposal  for  a 
Regulation of the European Parliament and of the Council on the  establishment of the Structural Reform 

6 

 
 
1)  Limited administrative and institutional capacity. 
2)  Challenges  in  the  design  and  implementation  of  growth-sustaining  structural 

reforms in line with the Union’s economic and social goals. 

3)  Limited application and implementation of EU law.  

The  SRSP  provided  demand-driven  and  tailor-made  support  to  the  specific  situation  of 
the concerned country. Actions that were eligible for funding included the formulation of 
policy strategies or roadmaps, the provision of long- and short-term experts, and capacity 
building activities, such as workshops, training and working visits. 

The above  actions  were  anticipated  to  lead to  the short-term expected results  of SRSP, 
also referred to as project outcomes. Achieving these outcomes was expected to lead to 
the  achievement  of  the  SRSP  specific  objectives.  Outcomes  included  the  adoption  of 
strategies,  reform  roadmaps  or  new  legislation;  the  adoption  of  new  or  improved 
procedures  and  methodologies;  organisational  change  and  enhanced  human  resource 
management.  

The  achievement  of  the  outcomes  depended  not  only  on  the  breadth  and  depth  of 
technical support provided to EU Member States, but also on many other factors – such 
as timing and sequencing of reforms, reform ownership by the Member State  – that are 
outside the control of the technical support project. 

Technical support projects could cover any part of the reform cycle, from identifying the 
needs  to  monitoring  and  evaluating  the  outcomes.  At  the  same  time,  the  actual 
implementation of the reforms remained the prerogative of the recipient Member State. 

Figure 1 below presents  the intervention logic of the Programme, in line with the main 
elements of the theory of change.6  

Support Programme for the period 2017 to 2020 and amending Regulations (EU) No 1303/2013 and (EU) 
No  1305/2013.  The  Ex-Ante  Evaluation  identified  specifically  two  challenges/needs:  (a)  Limited 
administrative  and  institutional  capacity  (b)  Inadequate  application  and  implementation  of  Union 
legislation towards achieving the Union's fundamental goals. 
6 According to the Commission’s Better Regulation Guidelines, a theory of change is oriented towards the 
objectives that are intended to be achieved under specific conditions, showing the ways and mechanisms 
that connect individual objectives, activities, and outcomes. 

7 

 
 
 
FIGURE 1: SRSP LOGIC OF INTERVENTION 

Based  on  the  results  of  the  ex-ante  and  mid-term  evaluations  and  of  the Study,  the  ex-
post evaluation considers two key types of impacts: 

- 

Institutional  Impact:  structural  changes  in  the  institutional  and  administrative 
capacity7 of Member States’ authorities to prepare and implement reforms and 
to apply EU law in an effective way. 

-  Policy Impact: achievement of relevant longer-term growth-sustaining reforms 
in EU Member States as indicated in the cooperation and support plans signed 
by DG REFORM and each Member State. 

7 Administrative capacity is considered in terms of structures, human resources, systems and tools, in line 
with  European  Commission  (2020),  Roadmaps  for  Administrative  Capacity  Building:  Practical  Toolkit. 
https://ec.europa.eu/regional_policy/sources/policy/how/improving-investment/roadmap_toolkit.pdf. 

8 

 
  
 
Clearly,  external  factors  such  as  economic,  social  and  institutional  conditions;  the 
election cycle and political support for reforms on all levels of government; the scope of 
reforms;  and  the  COVID-19  crisis  played  a  role  in  contributing  to  or  constraining  the 
achievement of the programme impacts. 

The SRSP Regulation does not specifically refer to any of the Sustainable Development 
Goals  (SDGs).  Nevertheless,  the  SRSP  was  expected  to  contribute  implicitly  to  the 
achievement of several SDGs. According to the Joint Research Centre (JRC) toolkit, the 
programme  was  expected  to  contribute  mainly  to  SDG  8  (decent  work  and  economic 
growth),  SDG  16  (peace,  justice,  and  stronger  institutions)  and  SDG  10  (reduced 
inequalities).  

2.2 Points of comparison 

This  ex-post  evaluation  aims  to  capture  change  that  the  programme  has  brought  over 
time. To do so, it compares the actual performance of the programme against: 

-  The situation before the intervention as described in the ex-ante evaluation of 

the programme (published in November 2015). 

-  The situation during the mid-term evaluation of the programme as described in 
the Study (published in February 2020 using the monitoring data from 2019).  

The ex-ante evaluation of the SRSP 

The  ex-ante  evaluation  of  the  SRSP  noted  that  Member  States  may  face  a  range  of 
challenges  in  the  design  and  implementation  of  structural  and  administrative  reforms, 
such  as  the  limited  administrative  and  institutional  capacity  and  the  inadequate 
application  and 
these 
problems/challenges  that  the  SRSP  was  deemed  to  solve,  was  translated  into  the 
following two parameters:  

legislation.8  Quantification  of 

implementation  of  Union 

-  The  implementation  of  Country  Specific  Recommendations  (CSRs)  of  the 
European  Semester,  indicates  the  difficulties  Member  States  face  in  adopting 
reforms.  For  example,  Member  States  made  at  least  “some  progress”  in  48.4  % 
the CSRs addressed to them in 2015.  In 2019, the implementation rate of CSRs 
was  lower  compared  to  2015,9  thus  suggesting  the  continued  need  for  technical 
support.  

-  The inadequate application and implementation of EU legislation was quantified 

in terms of formal infringement procedures.10  

1305/2013, 

8  European  Commission  (2015),  Ex-ante  Evaluation  Accompanying  the  document  Proposal  for  a 
Regulation of the European Parliament and of the Council on the establishment of the Structural Reform 
Support Programme for the period 2017 to 2020 and amending Regulations (EU) No 1303/2013 and (EU) 
No 
https://eur-lex.europa.eu/legal-
content/EN/TXT/PDF/?uri=CELEX:52015SC0750&from=EN  
9 No/limited progress was achieved for 60.2% of CSRs, some progress was achieved for 38.7% of CSRs, 
and  full/substantial  progress  was  achieved  for  1.1%  of  CSRs.  European  Parliament  (2020),  Country-
specific 
7. 
overview 
https://www.europarl.europa.eu/RegData/etudes/BRIE/2018/624404/IPOL_BRI(2018)624404_EN.pdf   
10 The number of total infringement cases stated in the ex-ante evaluation report was 1,347 at the end of 
2014. Ibid, p. 6.  

recommendations: 

September 

2020, 

An 

p. 

p. 

6. 

– 

9 

 
 
The ex-ante evaluation of the programme, envisaged that the SRSP will accelerate these 
reforms  and  ensure  their  quality  and  sustainability,  thus  having  a  considerable  positive 
impact  on  increasing  economic  and  societal  welfare  in  the  Member  States  in  the  long 
term.11  Increased  welfare  of individuals  and societies through non-market  benefits  (e.g. 
improvements in education, skills, social protection, health, long-term care) or improved 
competitiveness  of businesses through market  efficiency can only  be treated as indirect 
benefits of the SRSP, since the programme focuses on the reform process.  

The mid-term evaluation of the SRSP 

By the time of mid-term evaluation study, only 22% of the SRSP projects (54 out of 246) 
included in its scope were completed12, therefore, it was too early to measure the results 
and  impact  of  the  programme.  However,  the  mid-term  evaluation  found  general 
improvement  in  the  administrative  and  institutional  capacity  (by  73.7%  of  the 
stakeholders from the completed projects), the quality of the design and implementation 
of structural reforms (60.5%) and the application and implementation of EU law (42.1%) 
based on the results of stakeholder consultation.13  

3.  How has the situation evolved over the evaluation period? 

The submission, analysis and selection of technical support requests under the SRSP was 
organised in annual cycles, in accordance with the SRSP Regulation.  

Each  year,  Member  States  submitted  their  requests  for  technical  support  by  the  end  of 
October, as specified in the SRSP Regulation.14 Once they had done so, the Commission 
assessed these requests against the criteria set out in Article 7 of the SRSP Regulation.15 
The  Commission  also  verified  whether  requests  overlapped  with  actions  implemented 
under other EU instruments or programmes, with a view to avoiding double funding and 
ensuring complementarity.  

Thereafter,  the  Commission  selected  the  requests  that  were  assessed  to  be  a  highest 
quality  in  line  with  the  set  criteria.  The  technical  support  projects  that  were  financed 
through  the  Programme  did  not  require  co-financing  by  the  Member  State  concerned. 

1305/2013. 

(2016  ECFIN  009/A).  Publications  Office  of 

11  European  Commission  (2015),  Ex-ante  Evaluation  Accompanying  the  document  Proposal  for  a 
Regulation of the European Parliament and of  the Council on the establishment of the Structural Reform 
Support Programme for the period 2017 to 2020 and amending Regulations (EU) No 1303/2013 and (EU) 
No 
https://eur-lex.europa.eu/legal-
content/EN/TXT/PDF/?uri=CELEX:52015SC0750&from=EN  
12 Ernst & Young (2018), Mid-term Evaluation of the Structural Reform Support Programme (SRSP) 2017- 
2020 
the  European  Union,  p.  13,  17. 
https://doi.org/10.2887/656262 This analysis was based on the information available on Jira, consulted on 
30 April 2019.  
13 The consultation conducted during the mid-term evaluation involved an online targeted consultation and 
an  in-depth analysis  of 30  implemented projects.  Opinions  were  collected  from  the  beneficiaries  and  the 
providers of technical support during the consultation process. Ibid, p. 17.  
14  The  provision  of  support  under  Article  11  (support  to  be  funded  through  additional  voluntary 
contributions)  follows  a  different  timeline,  linked  to  the  transfer  of  the  additional  contributions  from  the 
Member State concerned. 
15 Urgency, breadth and depth of the problems identified, the support needs in the relevant policy area, the 
analysis  of  socioeconomic  indicators,  and  the  administrative  capacity  of  the  beneficiary  authority 
concerned. 

10 

 
 
The  SRSP  was  implemented  via  annual  work  programmes  that  served  as  a  financing 
decision. They set  out  the support  measures  to  be implemented  for the year concerned, 
the dedicated annual budget, and the expected results, as well as the priority actions. 

Subsequently, the Commission came to an agreement, titled as “Cooperation and Support 
Plan”  (CSP),  with  each  Member  State.  The  CSPs,  co-signed  annually,  covered  the 
priority  areas,  the  objectives,  an  indicative  timeline,  the  scope  of  the  support  measures 
and the estimated financial contribution. 

To  facilitate  efficient  communication  with  the  Commission  and  the  coordinated 
submission  of  technical  support  requests,  each  Member  State  designated  a  national 
Coordinating  Authority  within  its  government  structures.  The  Coordinating  Authority 
acted as the interlocutor of the Commission for the implementation of the CSP, and the 
SRSP in  general.  It  was  responsible for collecting all requests for the Member State in 
question, and subsequently submitting the requests to the Commission.  

The  Coordinating  Authority  had  the  opportunity  to  prioritise  the  requests  before 
submitting  them  to  the  Commission.  The  national  beneficiary  authorities  (such  as 
ministries,  public  agencies,  regulatory  authorities,  regional/local  authorities  and  other 
public  bodies  that  have  requested  assistance)  were  the  actual  recipients  of  technical 
support.  In  line  with  the  Regulation,  the  SRSP  never  provided  financial  support  to  the 
authorities, but only technical support.  

the  effective 
The  Coordinating  Authority  was 
implementation of the support measures in the country and was also encouraged to take 
appropriate action in case of challenges during the implementation of technical support in 
its country.  

for  coordinating 

responsible 

Technical support, such as strategic and legal advice, studies, training, and expert visits 
on  the  ground,  was  delivered  by  technical  support  providers  contracted  by  the 
Commission.  DG  REFORM  carried  out  all  financial  management  tasks  related  to  the 
contracts. 

The  Commission  was  responsible  for  monitoring  the  implementation  of  the  actions 
financed by the programme and measuring the  achievement of the general  and specific 
objectives of the SRSP in accordance with indicators set out in the Annex of the SRSP 
Regulation and monitoring the achievement of EU added value.  In addition to the mid-
term and ex post evaluation, the Commission also followed other mechanisms to monitor 
the  Programme’s  progress,  including  annual  monitoring  reports,  annual  activity  reports 
and the overall strategic planning and programming cycle.  

To  monitor  the  implementation  of  the  programme,  DG  REFORM  developed  and 
effectively  deployed  an  internal  monitoring  system16,  using  two  types  of  feedback 
questionnaires:  (1)  satisfaction  questionnaires  that  were  sent  to  main  stakeholders 
(beneficiary  authorities,  providers,  DG  REFORM  policy  officers)  at  the  end  of  the 
project  to  assess  the  satisfaction  level  with  the  design  and  the  implementation  of  the 

16  The  external  contractor  used  data  from  the  internal  monitoring  system  of  DG  REFORM  (JIRA)  to 
compile this Report. 

11 

 
 
project and (2) outcome questionnaires that were sent to beneficiary authorities within 6, 
12 or 18 months after the project completion to assess the achievement level of expected 
outcomes. 

Current state of play  

The demand for the SRSP was high and increased throughout SRSP implementation. In 
line with the principle of sound financial management,  some of the criteria of selection 
were  the  focus  and  maturity  of  requests  that  allowed  for  quick  delivery  of  support  and 
rapid  implementation  of  impactful  reforms.  Table  1  below  provides  the  number  of 
technical support requests received and selected per SRSP cycle. 

TABLE 1: TECHNICAL SUPPORT REQUESTS RECEIVED AND SELECTED 

SRSP 2017 
SRSP 2018 
SRSP 2019  
SRSP 2020  
Total 

Technical support requests 
approved 
submitted 
159 
271 
146 
444 
263 
580 
227 
609 
795 
1.904 

Ratio of approved 
requests 
59% 
33% 
45% 
37% 
42% 

Nr of submitting 
Member States 
16 
24 
27 
27 

Source: DG REFORM internal monitoring system. 

Once  requests  for  technical  support  were  approved,  they  became  projects.17  However, 
DG  REFORM  could  decide  to  implement  one  large  request  through  more  than  one 
project, or to implement multiple related requests through one single project. Therefore, 
the  number  of  selected  requests  could  –  and  usually  did  –  differ  from  the  number  of 
projects that were implemented.  

In  addition  to  the  four  annual  SRSP  rounds,  technical  support  projects  have  also  been 
funded  by  the  voluntary  contributions  of  Greece  (2018  and  2020)  and  Bulgaria  (2018) 
under Article 11 of the SRSP Regulation.  

A project could have one or more technical support providers, depending on the expertise 
that  is  required  to  address  the  needs  of  the  Member  State.  The  type  of  provider 
determined the contractual procedure. The type of contract/procedure used to deliver the 
technical support project was referred to as the means of delivery.  

When defining the selected request/projects by their circumstances, the majority of the 
82618  projects  covered  the  ex-post  evaluation  were  related  to  implementing  challenges 
identified in the context of economic governance processes (see table 2 below).  

17 Technically a project is a ‘support measure’ under the SRSP Regulation terminology. 
18 The present figure of 826 used in the ex-post evaluation refers to the actual number of funded projects 
(i.e.,  the  total  number  of  the  projects  funded  during  2017-2020  after  merging  some  of  the  selected 
requests), excluding the projects implemented by the contractor. 

12 

 
 
 
 
 
TABLE 2: CIRCUMSTANCES OF THE SELECTED REQUESTS UNDER THE SRSP 

CIRCUMSTANCES 
THE 
REQUESTS 

OF 
SELECTED 

Economic 
including CSRs 

governance  process, 

EU priorities 

EU law 

Member States’ own initiative 

Economic adjustment programmes 

SRSP 2017  SRSP 2018  SRSP 2019  SRSP 2020 

51% 

27% 

7% 

11% 

4% 

55% 

29% 

6% 

7% 

3% 

62% 

21% 

5% 

8% 

4% 

58% 

30% 

2% 

5% 

5% 

Source: SRSP annual monitoring reports 2017-2020. 

With respect to the thematic areas covered by SRSP, the largest number of projects was 
implemented  in  the  field  of  sustainable  growth  and  business  environment  (which 
includes  climate  change  and  green  issues),  and  labour  market,  education,  health,  and 
social services. 

FIGURE 2: SRSP-FUNDED PROJECTS BY THEMATIC AREA, 2017-2020 

Source: DG REFORM internal monitoring system (N=826). 

The SRSP covered projects in all Member States. Over the period of 2017-2020, Greece, 
Croatia and Romania had the highest numbers of SRSP-funded projects. Fewer technical 
support projects were implemented in the Nordic countries, Luxembourg, Germany and 
the  Netherlands,  which  were  consistently  the  countries  with  the  lowest  number  of 
submitted requests.  

13 

 
 
FIGURE 3: NUMBER OF REQUESTS SELECTED PER MEMBER STATE UNDER SRSP BY YEAR, 2017-
2020 

100
90
80
70
60
50
40
30
20
10
0

32

15

34

9

e
c
e
e
r
G

17

12
20 14

12 13

14 13

a
i
t
a
o
r
C

i

a
n
a
m
o
R

15

18

7
9

y
l
a
t
I

15 10

9

11 10 12 10
8
9
9

15 17 12 12
9
5
7
7
6
10 11 13 10 10

5

a
i
v
t
a
L

d
n
a
o
P

l

i

a
n
e
v
o
l
S

i

a
h
c
e
z
C

a
i
r
a
g
u
B

l

i

a
n
a
u
h
t
i
L

8

12

8

s
u
r
p
y
C

7
9
7

13

a
t
l
a
M

14 12

16

6

i

n
a
p
S

10
8
5

a
i
k
a
v
o
l
S

11 10

12 13
4
4

7

l

a
g
u
t
r
o
P

i

m
u
g
e
B

l

10
7
5
3
a
n
o
t
s
E

i

9
9
5

d
n
a
e
r
I

l

7
7
3
y
r
a
g
n
u
H

4
7
4
a
i
r
t
s
u
A

6
6
1
e
c
n
a
r
F

5
3
d
n
a
n
i
F

l

2
2
1
y
n
a
m
r
e
G

2
2
1
n
e
d
e
w
S

4
2

s
d
n
a
l
r
e
h
t
e
N

2
k
r
a
m
n
e
D

1
2
1
g
r
u
o
b
m
e
x
u
L

2017

2018

2019

2020

Source: DG REFORM internal monitoring system (N=826).  
Note: the number of requests in the countries with the lowest number of requests is presented in the brackets. The first 
figure in the brackets stands for the number of requests selected in 2020. 

Several delivery modes were used during the implementation of the SRSP, namely: 

•  Direct  management,  which  includes  public  procurement  contracts,  grants  and 

• 

reimbursement of costs incurred by external experts.  
Indirect management, through entities that have undergone an ex-ante assessment 
of  their  procedures  and  systems,  guaranteeing  a  level  of  protection  of  the  EU 
financial  interests  equivalent  to  that  guaranteed  by  the  Commission  in  direct 
management.  

•  Technical  Assistance  and  Information  Exchange  instrument  (TAIEX),  which 
provides short-term expertise exchange by public sector experts from the Member 
States,  through  a  Service  Level  Agreement  between  DG  REFORM  and  DG 
NEAR.19  

A technical  support project  could  consist  of  a mix  of different  delivery  modes to  tailor 
the support to the needs of the Member State; however, 78% of projects  used only one 
specific  delivery  mode.  In  total,  there  were  1,037  individual  delivery  modes  applied 
during the programme’s implementation in 2017-2020. 

19 European Commission (2020), Commission Staff working document: Mid-term evaluation of the 
Structural Reform Support Programme. https://ec.europa.eu/info/sites/default/files/report_en_final.pdf  

14 

 
 
 
 
 
 
FIGURE 4: USE OF MAIN DELIVERY MODES DURING THE IMPLEMENTATION OF THE SRSP, 2017-
2020 

416

332

500

400

300

200

100

0

171

54

25

Procurement

Grants

TAIEX

Delegation

Commission
expertise

Source: DG REFORM internal monitoring system (N=1,037).  

A  growing  number  of  procurement  procedures  were  used  during  2019  and  2020, 
following the substantial increase in the budget of the programme in these two years. The 
relative share of this delivery mode (out of all the delivery modes that year) was 37% in 
the 2017 round, 34% in 2018, 43% in 2019 and 43% in 2020.  

The  selection  of  an  individual  delivery  mode  or  a  support  provider  depended  on  the 
needs  of  Member  States  and  the  nature  of  activities/outputs  planned  in  each  project. 
Public  procurement  via  framework  contracts  with  private  companies  was  mostly  used 
when project activities required many experts, interdisciplinary teams and other resources 
on  the  ground,  close  contacts  with  national  authorities  and  good  knowledge  of  the 
national  language.  In  contrast,  grants  and  delegation  agreements  with  international 
organisations were usually used when an exchange of international good practice and a 
transfer of knowledge from other countries was key to achieving the project objectives.  

Main outputs and results achieved during the implementation of the programme  

According to the project descriptions, the SRSP was expected to produce 1,462 outputs 
in 2017-2020.  

the  outputs 

to  produce  were 
Most  of 
‘recommendations’,  ‘analysis  and  reports’,  ‘workshops’,  ‘action  plans  and  roadmaps’ 
(see Figure 5).20  

the  programme  was  expected 

that 

Overall, there were 1,352 technical support activities planned for implementation under 
the  four  rounds  of  the  SRSP.21  The  Programme  most  often  planned  the  following 
activities:  study,  research,  evaluation  (313  such  activities  within  826  projects)  and 
workshops,  conferences  and  seminars  (270  such  activities).  Long  term  expert  missions 
(26  activities)  and  organisation  of  local  operations  (27  activities)  were  the  least 
frequently used activities in the technical support projects under the 2017-2020 rounds. 
76% of the SRSP projects included more than one activity. 

20 Within the scope of the SRSP, recommendations are non-binding suggestions that are addressed to the 
beneficiary authorities to tackle the problems they highlighted in their request for support,  or to promote 
their reform agenda. 

21 The number of annual activities increased from 281 under the 2017 round to 339 under the 2018 round 
and to 432 under the 2019 round, with some decrease in the volume of activities (to 316) under the 2020 
round due to the selection of fewer larger projects. Similarly, the average number of activities per project 
increased from 2017 to 2020 and this increase continued across all the rounds – around 2.4, 2.8, 3.1 and 3.2 
activities  per  project  respectively  in  each  round.  The  monitoring  system  of  DG  REFORM  contains  12 
kinds of SRSP eligible activities.  

15 

 
 
 
The  technical  support  projects  became  more  complex  and  significant  over  time,  as 
evidenced by an increasing number of activities and outputs per project,  as they started 
addressing more important reform needs. According to the results of the interviews, the 
financial  size  of  the  projects  was  also  matched  by  a  substantial  improvement  in  the 
quality of requests for technical support from EU Member States.  

FIGURE 5: TOTAL EXPECTED OUTPUTS OF THE SRSP, 2017-2020 

Terms of reference

16

Legislative proposal

Communication strategy (including material) and other
events

39

45

Study visit reports

Guidelines, handbook

Action plan, roadmap

Workshops (including material), training, training
material

Analysis, report

Recommendation

110

153

207

234

330

334

Source: DG REFORM based on internal monitoring data (N=1,462). Note: the categories of ‘other outputs’ and ‘none 
of the outputs’ were excluded from the analysis. 

In the monitoring system of DG REFORM, projects could be associated to four groups of 
pre-defined outcomes (results), which corresponded to the four specific objectives of the 
Programme.  Most  often  the  Programme  supported  the  achievement  of  the  following 
outcomes: ‘adoption of (new) procedures and actions to enhance the implementation of 
reforms’  and  ‘improved  internal  working  procedures,  methodologies  and  processes, 
organisation’. A similar distribution was observed during the mid-term evaluation of the 
2017-2018 Programme. 22 

FIGURE 6: BREAKDOWN OF THE PRE-DEFINED OUTCOMES IN THE SRSP, 2017-2020 

447

447

267

119

Adoption of (new)
procedures and actions to
enhance the
implementation of reforms

Improved internal working
procedures,
methodologies and
processes, organisation

Adoption of a strategy, a
new law /act or
modification of an existing
one

Organisational change,
change management,
improved human resource
management

(2016  ECFIN 

22 Ernst & Young (2018). Mid-term Evaluation of the Structural Reform Support Programme (SRSP) 2017-
2020 
159. 
https://doi.org/10.2887/656262. Since this analysis was based on the results of a survey, the formulations 
of  the  expected  outcomes  do  not  fully  correspond  to  those  provided  in  the  monitoring  system  of  DG 
REFORM.  

009/A).  Publications  Office 

the  European  Union, 

of 

p. 

16 

 
 
 
 
Source: DG REFORM internal monitoring system (N=1,338). 

4.  Evaluation findings (analytical part) 

This ex-post evaluation assesses the performance of the SRSP against the following five 
criteria: effectiveness, efficiency, coherence, EU added value, and relevance. The overall 
analysis is  based on evidence from both  the external  Study  and the Commission’s own 
sources.  

4.1.  To what extent was the intervention successful and why? 

4.1.1.  Effectiveness: How effective has the SRSP been in achieving its objectives? 

The overall assessment on the Programme’s effectiveness is analysed threefold:  

•  The  extent  to  which  the  programme  contributed  to  its  specific  objectives  and 
achieved  its  expected  results,  as  well  as  to  the  institutional  and  policy  changes 
that occurred during the implementation of the programme. 

•  The  extent  to  which  the  programme  contributed  to  its  expected  impact,  both  in 

• 

terms of structural reforms and capacity building. 
Identification  of  factors  that  either  facilitated  or  stalled  progress  towards  the 
results and impact.  

Contribution of SRSP actions to the expected results 

The  SRSP  had  four  specific  objectives  to  be  pursued  in  close  cooperation  with 
beneficiary  Member  States.  According  to  the  Study  (see  Figure  7),  technical  support 
projects  most  often  contributed  to  specific  objective  2  and  3.  The  programme  less 
frequently  contributed  to  objective  4  because  this  very  specific  objective  was 
implemented  mostly  through  the  technical  support  in  the  single  field  of  public 
administration and governance. 

FIGURE 7: CONTRIBUTION OF TECHNICAL SUPPORT PROJECTS TO THE SPECIFIC OBJECTIVES OF 
THE PROGRAMME IN 2019 AND 2021 

70%

60%

50%

40%

30%

20%

10%

0%

61%

55%

56%

59%

43%

42%

14%

11%

OBJECTIVE 1. Support
to design of reforms

OBJECTIVE 2. Support
to the development and
implementation of
reform policies

OBJECTIVE 3. Support
to the definition and
implementation of
processes and
methodologies

OBJECTIVE 4. Support
to more efficient and
effective human
resource management

2019 mid-term evaluation

2021 ex-post evaluation

17 

 
 
Source: the mid-term evaluation of the SRSP (data from 2019), the ex-post evaluation of the SRSP (data from 2021). 
The 2021 data does not include 11 cases where the projects are contributing to none of the specific objectives (N=7) or 
contributed to other objectives (N=4).  

Data  from  the  DG  REFORM  evaluation  system  and  the  subsequent  analysis  of  the 
external contractor, suggests that the specific outcomes of projects were to a large extent 
achieved. In particular, the average score for achieving the specific outcomes (results) of 
the SRSP-funded projects  was  8.2 out  of 10  (on  a scale  from  1 to  10).23 Two thirds of 
technical  support  projects  achieved  their  outcomes  (based  on  average  scores  8.0  and 
above per project).24  

Beneficiary  authorities  also  generally  considered  the  technical  support  projects  to  be 
successful.  According  to  the  results  of  DG  REFORM  satisfaction  questionnaires,  both 
beneficiary authorities and DG REFORM policy officers assessed positively the success 
of  the  technical  support  projects  (average  score  was  8.7  for  both  dimensions),  with  the 
success of the projects depending mostly on the effectiveness of the interactions with the 
support providers, beneficiary authorities and DG REFORM. 

Overall,  the  programme’s  contribution  to  the  achievement  of  the  expected  results  was 
possibly due to the successful implementation of technical support activities and the good 
achievement of the intended outputs (deliverables), according to the case study analysis. 

The  achievement  of  the  project  outputs  and  results  depended  on  the  quality  of 
technical  support  provided  on  the  ground,  including  the  providers’  expertise  and 
skills.25  The  Study26  also  hints  that  the  quality  of  technical  support  depended  on  the 
thematic and national expertise of support providers, their skills in project management 
and  communication,  as  well  as  their  ability  to  adapt  to  any  changes  in  the  project 
environment.  Conversely,  the  absence  of  these  conditions,  language  barriers  or 
fragmentation  of  responsibilities  sometimes  had  a  detrimental  effect  on  the  quality  of 
technical support.  

National  authorities  were  closely involved in  the project  pre-implementation  phase;  the 
ex-post  evaluation  suggests  that  in  future  programmes,  national  authorities  should 
continue  to  be  closely  involved  in  the  pre-implementation  phase,  in  terms  of  being 
consulted  on  a  suitable  type  of  provider  and/or  contributing  to  the  drafting  of  project 
description/specifications. The latter is even more important when national priorities and 
needs  undergo  substantial  changes  following  the  selection  of  requests  for  technical 
support, e.g., due to election cycle or change in hierarchy in the beneficiary authority.  

23 N=128; with the total number of outcomes being 237. 
24  Interestingly,  the  achievement  of  the  first  project  outcome  was  above  the  average  (the  score  of  8.45), 
while other outcomes came below the average (ranging from 8.07 to 7.55), pointing to potential difficulties 
in the implementation of more complex projects. 
25 According to the targeted survey of the external study, 88% of beneficiary authorities strongly agreed or 
agreed (109 out of 124) that the providers of technical support had the required expertise and skills, and 
86% of them accordingly agreed that they had delivered according to the expected quality (106 out of 123). 
These  findings  were  also  supported  by  those  coordinating  authorities  and  beneficiary  authorities  who 
participated  in  the  focus  group  discussions,  who  agreed  that  the  international  experts  selected  by  the 
Commission had the right competencies. 
26 According to the results of the case studies. 

18 

 
 
Overall, the SRSP generally contributed to the results expected, in terms of design 
and  implementation  of  institutional,  administrative  and  structural  reforms  in  EU 
Member States. The SRSP actions achieved to large extent their expected outcomes and 
the Programme’s specific objectives. Some variations were however observed across the 
different specific objectives and expected results of the programme.  

The beneficiary authorities and coordinating authorities were overall satisfied with their 
participation  in  the  SRSP  and  with  the  results  of  technical  support  projects.  The 
achievement of the project results, and consequently of the Programme results, depended 
on the quality of technical support provided on the ground, including the providers’ skills 
and expertise.  

Contribution of SRSP actions to the expected impacts 

The  ex-post  evaluation  of  SRSP’s  contribution  to  its  expected  impacts  faced  a  few 
limitations/challenges. In particular:  

-  A  considerable  share  (about  30%)  of  projects  were  still  ongoing  during  the 

external study27.  

-  Evidence  also  suggests28  that  most  projects  implemented  under  the  SRSP 
concerned policy development. Transforming these types of outputs into impacts 
in terms of institutional and policy change takes a lot of time. In parallel, external 
factors  affect  more  the  achievement  of  longer-term  growth-sustaining  reforms, 
compared to short term reforms/results. Consequently, the external study finalised 
in October 2022 did not observe many impacts (especially long-term and policy 
impacts),  because  such  impacts  are  inevitably  expected  to  arise  at  a  later  stage 
and also because they are subject to many confounding factors.  

The ex-ante  evaluation  of the SRSP  concluded in 2015 that the  provision of technical 
support  for  the  design  and  implementation  of  reforms  will  accelerate  their 
implementation and ensure their quality and sustainability.29  

Consequently,  the  Study  also  explored  SRSP’s  contribution  to  the  design  and 
implementation of specific types of reforms and policies. The data30 shows a substantial 
contribution  of  the  programme  to  reforms  of  public  administration;  in  particular,  since 
DG REFORM is the competence centre on public administration and governance in the 

27  According  to  the  targeted  survey  of  beneficiary  authorities,  as  many  as  30%  (39  out  of  131)  of  all 
evaluated  projects  were  still  ongoing  during  the  ex-post  evaluation.  Only  31%  (40  out  of  129)  of  all 
surveyed projects achieved all results by May-June 2022. This is in line with the monitoring data of DG 
REFORM that 79% of all SRSP projects were finished in September 2022. 
28 Based on the targeted survey of beneficiary authorities  76% (100 out of 129) of projects implemented 
under  the  SRSP  concerned policy  development  (analysis  of  the  situation,  identification of  best practices, 
definition of a strategy, advice on policy options and recommendations, etc.)  
29  European  Commission  (2015),  Ex-ante  Evaluation  Accompanying  the  document  Proposal  for  a 
Regulation of the European Parliament and of the Council on the establishment of the Structural Reform 
Support Programme for the period 2017 to 2020 and amending Regulations (EU) No 1303/2013 and (EU) 
https://eur-lex.europa.eu/legal-
No 
content/EN/TXT/PDF/?uri=CELEX:52015SC0750&from=EN  
30 46% (41 out of 89) beneficiary authorities agreeing to a large extent or to some extent. 

1305/2013. 

19 

 
 
European  Commission,  it  was  able  to  exploit  several  synergies  in  providing  support  in 
this field (see box 2).  

The Study concludes31 that the SRSP improved all types of administrative capacities. 
It  contributed  more  to  the  adoption  of  different  administrative  systems  and  tools 
(different  instruments,  methods,  guidelines,  manuals,  systems,  procedures,  forms,  etc.) 
compared  to  the  other  two  types  of  administrative  capacities  (institutional  structures, 
human  resources).  The  report  also  observed  that  the  programme  strengthened  human 
resources’  capacity  to  implement  reforms  more  significantly  in  those  beneficiary 
authorities that implemented three or more technical support projects. 

BOX 2 –  The  provision of technical and other  support to EU Member States  in 
the field of governance and public administration 

In the field of governance and public administration, DG REFORM provided specific 
support  to  many  Member  States,  for  instance,  by  improving  the  recruitment  and 
training of senior civil servants; increasing efficiency and the modernisation of public 
administrations;  improving  Member  States’  policy-  and  law-making  processes, 
strategic planning and internal procedures, and capacity to monitor and evaluate public 
policies  and  reforms.  Most  of  these  projects  were  implemented  in  the  countries  of 
Southern  Europe  and  Central  and  Eastern  Europe  (as  opposed  to  the  countries  of 
Western Europe and the Nordic countries).  

Since  DG  REFORM  also  performs  the  role  of  a  Competence  Centre  on  Public 
Administration  and  Governance  in  the  European  Commission,  the  provision  of 
technical  support  was  sometimes  complemented  by  the  implementation  of  other 
activities  (sharing  knowledge  and  evidence,  providing  peer-based  guidance  or 
fostering exchanges of good practices) in EU Member States.  

As  a  consequence  of  the  activities  carried  out  with  the  Programme,  in  2021,  the 
Commission established an expert group on public administration and governance that 
will serve as a platform  to  facilitate the dialogue between the European  Commission 
and representatives of the Member States (including such issues as better tailoring the 
technical and financial support of the EU to the needs of the Member States).  

Sources: European Commission (2021), Annual Activity Report 2020: DG REFORM – 
Directorate-General  for  Structural  Reform  Support,  p.  17;  European  Commission 
(2021), Supporting public administrations in EU Member States to deliver reforms and 
prepare  for  the  future.  https://ec.europa.eu/info/supporting-public-administrations-eu-
member-states-deliver-reforms-and-prepare-future_en;  the  results  of  the  interviews 
with the Commission officials.   

The  SRSP  contributed  to  the  implementation  of  administrative  reforms  and 
capacity building more than to other structural reforms in individual EU Member States. 
This is related to the fact that other structural reforms, which encompass different types 
of reforms (e.g., product market reforms, labour market reforms or fiscal, tax and pension 
reforms)32,  are  usually  complex,  challenging  and  far-reaching  initiatives  aimed  at 

31 According to the targeted survey data 
32  Based  on  the  Commission’s  (from  DG  ECFIN)  methodologies  for  assessing  the  impact  of  structural 
reforms,  we  drew  a  difference  between  product  market  reforms  (addressing  cost  of  starting  a  business; 
administrative  burden;  sectorial  regulation;  and  access  to  finance);  labour  market  (including  education) 
reforms (unemployment benefit reforms; reforms to active labour market policies; education reforms; etc.); 

20 

 
 
promoting sustainable longer-term growth, employment and competitiveness, especially 
compared  to  administrative  changes  or  institutional  capacity  building  whose  benefits 
could occur over a shorter period of time.  

Furthermore,  the  case  study  analysis33  indicated  that,  at  the  time  of  the  ex-post 
evaluation, for 73% of the technical support projects covered in the case studies expected 
changes had occurred totally or moderately and/or were probable to still occur (22 out of 
30).  The  fact  that  the  change  did  not  fully  occur  under  several  projects  during  their 
implementation  was  related  to  the  nature  of  these  projects:  most  of  them  were 
preparatory  projects  intended  to  design  roadmaps,  prepare  guidelines,  introduce  new 
procedures  or  develop  target  values  for  measurement.  Therefore,  it  is  possible  that  the 
use  of  these  outputs  and  some  follow-up  activities  (e.g.,  new  legislation  or  additional 
projects) could produce additional policy or institutional changes in the future.  

Overall,  the  programme  to  some  extent  contributed  to  structural  changes  in  the 
institutional  and  administrative  capacity  of  Member  States’  beneficiary  authorities  to 
prepare  and  implement  reforms  and/or  adequately  apply  EU  law.  Evidence  is  more 
limited with  regard  to  the achievement of  other longer-term  growth-sustaining reforms, 
also because of the inherent difficulty to assess and attribute the long-term impact of the 
SRSP projects on those reforms.  

The materialisation of the expected results and impacts was found to be proportionate to 
the  level  of  progress  in  the  programme’s  implementation,  considering  the  long  logical 
chain between the provision of technical support and the expected results/impacts. Where 
the projects have not generated the full intended results and impacts, this tends to reflect 
wider  challenges  encountered  in  the  process  of  administrative  or  structural  reforms  -
where other confounding factors and external conditions are in place. 

Factors facilitating or constraining the achievement of expected results and impacts.  

Designing  and  implementing  reforms  is  a  complex  and  long  process  whose  success 
depends on a combination of external and internal factors and cannot be attributed only 
to the outputs of technical support projects.  

The  Study  suggests  that  close  cooperation  between  the  European  Commission, 
beneficiary authorities and providers of technical support was an important factor 
having  a  moderate  or  strong  positive  effect  during  implementation34.  This  is 
particularly  significant  when  technical  support  was  provided  via  public  procurement, 
such as in the case of the of the projects dedicated for the preparation of territorial Just 
Transition  Plans  (TJTPs).  Also,  this  cooperation  was  especially  positive  in  those  EU 

as well as fiscal, tax and pension reforms. European Commission (2016), The Economic Impact of Selected 
Structural Reform Measures in Italy, France, Spain and Portugal, European Economy. Institutional Paper 
23. https://doi.org/10.2765/195899 
33 See Annex II: Methodology and analytical models used 

34 A total of 78% (91 out of 116) of beneficiary authorities and 89% of technical support providers (87 out 
of 98) from the survey programme of the external contractor.  

21 

 
 
countries  with  the  higher  number  of  technical  support  projects  during  the  2017-2020 
period.35  

Positive  cooperation  was  obtained  through  regular  meetings  of  steering  committees 
comprised of representatives of DG REFORM,  beneficiary authorities and providers of 
technical  support,  as  well  as  other  ‘tripartite  meetings’  involving  these  organisations. 
During  the  implementation  of  technical  support  projects,  DG  REFORM  policy  officers 
not only executed managerial tasks but were regularly involved in the activities to build a 
consensus on reforms at the national level.  

At  the  same  time,  case  study  interviews  suggest  that  the  increasing  workload  of 
responsible DG REFORM policy officers and  the staff turnover reduced their ability to 
provide  tailor-made  and  swift  expertise  and  support  during  the  provision  of  technical 
support on the ground. 

In  addition,  the  Study  suggests  that  consultation  with  different  stakeholders  during 
the  reform  process  was  an  important  factor  having  a  moderate  or  strong  positive 
effect36.  The  study  also  indicates37  the  positive  effect  of  partnership  among  national, 
regional and local authorities during the reform process. For instance, evidence suggest 
to the importance of involving a broad range of stakeholders during the implementation 
of the TJTP projects.  

The  COVID-19  pandemic  had  a  moderate  or  strong  negative  effect  on  the  delivery  of 
technical  support,  delaying  the  implementation  of  some  activities  or  weakening  the 
ownership  of  some  projects  because  of  a  lack  of  in-person  meetings  and  face-to-face 
interaction.38  Generally,  the  providers  of  technical  support  maintained  high  levels  of 
service to the extent possible, also moving activities online, thus providing flexibility and 
producing additional digital learning resources that could be used in follow-up activities.  

Resistance  to  change  at  national  level  was  one  of  the  negative  factors  (having  a 
moderate  and  strong  negative  effect)  during  the  implementation  of  project 
activities.  39  It  also  indicated that some public sector employees and other stakeholders 
opposed  the  implementation  of  structural  or  administrative  reforms  in  different  EU 
countries.  

According  to  the  study,  the  resistance  could  however  be  alleviated  by  consulting  and 
communicating  with  the  key  groups  of  stakeholders  throughout  the  project  and  the 
reform  process.  This  supports  previous  observations  that  effective  consultation  and 

35 Greece, Croatia, Romania, Italy, Cyprus, Latvia, Poland, Lithuania; each of these countries implemented 
more than 40 projects during this period. 
36 A total of 79% (89 out of 113) of beneficiary authorities and 83% of technical support providers (79 out 
of 95) 
37 63% (65 out of 104) of beneficiary authorities and 73% (66 out of 91) of technical support providers, 
stated a moderate or strong positive effect respectively. 
38 A total of 51% of beneficiary authorities (55 out of 107) and 63% of technical support providers (55 out 
of  87)  suggested  that  the  COVID-19  pandemic  was  an  important  factor  having  a  moderate  or  strong 
negative effect on the delivery of technical support. The focus group discussions and interviews conducted 
with  national  stakeholders  indicated  that  the  pandemic  delayed  the  implementation  of  some  activities  or 
weakened  the  ownership  of  some  projects  because  of  a  lack  of  in-person  meetings  and  face-to-face 
interaction. 
39 38% (40 out of 105) of beneficiary authorities from the targeted survey  

22 

 
 
communication with various reform stakeholders can help overcome resistance and 
increase the long-term sustainability of reforms.40 

The  political  cycle  might  also  create  opportunities  for  reforms.41  Having  the  necessary 
political  backing  was  identified  as  a  pre-condition  for  the  success  of  successful 
projects. Conversely, there were examples of political instability at the national level that 
had a negative effect on the delivery of technical support and the use of its results.42  

Good collaboration among different types of the programme’s stakeholders was the 
main factor driving the achievement of the programme’s results and impacts, while 
the  COVID-19  pandemic  and  some  resistance  to  change  hindered  these  achievements 
during  the  implementation  of  the  2017-2020  SRSP.  The  ownership  and  commitment 
from  Member States’ beneficiary authorities  had a mostly  positive effect on the design 
and  implementation  of  reforms  at  the  national  level  (with  some  negative  effects  of 
political instability on reforms).  

A lack of domestic reform ownership and commitment was perceived to be a serious 
hurdle  for  planning,  adopting  and implementing  complex  reforms.43  Therefore,  the 
CSPs of the SRSP expect that national authorities maintain full ownership of their reform 
agenda  and  assume  full  political  ownership  of  technical  support  provided  by  the 
programme. The supporting study indicates that the strong ownership of the reform in the 
beneficiary Member States was a positive factor in the delivery of projects’ outputs and 
results.44  

It also underlined that since the CSPs were not legal documents, while the contracts for 
technical support were signed by the Commission and the providers of technical support. 
Therefore, national  authorities were not bound by any contractual obligations regarding 
their participation in the SRSP-funded projects.  

BOX 3 – The role of the coordinating authorities in monitoring progress  

The  Study  highlights  that  DG  REFORM  has  the  possibility  to  alert  political  level 
counterparts in beneficiary Member States if a project is not working on the ground and 
even  stops  its  implementation.  According  to  the  Study,  some  coordinating  authorities 

40 European Commission (2020), Understanding the Political Economy of Reforms: Evidence from the EU: 
Technical  note  for  the  Eurogroup,  Brussels,  p.  4-5.  https://www.consilium.europa.eu/media/45511/ares-
2020-4586969_eurogroup-note-on-political-economy-of-reforms.pdf 
41  According  to  the  targeted  survey  data,  a  total  of  56%  (50  out  of  89)  of  technical  support  providers 
recognised the positive effect of political reform commitments, while 52% (47 out of 91) of them admitted 
the relevance of timing and sequencing of reforms (having a moderate and strong positive effect). 
42 For instance, due to the unstable political situation, Bulgaria did not submit its TJTP to the Commission 
by  May  2022,  despite  the  fact  that  the  finalisation  of  this  plan  was  the  expected  specific  outcome  of  its 
technical support project (‘Support for the Preparation of Territorial Just Transition Plans’ under the 2020 
dedicated call). 
43 European Commission (2020), Understanding the Political Economy of Reforms: Evidence from the EU: 
Technical 
https://www.consilium.europa.eu/media/45511/ares-2020-
4586969_eurogroup-note-on-political-economy-of-reforms.pdf 
44 A total of 64% of technical support providers (61 out of 96), 62% of coordinating authorities (13 out of 
21), as well as 55% (53 out of 96) of beneficiary authorities (with 36% of these authorities, 35 out of 96, 
indicating  a  neutral  effect)  agree  and  strongly  agree  that  national  ownership  of  reforms  was  a  positive 
factor in the delivery of outputs and results. 

the  Eurogroup. 

note 

for 

23 

 
 
were involved in the process of discussing progress on the implementation of the projects 
with different EU and national stakeholders (11 out of 20 coordinating authorities agree 
or  strongly  agree  to  this  statement),  however  they  were  less  actively  involved  in  the 
initiation  of  corrective  management  actions  to  improve  the  performance  of  the  projects 
(with 7 out of 17 coordinating authorities agreeing) at the national level.  

National  coordinating  authorities  could  play  a  more  important  role  during  the 
implementation of technical support projects and after their completion. According to the 
targeted  survey  of  coordinating  authorities,  only  some  respondents  agreed  to  a 
large/some  extent  that  they  collected  and/or  analysed  monitoring  data  on  the 
implementation  of  the  projects  (11  out  of  19,  or  58%),  discussed  progress  on  the 
implementation of the projects with different EU and national stakeholders (11 out of 20, 
or  55%)  or  initiated  corrective  management  actions  to  improve  the  performance  of  the 
projects (7 out of 17, or 41%). 

Interviews  with  a  few  representatives  of  national  coordinating  authorities  showed  that 
these institutions normally did not gather information about the outputs and results of the 
SRSP-funded  projects.  The  formal  role  of  national  coordinating  authorities  usually 
stopped  when  the  projects  started,  and  there  was  no  monitoring  and  reporting  tool 
developed  to  track  the  implementation  of  the  projects  at  national  level.  The  Study 
suggested that DG REFORM and national coordinating authorities could establish a joint 
framework  for  monitoring  and  reporting  that  could  also  include  the  organisation  of 
annual  or  biannual  reporting  meetings  to  take  stock  of  all  ongoing  and  recently 
completed technical support projects.  

In their replies to open ended questions, coordinating authorities would appreciate better 
feedback  on  the  rejected  applications  and  more  clarity  on  how  requests  for  technical 
support are being assessed and selected by DG REFORM 

The  focus  group  discussions  also  confirmed  that  the  coordinating  authorities  would 
appreciate some legal provisions or written recommendations for prioritising requests at a 
national  level,  as  this  would  help  them  to  justify  their  decisions  vis-à-vis  potential 
beneficiary authorities. They also reported that they would appreciate more transparency 
and  clarity  in  the  selection  of  technical  support  requests,  so  that  they  can  feedback  the 
reasons for rejection to unsuccessful applicant authorities.  

The  coordinating  authorities  and  beneficiary  authorities  reported  a  high  level  of 
satisfaction  with  the  selected  technical  support  providers  and  largely  agreed  that  the 
international experts selected by the Commission had the right competencies. There was 
a general agreement that this was crucial, as the beneficiary authorities required the best 
expertise  in  the  provision  of  technical  support  and  contracting  by  the  Commission 
allowed access to technical support providers, which otherwise would be difficult for the 
beneficiary authorities to access via other financing instruments. 

Sources: SRSP ex-post evaluation external independent study 

While  receiving  and  flowing  up  the  technical  support,  evidence  indicates  that  the 
beneficiary  authorities  sometimes  lacked  adequate  administrative  and  financial 

24 

 
capacity.  There  is  also  some  indication  that  the  staff  of  beneficiary  authorities  were 
overburdened with conflicting/concurrent tasks, underestimated the effort required during 
the implementation of technical support and were thus unable to provide sufficient input 
or support during the provision of technical support on the ground.  

FIGURE  8:  FOLLOW-UP  ACTIONS  TAKEN  AFTER  THE  PROJECT  END  AND  BUILDING  UPON  THE 
PROJECT OUTPUTS AND RESULTS 

45%

40%

35%

30%

25%

20%

15%

10%

5%

0%

38%

42%

38%

24%

Yes, a legal and/or 
policy decision of 
the country’s 
authorities 

Yes, the 
incorporation of 
some technical 
support 
deliverables into 
the country’s 
Recovery and 
Resilience Plan for 
2021-2026 

Yes, a request for
technical support
for a follow-up
project or a new
stage of the larger
project has been
submitted

Follow-up actions
are currently being
planned/organised

22%

No

Source: the targeted survey of beneficiary authorities, May-June 2022 (N=152).  

The  findings  suggest  that  there  was  a  good  ownership  and  commitment  from 
Member  States’  beneficiary  authorities  to  implement  the  outputs  of  technical 
support and to follow-up on the reform process, but the insufficient administrative 
capacity of some administrations was still one of the weakest links in the process of 
implementing structural and administrative reforms.  

4.1.2 Efficiency: How efficient has the SRSP been in achieving its objectives? 

The overall evaluation on the Programme’s efficiency is composed of the following three 
sub-analyses:  

•  To  what  extent  was  the  scale  of  the  SRSP-funded  actions  proportionate  to  the 
request for support submitted by the Member States and to the expected benefits 
from the reform projects? How cost-effective were the actions funded under the 
SRSP? 

•  Time-efficiency: how efficient is the SRSP as regards the process duration from 
request submission by Member States to the deployment of the technical support 
on  the  ground,  and  to  completion  of  technical  support?  How  time-efficient  was 
the budget execution from commitments to payments? 

•  The  adequacy  of  management  arrangements  for  the  implementation  of  the 
programme:  To what extent  was the governance  of the SRSP efficient  and how 
efficient was the cooperation with other Commission Services?  

25 

 
 
Proportionality of the allocated resources and the cost-effectiveness of the SRSP  

The demand for technical support funding among the Member States and the number of 
corresponding requests for technical support grew steadily during the 2017-2020 period. 
This  increase  in  demand  meant  that  the  annual  selection  rate  fell  from  59%  in  2017  to 
33%, 45% and 37% in 2018-2020, despite an increase in the SRSP’s annual budget each 
year.  

FIGURE 9: OVERALL CONTRACTED BUDGET BY YEAR (in EUR) 

 100.000.000
 90.000.000
 80.000.000
 70.000.000
 60.000.000
 50.000.000
 40.000.000
 30.000.000
 20.000.000
 10.000.000
 -

85.290.672 

89.719.513 

23.139.055 

30.965.002 

SRSP 2017

SRSP 2018

SRSP 2019

SRSP 2020

Source: internal DG REFORM monitoring system 

The overall selection rate of proposals during this period was 42%, which demonstrated a 
sufficient level of competition and the selection of high-quality proposals. The collected 
evidence  suggested  that  the  quality  of  requests  for  technical  support  grew  over  this 
period. 

FIGURE 10: SUBMITTED AND SELECTED REQUESTS UNDER THE 2017-2020 SRSP 

700

600

500

400

300

200

100

0

59%

580

609

444

33%

146

271

159

45%

263

37%

228

70%

60%

50%

40%

30%

20%

10%

0%

SRSP 2017

SRSP 2018

SRSP 2019

SRSP 2020

No. of submitted requests

No. of selected requests

Success rate (%)

Source:  DG  REFORM  based  on  desk  research.  Note:  this  figure  does  not  include  the  2020  dedicated  call  for  the 
preparation of the TJTPs. 

26 

 
 
 
The  average  budget  per  selected  request  increased  from  EUR  142  thousand  under  the 
2017  round  to  EUR  372  thousand  under  the  2020  round,  which  corresponds  to  an 
increase  of  2.6  times.  A  higher  average  budget  per  selected  request  resulted  in  the 
number of selected requests growing slower compared to the increase in the SRSP annual 
budget45.  

During  the  first  two  years  of  implementing  the  programme,  requests  for  support  were 
often  related  to  small  studies  or  surveys,  while  in  the  following  rounds  requests  for 
support typically aimed at much broader and more significant reforms.  

FIGURE 11: AVERAGE BUDGET PER SELECTED REQUEST FOR SUPPORT (EUR THOUSAND), 2017-
2020 

372

302

209

142

400

350

300

250

200

150

100

50

0

SRSP 2017

SRSP 2018

SRSP 2019

SRSP 2020

Source: DG REFORM based on desk research. 

The SRSP tailor-made support requires the close involvement of DG REFORM staff in 
the  management  and  implementation  of  technical  support  projects.  Consequently,  the 
number  of  technical  support  projects  was  also  limited  by  the  number  of  corresponding 
DG  REFORM  staff  who  could  be  involved  in  the  management  and  implementation  of 
such projects.  

The evidence shows that beneficiary authorities and coordinating authorities46 tended to 
agree  or  strongly  agree  that  the  budget/financial  contribution  of  the  SRSP  was 
proportionate to the needs and estimated scope of technical support, especially requests 
approved after 2018.  

45 The budget of SRSP annual calls increased 3.8 times from 2017 (EUR 22.5 million) to 2020 (EUR 84.9 
million), while the number of selected requests grew only 1.4 times (from 159 in 2017 to 228 in 2020).  

46  From  the  targeted  survey  of  beneficiary  authorities,  coordinating  authorities  and  technical  support 
providers revealed that 93% of beneficiary authorities (107 out of 115) and 89% of national coordinating 
authorities (17 out of 19) although the corresponding share of technical support providers was somewhat 
lower but still high (80%, 79 out of 99). However, only one technical support provider (out of 99) from this 
survey indicated that the project budget was too small, which had a strong negative effect on the delivery 
of the expected outputs and results.  

27 

 
 
 
FIGURE  12:  DEGREE  OF  AGREEMENT  ON  THE  PROPORTIONALITY  BETWEEN  BUDGET  AND 
SCOPE OF TECHNICAL SUPPORT 

Beneficiary authorities (N=115)

55%

38%

7%

National Coordinating Authorities (N=19)

26%

63%

5%

5%

Technical support providers (N=99)

28%

52%

17%

3%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Strongly agree

Agree

Disagree

Strongly disagree

Source: Targeted survey of beneficiary authorities, coordinating authorities and technical support providers, May-June 
2022. 

Further  evidence47  also  indicates  that  SRSP  actions  were  perceived  as  user-friendly, 
timely, and cost-effective to a moderate and high extent compared to other EU actions48 
and national actions by about two thirds of the respondents.49  

According  to  the  Study,  DG  REFORM’s  overall  cost  of  controls  over  total  annual 
payments was estimated to be 6.7% of the payments executed in 2018, 4.8%  – in 2019 
and  4.5%  –  in  202050.  These  costs  of  controls  at  DG  REFORM  were  comparable  with 
other EU instruments51.  

The targeted survey of beneficiary authorities and technical support providers shows that 
effective  cooperation  with  DG  REFORM  policy  officers  was  one  of  the  key  factors 
related to the successful implementation of the SRSP projects. 

The  implementation  of  the  SRSP  revealed  a  growing  demand  for  technical  support 
among  the  Member  States  and  a  sufficient  level  of  competition  among  the  projects, 
which enabled the selection of high-quality proposals.  The  average budget  per selected 
request significantly increased between 2017-2020, in line with the increasing depth and 
breadth  of  the  scope  of  reforms  supported  under  the  programme.  The  stakeholders 

47 The analysis of the OPC data revealed that more than half of OPC respondents perceived SRSP actions 
as user-friendly, timely, and cost-effective to a moderate and high extent compared to other EU actions (17 
out of 26 respondents) and national actions.  
48 17 out of 26 respondents 
49 18 out of 26 respondents 
50 Note that the estimated costs of controls over the total annual payments progressively decreased over the 
evaluation period considering the progressively increasing value of the total payments made. 
51 For instance, the programme management costs  (the ratio between their administrative and operational 
budgets in terms of executed payment appropriations) of the Commission’s Executive Agencies on average 
constituted 4.25% in 2020 (based on the Annual activity reports 2020 of INEA, CHAFEA, REA, EACEA, 
ERCEA and EASME). The programme management costs in the agencies ranged from 0.75% (in the case 
of  the  Innovation  and  Networks  Executive  Agency)  to  8.65%  (in  the  case  of  the  Consumers,  Health, 
Agriculture and Food Executive Agency); these differences were related to the specific characteristics of 
the programmes delegated to the respective agencies (such as the average project size, the competition rate 
in  the  application  and  evaluation  stage,  the  heterogeneity  of  applicants  and beneficiaries,  the  complexity 
and  diversity  of  the  agency’s  programme  portfolio,  etc.)  Similarly,  the  Commission's  administrative 
expenditure (programme management costs) can constitute up to 5% of the Horizon 2020 overall budget. 

28 

 
 
 
generally  agreed  that  the  financial  contribution  of  the  SRSP  was  proportionate  to  the 
needs and estimated scope of technical  support. The costs of controls  at DG REFORM 
were comparable with other EU instruments. 

Time-efficiency of the SRSP processes and the budget execution  

DG REFORM achieved very good results in the timely selection of requests for support, 
according  to  the  external  evaluation  report.  Specifically:  the  average  time  from  the 
application  deadline  to  the  financing  decision  (the  adoption  of  the  respective  work 
programme)  was  around  four  months  during  the  2017-2020  period.  This  was 
substantially  below  the  applicable  time-to-inform  target  of  six  months  set  in  Article 
194.2(a) of the Financial Regulation52.  

As an element of comparison, the timeliness of the selection of requests for support under 
the  SRSP  compares  favourably  with  other  EU  programmes.  For  example,  the  ‘time-to-
inform’  within  programmes  managed  by  the  Research  Executive  Agency  in  2017  was 
130 days53, and the corresponding indicator for the Education, Audiovisual and Culture 
Executive Agency was 125 days54,55.  

Preparations  for  the  implementation  phase  (i.e.,  time  taken  from  the  submission  of 
technical support requests to the start of delivering technical support) were carried out in 
a timely and efficient manner (see the figure below), as evidence supports. In particular, 
the  targeted  survey  carried  out  by  the  external  contractor  revealed  that  87%  of 
beneficiary  authorities  (101  out  of  116)  agreed  to  this  statement.  This  was  an 
improvement  compared  to  the  results  of  the  survey  carried  out  during  the  mid-term 
evaluation  of  the  SRSP,  where  the  corresponding  result  for  the  beneficiary  authorities 
stood at 77% (N=100).  

The  element  to  be  improved  in  the  preparation  of  the  implementation  phase  is  the 
contracting  process.  Some  interviewees  indicated  that  the  contracting  process  was  the 
most  frequently  mentioned  negative  aspect  of  the  SRSP’s  administrative  arrangements. 
In some cases, it took around one year from the conception of the project until the start of 
project activities; this was considered, since the reform priorities and the general context 
could have evolved in the meanwhile.  

52 Regulation (EU, Euratom) 2018/1046 of the European Parliament and of the Council of 18 July 2018 on 
the  financial  rules  applicable  to  the  general  budget  of  the  Union,  amending  Regulations  (EU)  No 
1296/2013, (EU) No 1301/2013, (EU) No 1303/2013, (EU) No 1304/2013, (EU) No 1309/2013, (EU) No 
1316/2013,  (EU)  No  223/2014,  (EU)  No  283/2014,  and  Decision  No  541/2014/EU  and  repealing 
Regulation  (EU,  Euratom)  No  966/2012.  Official  Journal  of  the  European  Union,  L  193/1,  30.7.2018. 
http://data.europa.eu/eli/reg/2018/1046/oj. 
53  PPMI  and  CSES  (2020),  Study  supporting  the  evaluation  of  the  Research  Executive  Agency  (2015-
2018): final report. Publications Office, https://data.europa.eu/doi/10.2777/299516. 
54  PPMI  (2020),  Study  supporting  the  evaluation  of  the  Education,  Audiovisual  and  Culture  Executive 
Agency (2015 – 2017). 
55 Other examples: The ‘time-to-inform’ for the main calls in 2015-2018 within programmes managed by 
the  European  Research  Council  Executive  Agency  varied  from  133  days  to  237  days  (PPMI  and  CSES 
(2020), Study supporting the evaluation of the European Research Council Executive Agency (2015-2018): 
final report. Publications Office, https://data.europa.eu/doi/10.2777/487469). Within the Urban Innovative 
Actions,  the  ‘time-to-inform’  was  between  157  and  187  days  (CSES  (2021),  Assessment  Study  of  the 
Urban 
office, 
https://data.europa.eu/doi/10.2776/5314.  

Publications 

2014-2020: 

Innovative 

Actions 

report. 

final 

29 

 
 
Conversely, the project  duration was appropriate for carrying out the activities: 96% of 
beneficiary authorities  (117 out  of 122)  and 90% of  coordinating authorities (18 out  of 
20) agreed or strongly agreed with this statement. Only some interviewees considered the 
project implementation phase too short – especially compared to the period preceding the 
start of project activities – although nearly all projects managed to achieve their outputs 
in time. 

FIGURE  13:  THE  EXTENT  TO  WHICH  THE  RESPONDENTS  AGREED  THAT  THE  FOLLOWING 
PROJECT PHASES HAVE BEEN IMPLEMENTED IN A TIMELY AND EFFICIENT MANNER 

2%

8%

28%

5%

10%

24%

63%

62%

2%

11%

33%

54%

5%

29%

33%

33%

2%
5%

27%

66%

5%

5%

42%

47%

100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

BA (N=115)

CA (N=21)

BA (N=116)

CA (N=21)

BA (N=116)

CA (N=19)

The evaluation and selection of
technical support requests

Preparation for implementation

The implementation of
technical support projects

To a large extent

To some extent

To a limited extent

Not at all

Source: the targeted survey of beneficiary authorities and coordinating authorities, May-June 2022. Note: ‘BA’ stands 
for beneficiary authorities, and ‘CA’ – for coordinating authorities. 

The  targeted  survey  revealed  that  93%  of  beneficiary  authorities  (108  out  of  116)  and 
89%  of  coordinating  authorities  (17  out  of  19)  agreed  that  the  implementation  of 
technical support projects56 was carried out in a timely and efficient manner, in line with 
the mid-term evaluation. 

DG  REFORM  achieved  good  results  in  the  execution  of  SRSP-related  commitment 
appropriations and, starting from 2018, payment appropriations: DG REFORM managed 
to achieve nearly a full execution of SRSP-related commitment appropriations in 2017-
2020.57 Since the implementation of SRSP-funded projects usually takes more than one 
year, the amount of SRSP commitments to be settled at the end of 2020 constituted EUR 
104.84 million.  

56  Time  taken  from  the  conclusion  of  contracts/agreements  with  technical  support  providers  to  the 
completion of projects. 
57 The amount of SRSP payment appropriations grew constantly during 2017-2020, and the execution level 
of payment appropriations rose from 67% in 2017 to 97% in 2018 and nearly full execution in 2019-2020 

30 

 
 
 
FIGURE  14:  EXECUTION  OF  SRSP  RELATED  COMMITMENT  AND  PAYMENT  APPROPRIATIONS 
(EUR MILLION), 2017-2020 

90

80

70

60

50

40

30

20

10

0

100,0%

100,0%

99,9%

78,8

78,7

96,6%

99,3%

84,2

84,0

99,8%

100,0%

54,7

54,7

22,5

22,5

66,9%

3,9

2,6

29,4

29,4

15,8

15,2

34,9

34,6

2017

2018

2019

2020

Commitment appropriations authorised

Commitments made

Payment appropriations authorised

Payments made

Executed commitments (%)

Executed payments (%)

100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

Source: DG REFORM based on desk research. 

The  share  of  payments  processed  within  the  set  time  limits  of  DG  REFORM  stood  at 
79%  in  2017  and  gradually  increased  to  96%  in  2020  and  98%  in  2021.  The  average 
number of days taken to make a payment constituted around 30 days in 2018-2021. The 
targeted  survey  revealed  that  88%  of  technical  support  providers  agreed  or  strongly 
agreed that payments for the provision of technical support were received in due time.  

The overall findings reveal that DG REFORM achieved very good results in the timely 
selection  of  requests  for  support.  The  stakeholders  were  generally  positive  about  time-
efficiency  in  the  evaluation  and  selection  of  technical  support  requests  and  the 
implementation of technical support projects.  

Lower  satisfaction  levels  were  sometimes  observed  concerning  the  timeliness  of 
preparation for the implementation phase, in particular the contracting process under the 
delivery  mode  of  public  procurement.  This  posed  challenges 
the  smooth 
implementation of technical support projects and the relevance and utility of their results. 
DG  REFORM  achieved  good  results  in  the  execution  of  SRSP-related  commitment 
appropriations and, starting from 2018, payment appropriations. 

to 

Efficiency of the SRSP governance and administration procedures  

The survey programme revealed that the beneficiary authorities, coordinating authorities 
and  technical  support  providers  were  generally  very  positive  concerning  the  clarity, 
transparency  and  user-friendliness  of  the  procedures  related  to  the  application  and 
selection  of  requests  for  funding  and  monitoring  of  the  projects,  including  relevant  IT 
tools (Figure 14).  

31 

 
 
FIGURE  15:  THE  EXTENT  TO  WHICH  THE  RESPONDENTS  AGREED  THAT  THE  SRSP 
ADMINISTRATIVE PROCEDURES WERE CLEAR, TRANSPARENT AND USER-FRIENDLY 

14%

5%

57%

0%
3%

46%

51%

100%

1%
4%

56%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

39%

24%

15%

5%

70%

10%

1%
4%

0%

8%

49%

64%

3%

8%

54%

16%

11%

53%

43%

30%

35%

21%

BA (N=117) CA (N=21) BA (N=113) CA (N=20) TSP (N=89) BA (N=119) CA (N=19) TSP (N=89)

The procedures of
application and selection
of requests for funding
were clear, transparent
and user-friendly

The monitoring of the project(s)
(including relevant IT tools)  was clear,
transparent and user-friendly

The evaluation procedures and
processes after the completion of
project(s) (satisfaction and outcome
questionnaires) were clear, transparent
and user-friendly

Strongly agree

Agree

Disagree

Strongly disagree

Source: the targeted survey of beneficiary authorities, coordinating authorities and technical support providers, May-
June 2022. Note: ‘BA’ stands for beneficiary authorities, ‘CA’ – for coordinating authorities and ‘TSP’ – for technical 
support providers. 

Slightly  lower  scores  were  observed  related  to  the  clarity,  transparency  and  user-
friendliness of the evaluation procedures and processes after the completion of projects, 
based  on  the  feedback  mechanism  set  up  by  DG  REFORM  to  evaluate  the  completed 
projects (see box 4 below).  

BOX 4 – Feedback mechanism: satisfaction and outcome questionnaires 

To evaluated completed projects, DG REFORM  has set up a feedback mechanism to 
collect information through questionnaires. 

The satisfaction questionnaires  were  sent  to  beneficiary  authorities, technical  support 
providers and DG REFORM policy officers at the end of the projects and were used to 
provide feedback on the project and assess the extent to which the support provided to 
the  Member  States  met  its  objectives,  to  identify  factors  which  influenced  the 
successful delivery of the support measures. 

The  outcome  questionnaires  were  sent  to  the  beneficiary  authorities  within  6-18 
months  after  the  project  completion  to  provide  follow-up  on  the  actual  use  of  the 
results  and  outputs  of  the  technical  support  received  for  achieving  specific  project 
outcomes and project impacts. However, these questionnaires and their use had a few 
limitations.  

First,  the  information  provided  in  the  outcome  questionnaires  was  based  on  the 
perception of stakeholders and the quality of the provided information varied. Second, 
their response rate, especially in the case of the outcome questionnaires, was somewhat 

32 

 
 
lower  (about  66%  in  the  last  round  of  these  questionnaires  compared  to  the  higher 
response rates of the satisfaction questionnaires).  

Interviews  with  DG  REFORM  staff  indicated  that  the  practical  use  of  information 
from  the  feedback  questionnaires  for  monitoring  and  policy  feedback  purposes  was 
limited  and  these  tools  were  not  sufficiently  integrated  into  programme  monitoring 
practices.  

In their open replies to the survey questions, several respondents from the coordinating 
authorities and technical support providers stated the need to improve the monitoring and 
evaluation  of  technical  support  outputs,  and  better  use  these  monitoring/evaluation 
findings, considering that the projects and reforms are sometimes managed by small staff 
in the beneficiary institutions, while strong political commitment is not always in place 
and knowledge can be lost due to factors such as high staff turnover.  

The  administrative  burden  related  to  the  application  and  project  implementation 
processes  were  regarded  as  reasonable  and  proportionate  by  the  beneficiary  authorities 
and coordinating authorities, as revealed by the survey programme, by the analysis of the 
OPC data,58 and by the results of the case studies59  

Interviewees indicated that the administrative burden was comparable or lower compared 
to  the  management  of  other  EU  programmes  and  instruments  (e.g.,  representatives  of 
beneficiary  authorities  compared  the  implementation  of  the  SRSP  to  the  operational 
programmes  under  ESIF60),  describing  arrangements  and  procedures  as  overall  rather 
user-friendly and not too complex in terms of required documentation.  

Beneficiary  authorities  highlighted  as  a  positive  factor  that  much  of  the  project 
management  and  implementation  tasks  were  shifted  to  the  Commission  (e.g.,  in 
contracting procedures) and technical support providers (in terms of carrying out project 
activities, such as studies, research, evaluations, etc., and delivery of the project outputs, 
such as recommendations, analysis and reports, action plans, etc.)  

While  this  limited  the  administrative  burden  on  beneficiaries  and  allowed  them  to 
dedicate  a  smaller  number  of  staff  to  technical  support  projects,  it  also  to  some  extent 
limited their involvement and influence in some project phases (see section 4.1 

.3  concerning  the  insufficient  capacities  of  some  administrations  to  provide  support 
during the implementation of project activities). 

58  The  majority  of  OPC  respondents  (19  out  of  26)  considered  the  proportionality  of  the  administrative 
burden as a moderately or highly suitable factor in making the SRSP`s implementation efficient. 
59 The case studies confirmed that the administrative arrangements under the SRSP were overall adequate 
and  did  not  impose  an  excessive  administrative  burden  on  either  technical  support  providers  or 
beneficiaries. 
60 It is important to note that the SRSP and ESIF fall under different implementation modes. The SRSP is 
implemented in a direct management mode, and the  European Commission is directly responsible for all 
steps in the programme's implementation. The ESIF operational programmes are implemented in a shared 
management mode where the European Commission and national authorities jointly manage the funding. 
Under the ESIF, Member States' administrations select which projects to finance and take responsibility for 
the day-to-day management; the final beneficiaries are usually responsible for contracting and management 
of  the  specific  ESIF  projects.  Further,  the  ESIF  projects  generally  require  co-financing  by  the  Member 
States, whereas SRSP does not require co-financing. Therefore the instruments are not directly comparable.  

33 

 
 
FIGURE  16:  THE  EXTENT  TO  WHICH  THE  RESPONDENTS  AGREED  THAT  THE  ADMINISTRATIVE 
BURDEN  RELATED  TO  THE  SRSP  ADMINISTRATIVE  PROCEDURES  WAS  REASONABLE  AND 
PROPORTIONATE 

0%
5%

51%

44%

100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

5%
0%

70%

25%

1%
3%

48%

5%
0%

45%

48%

50%

6%

10%

50%

34%

BA (N=118)

CA (N=22)

BA (N=117)

CA (N=20)

TSP (N=100)

The administrative burden related to
the submission of requests was
reasonable and proportionate

The administrative burden related to the
implementation of the project(s) was reasonable and
proportionate

Strongly agree

Agree

Disagree

Strongly disagree

Source: the targeted survey of beneficiary authorities, coordinating authorities and technical support providers, May-
June 2022. Note: ‘BA’ stands for beneficiary authorities, ‘CA’ – for coordinating authorities and ‘TSP’ – for technical 
support providers. 

The  satisfaction  level,  with  respect  to  the  administrative  burden,  among  the  technical 
support  providers  was  also  high  although  slightly  lower  than  among  the  beneficiary 
authorities and coordinating authorities. A total of 84% of technical support providers (84 
out of 100) agreed or strongly agreed that the administrative burden was reasonable and 
proportionate.  However,  these  results  varied  substantially  across  different  delivery 
modes, with 94% of support for public procurement and 67% of support for grants. 

DG  REFORM  coordinated  efficiently  and  effectively  the  technical  support  to  Member 
States  with  other  Commission  DGs  and  services,  as  the  results  of  the  targeted  survey 
evidence  suggests,  and  actively  participated  in  the  European  Semester  framework  for 
economic and employment policy coordination.61 The study indicated that in the period 
2017-2020 there could have been closer cooperation with other DGs and services of the 
Commission  and  disseminate  and  exploit  the  outputs  of  technical  support  after  the 
completion  of  the  projects.  This  was  done  as  of  mid-2020  onwards,  also  through  the 
elaboration of flagships reforms in close collaboration with other DGs. 

As with any EU programme, the Commission was required to establish reliable financial 
management and internal control processes. The main aim of such controls was to ensure 
that the residual error rate did not exceed the materiality threshold of 2% calculated on a 
cumulative  multi-annual  basis.  Overall,  the  residual  error  rate  estimated  for  the  SRSP 

61 The efficient and effective coordination activities with other DGs showed that 100% of the other DGs 
were satisfied with the coordination process in 2018 (15/15) and 2019 (10/10).  The corresponding result 
for 2020 is 95%, however, it relates to the coordination process for the TSI round.  

34 

 
 
 
was  below  the  materiality  threshold  for  most  parts  of  the  Programme.62  This  indicates 
that  suitable  control  arrangements  were  put  in  place,  except  for  grants  to  non-pillar-
assessed organisations for which a reservation was recorded.  

The  higher  error  rate  in  the  non-pillar  assessed  grants  was  related  primarily  to  the 
implementation modes of such grants, including the mechanism of grant reimbursement 
based on eligible actual costs and the related risk of errors in cost reimbursement claims 
submitted by respective technical support providers. To address this issue, DG REFORM 
improved its practices of instructing grant beneficiaries on the eligible actions and costs 
for final cost claim, reduced thresholds for audit certificates and increased ex-ante control 
checks on sampled final payments.  

In  general,  Programme’s  stakeholders  were  positive  about  the  clarity,  transparency  and 
user-friendliness of the SRSP’s administration procedures and the proportionality of the 
administrative  burden.  An  area  for  improvement  is  however  related  to  the  further 
development and refining of monitoring and project evaluation practices. DG REFORM 
ensured the efficient and effective coordination of technical support to the Member States 
with other Commission services.  

4.1.3 Coherence  

This section addresses the internal and external coherence of SRSP actions where: 

- 

Internal  coherence  refers  to  the  extent  to  which  the  actions  funded  under  the 
SRSP are coherent with the objectives as set out in the SRSP Regulation. 
-  External  coherence  refers  to  the  extent  to  which  the  SRSP  was  externally 
coherent  with  other  interventions  at  regional,  national  and  EU  level,  such  as 
technical assistance through structural and investment funds. 

Internal coherence of the SRSP 

No major gaps, inconsistencies or overlaps in the internal structure of SRSP actions were 
observed.  There  was  a  coherent  internal  logic  of  the  SRSP  structure,  linking  the 
programme’s  general  objectives,  the  specific  objectives  and  the  activities  that  should 
produce specific outputs and eventually lead to the achievement of reform results.  

The  programme  contributed  to  a  large  extent  to  the  implementation  of  the  specific 
objectives set out in the SRSP Regulation, as indicated in section 4.1.1 above. However, 
the evaluation evidence shows that the project outputs could be better disseminated and 
utilised  in  all  thematic  areas  (based  on  the  example  of  good  practices  in  the  field  of 
governance  and  public  administration)  and  in  communication  or  consultation  activities 
could be added more frequently to the design of future projects.  

The  Programme  developed  a  consistent  internal  project  monitoring  system,  allowing  a 
clear linkage at the level of individual projects between such key elements as: objective 

62  DG  REFORM  –  Directorate-General  for  Structural  Reform  Support  (2021),  Annual  Activity  Report 
2020.https://ec.europa.eu/info/sites/default/files/annual-activity-report-2020-structural-reform-
support_en.pdf. 

35 

 
 
of  the  support;  means  of  delivery,  eligible  activities,  outputs,  pre-defined  groups  of 
outcomes (results), and expected impact. 

A  variety  of  delivery  modes  ensures  that  the  activities,  type  of  support  and  expertise 
provided  to  beneficiary  correspond  to  the  project  objectives  and  needs,  as  indicated  by 
the  evaluation  report.  The  results  of  stakeholder  consultations  suggest  that  the  internal 
structure  and  design  of  the  SRSP  allowed  for  a  wide  flexibility  when  selecting  project 
activities that were best tailored to specific project objectives and beneficiary needs.  

Over  time  the  structure  and  activities  of  the  SRSP  were  adjusted  to  better  respond  to 
newly  emerging  challenges  and  to  achieve  its  objectives  in  a  more  effective  way,  as 
shown in some instances by the evaluation evidence. For example, a dedicated call was 
initiated  to  respond  to  the  specific  emerging  needs  of  Member  States  or  the 
implementation of new EU priorities.  

External coherence of the SRSP with other programmes and interventions 

Evidence  suggests  that  the  SRSP  was  complementary  to  support  provided  through  the 
ESIF.63  In  particular,  the  targeted  survey  results  show  that  72%  of  national  beneficiary 
authorities (35 out of 49) considered that SRSP to a moderate/high extent complemented 
the support provided through the ESIF. Similarly, 69% of the beneficiary authorities that 
offered an opinion on this issue (34 out of 49) considered that the SRSP complemented 
the  interventions  enhancing  the  institutional  capacity  of  public  authorities  and 
stakeholders and efficient public administration supported by the ESF and the ERDF.  

When  it  comes  to  employment  policies,  the  respondents  claimed  that  the  SRSP 
complemented the modernisation of public and private employment services (financed by 
the  ESF)  and  investments  in  employment infrastructure  by  the  ERDF  only  to  a  limited 
extent.64 However, there is mixed evidence regarding the SRSP’s complementarity with 
national/regional  schemes  (i.e.,  the  schemes  implemented  at  national/regional  level  and 
providing technical support similar to the SRSP)65. 

63  It  must  be  noted  that  the  technical  assistance  under  the  ESIF  was  mainly  focused  on  supporting  the 
measures for the implementation of funds, while the support aimed to enhance the institutional capacity of 
public  authorities  and  stakeholders  and  efficient  public  administration  was  provided  under  thematic 
objective 11. 
64 This evidence was also supported by the OPC results where 18 out of 26 respondents indicated that the 
SRSP  complemented  (by  funding  different  aspects  of  similar  activities,  targeting  different  groups,  etc.) 
most EU cohesion policy instruments (the ERDF, the Cohesion Fund, the ESF). 13 out of 26 respondents 
indicated that SRSP complemented the Horizon 2020 programme. 
65 According to the targeted survey results, more than half (54% of those who offered an opinion on the 
issue,  N=44)  of  surveyed beneficiary  authorities  indicated that  the  programme  to  a  high/moderate  extent 
complemented  the  actions  of  similar  national  and  regional  programmes  (e.g.,  by  addressing  the  same 
issues,  target  groups,  aiming  at  the  same  objectives,  etc.).  Similarly,  11  out  of  26  OPC  respondents 
indicated  that  the  SRSP  complemented  the  actions  of  similar  national  and  regional  programmes  to  a 
high/moderate  extent  (in  addition,  13  stakeholders  indicated  they  had  no  opinion/cannot  answer  the 
question, with only 2 respondents indicating there was no complementarity).  

36 

 
 
 
FIGURE 17: COMPLEMENTARITY OF THE SRSP TO OTHER EU FUNDS 

Modernisation of labour market institutions,
such as public and private employment services
(supported by the European Social Fund), and
investing in infrastructure for employment
services (supported by the European Regional
Development Fund) (n=35)

Interventions enhancing institutional capacity of
public authorities and stakeholders and efficient
public administration supported by the
European Social Fund and the European
Regional Development Fund (n=49)

17%

17%

23%

8%

24%

22%

Technical assistance provided through the
European Structural and Investment Funds
(n=49)

27%

12%

16%

43%

45%

45%

To a high extent

To a moderate extent

To a limited extent

Not at all

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

Source: the targeted survey of beneficiary authorities, May-June 2022. 

The  Study  shows  also  that  the  SRSP  had  strong  complementarities  and  synergies  with 
other  programmes/interventions  implemented  at  the  level  of  Member  States  or  the  EU: 
For  example,  in  addition  to  several  SRSP-funded  projects,  the  curriculum  reform  in 
Croatia was supported by an ESF-funded project that provided funding for materials, IT 
equipment, honoraria for mentors, etc.  

The  key  difference  between  the  ESIF  and  the  SRSP  lies  in  the  requirement  for  co-
financing from the Member States, which is present in case of the ESIF and absent in the 
SRSP.  In  addition,  several  interviews  with  Commission  officials  (including  those  from 
DG  REFORM)  supported  the  finding  that  while  thematically  the  SRSP  to  some  extent 
overlapped with other EU-level programmes (such as the ESF, Fiscalis or the programme 
for Competitiveness of Enterprises and Small and Medium-sized Enterprises (COSME)), 
the SRSP largely  complemented these interventions primarily by providing tailor-made 
technical support for national authorities in a specific sectoral area.66  

Evidence stemming from interviews and case studies also shows that in some cases, the 
SRSP  had  synergies  with  other  EU-level  programmes,  where  the  instruments  mutually 
reinforced each other and helped to achieve the reform objectives. More specifically, the 
technical  support  projects  under  SRSP  often  contributed  to  the  preparation  of  the 
strategies  and  workplans  for  the  reform  actions  foreseen  to  be  funded  under  the  ESF, 
ERDF or other funds. One example of such synergies is provided by Romania, where the 
methodologies  and  recommendations  developed  during  the  SRSP-funded  project  were 

66  For  example,  the  Fiscalis  programme  featured  multilateral  and  joint  actions,  including  sharing  good 
practices  and  training  for  national  tax  administrations  to  fight  against  tax  fraud,  tax  evasion  and  tax 
avoidance. However, Fiscalis did not feature tailored support provided for addressing the specific needs of 
individual Member States. Therefore, the SRSP complemented Fiscalis by providing Member States with 
such tailored support, e.g. in the area of tax administration reforms. 

37 

 
 
 
incorporated  and  supported  colorectal  cancer  screening  project  co-funded  by  the  ESF 
2014-2020. 

Existing  links and cooperation channels  between  DG REFORM  and other Commission 
services contributed to synergies between different instruments: e.g., Directorate-General 
for Regional and Urban Policy (DG REGIO) experts cooperated with DG REFORM on a 
systemic basis by contributing to the design of SRSP-funded projects that have a direct 
link with the ERDF and Cohesion Fund. This support involved helping authorities  draft 
terms  of  references,  identifying  project  tasks  and  participating  in  project  steering 
committees.  In  addition,  DG  REGIO  provided  DG  REFORM  with  potential  priority 
areas for intervention to be considered during bilateral meetings between DG REFORM 
and national authorities. 

The  SRSP  complemented  some  other  EU-level  interventions  (particularly  EU  cohesion 
policy  instruments,  such  as  the  ESIF),  as  well  as  similar  national/regional  schemes  by 
addressing  different  aspects  of  similar  objectives,  issues  and  target  groups.  The  SRSP 
complemented  other  EU  interventions  primarily  by  providing  tailor-made  technical 
support for national authorities in a specific sectoral area. 

SRSP coherence with the European Semester process and the EU priorities 

In  general,  the  Study  assessed  very  positively  the  contribution  of  the  programme  to 
addressing CSRs issued in the context of the European Semester process. According to 
the  targeted  survey  results,  most  beneficiary  authorities  (61  out  of  82,  or  around  74%) 
and coordinating authorities (18 out of 20, or 90%) indicated that their project addressed 
CSRs  issued  in  the  context  of  the  European  Semester  to  a  moderate  or  high  extent.  In 
comparison,  the  SRSP  mid-term  evaluation  of  2019  found  that  only  45%  of  projects 
addressed the European Semester recommendations67.  

A  more  in-depth  analysis  of  the  targeted  survey  data  also  revealed  that  most  of  the 
projects  addressing  CSRs  were  in  the  sectoral  area  of  governance  and  public 
administration68,  and  to  a  somewhat  smaller  extent,  in  the  area  of  labour  market, 
education, health and social services. 

In terms of the coherence of the SRSP with the EU priorities for 2019-2024: 

•  66%  of  beneficiary  and  coordinating  authorities  (67  out  of  102)  considered  that 
their  projects  contributed  to  the  EU  priority  of  developing  a  strong  and  vibrant 
economic base.  

•  58%  of  them  considered  that  their  projects  contributed  to  the  EU  priority  of 

building a climate-neutral, green, fair and social Europe (54 out of 93).69  

67 It must be noted that in the SRSP mid-term evaluation, the assessment of the share of projects related to 
European  Semester  recommendations  was  based  on  the  contractors’  analysis  of  the  administrative  data 
from  the  Jira  system  available  at  the  time,  i.e.  not  the  evidence  from  beneficiary  authorities’  targeted 
survey.  
68 30 out of 61 beneficiary authorities that indicated that their project addressed CSRs at least to a moderate 
extent, also indicated that their technical support project(s) were delivered in this thematic area.  
69 Moreover, the dedicated call on the preparation of the TJTPs particularly strengthened the programme’s 
contribution to the EU priority of building a climate-neutral,  green, fair and social Europe  because these 
plans fall under the Just Transition Mechanism (JTM) that is an integral part of the European Green Deal. 

38 

 
 
•  38% of them indicated that the projects moderately and highly contributed to the 

protection of citizens and freedoms (29 out of 77). 

In  general,  the  evidence  from  the  interviews  with  stakeholders  strongly  shows  a  clear 
complementarity  and  synergies  between  the  European  Semester  process  and  the  SRSP. 
Since  CSRs  are  often  formulated  in  very  general  terms,  the  SRSP  functioned  as  an 
instrument  to  better  define,  clarify,  operationalise  and  provide  specific  guidelines  for 
Member  States  to  implement  these  recommendations.  In  other  instances,  the  results  of 
the  SRSP-funded  projects  (guidelines,  recommendations,  etc.)  were  occasionally  taken 
up and used when defining CSRs.  

Evidence from the interviews with  stakeholders  also  suggests that  there  were sufficient 
mechanisms to ensure the linkages and integration between the SRSP and the European 
Semester  process/CSRs.  For  instance,  the  linkage  and  contribution  to  the  fulfilment  of 
CSRs was considered during the evaluation of requests for technical support. However, 
the  study  indicates  that  there  was  insufficient  tracking  of  whether  SRSP  projects 
contribute to the fulfilment of specific CSRs.  

While the fulfilment of CSRs is a responsibility of Member States and does not depend 
on  the  provision  of  technical  support,  in  some  cases  CSRs  could  trigger  the  technical 
support under the SRSP and the latter could help Member States to address CSRs. The 
SRSP would therefore benefit from a more systematic monitoring of how and in which 
specific  ways  the  technical  support  provided  by  the  SRSP  contributes  to  the 
implementation of CSRs in specific national contexts.  

In  addition,  over  the  study  period  the  European  Semester  country  reports  did  not 
consistently provide references to the SRSP technical support and the contribution of this 
support  to  the  progress  that  the  countries  achieved.  The  introduction  of  systematic 
referencing  to  the  SRSP/TSI  projects  in  the  respective  European  Semester  country 
reports should also be considered. 

As a conclusion, the evaluation evidence shows that the SRSP was highly coherent with 
the  European  Semester  process  (particularly  by  helping  address  CSRs),  and  there  were 
sufficient  mechanisms  to  ensure  linkages  and  integration  between  the  SRSP  and  the 
European Semester process.  

The SRSP was also coherent with some of the EU priorities (particularly in the areas of 
economic  development  and  green  transition).  However,  the  programme  would  benefit 
from a more systematic monitoring of how the technical support provided by  the SRSP 
contributes to the implementation of CSRs in specific national contexts. 

4.2 How did the EU intervention make a difference and to whom?  

EU valued added of SRSP compared to what could be achieved at national/regional and 
local levels 

The  evidence  from  the  targeted  survey  of  beneficiary  and  coordinating  authorities 
indicates  that  80%  of  stakeholders  (106  out  of  132)  considered  that  without  the  SRSP, 
beneficiary  authorities  could  not  have  received  similar  technical  support  or  could  have 

39 

 
received  it  only  to  a  limited  extent,  be  it  at  national,  regional  or  local  level.  Similar 
results were provided during the open public consultation (OPC).70  

Projects  analysed  in  the  case  studies  also  demonstrated  that,  whilst  the  project  benefits 
could  have  been  realised  without  EU  technical  support,  these  would  have  been  much 
smaller in scope and/or realised later in time.  

The study also provided some more in-depth evidence on why the results of the SRSP-
funded  projects  could  not  have  been  achieved  at  national/regional  level  and  the  added 
value of SRSP:  

-  The  Programme’s  funding  provided  access  to  high-level  international  expertise, 
perspective and knowledge, which was considered less likely to be available if the 
project was funded and managed exclusively at the national level.  

-  Technical  support  activities  would  have  been  unavailable  without  EU  funding, 
mainly due to limited administrative and financial resources at the national level 
(this factor was particularly emphasised by the smaller Member States). 

-  The tailored solutions reflecting the needs of each Member State were offered by 

the SRSP and were unavailable or too costly at national and regional level.  

-  The involvement of the Commission in the design and implementation of reforms 
provided  greater  institutional  legitimacy  and  political  impetus  for  structural  or 
administrative  reforms  that  were  contested  or  where  there  was  insufficient 
political  momentum  to  overcome  institutional  reluctance  to  change.  The 
legitimising  factor  of  EU-level  support  was  especially  relevant  in  countries 
overcoming  political  turbulence  or  in  the  case  of  the  reforms  focused  on  issues 
lacking a full consensus.  

-  Low  administrative  burden,  clear  procedures  and  the  absence  of  co-financing 
requirements  distinguished  the  SRSP  from  other  national/regional  and  EU 
programmes. This was also identified as a source of the SRSP’s added value and 
one  of  the  key  reasons  why  Member  States  could  not  achieve  similar  results 
without  the  programme.  The  technical/administrative  support  provided  by  the 
European Commission at the stages of contracting, procurement and evaluation of 
tenders helped to reduce the administrative burden on beneficiary authorities.  
-  Based on the results of DG REFORM satisfaction questionnaires, guidance, help 
and  supervision  provided  during  the  project  design  and  implementation  stages 
were the most commonly mentioned type of added value identified by beneficiary 
authorities.71  The  Commission’s  guidance  and  support  contributed  to  the  quality 

70  The  majority  of  OPC  respondents  (11  out  of  26)  indicated  that  Member  States  could  have  achieved 
similar  results  in  their  design  and  implementation  of national  reforms  without  the  technical  support  they 
received through the SRSP only to a limited extent or not at all. In addition, 9 out 26 stakeholders indicated 
they  had  no  opinion/cannot  answer  the  question,  with  6  out  26  stakeholders  considering  that  Member 
States  could  have  achieved  similar  results  in  their  design  and  implementation  of  reforms  without  the 
technical support they received through the SRSP. 
71 Answers to open-ended questions indicate that beneficiary authorities perceived the following as some of 
the  most  important  sources  of  added  value  provided  by  DG  REFORM:  (i)  expertise  and  support  when 
assessing  the  needs,  (ii)  defining  project  objectives  and  terms  of  references,  (iii)  planning  the  timing  of 
project activities and deliverables, reviewing and supervision of the quality of project deliverables, and (iv) 
setting requirements for experts. 

40 

 
 
of  the  support  received  by  beneficiaries,  compared  to  what  is  available  at 
national/regional level.  

The SRSP (2017-2020) had strong additionality with  only  a few beneficiary authorities 
being  able to  receive similar technical support at  national,  regional or local  levels  (i.e., 
without  a  SRSP  intervention).  Access  to  international  expertise,  solutions  tailored 
specifically for the Member State in concern, the low administrative burden and the EU’s 
role in facilitating and legitimising the reforms were all identified as the key sources of 
the SRSP’s value added. 

Cross-border and Union-wide impacts of the SRSP 

A large share of SRSP (2017-2020) projects was related to some cross-border/union wide 
impacts, as shown by the collected data:  720 out  of 826 SRSP-funded projects  (around 
87%) covered in the evaluation study reported at least one such impact, with only 106 (or 
13%) of the projects indicating no EU value added/cross-border impacts (see Figure 17).  

FIGURE  18:  SHARE  OF  SRSP-FUNDED  PROJECTS  (2017-2020)  INDICATING  SPECIFIC  EUROPEAN 
ADDED VALUE 

Contribution to the consistent and coherent
implementation of Union law and policies, as well as
the promotion of European values, including solidarity

Contribution to the sharing of good practices, also with
a view to increasing the visibility of the reform
programmes, and to building a Union-wide platform
and network of expertise

43%

43%

Impact on cross-border or Union-wide challenges

33%

Promotion of mutual trust between beneficiary Member
States and the Commission and of cooperation among
Member States

23%

Source: DG REFORM internal monitoring system (N=826). 

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

The  above  SRSP’s  cross-border  and  Union-wide  impacts  were  also  confirmed  by  the 
targeted survey of coordinating authorities72.  

72  The  majority  (18  out  of  21,  or  86%)  of  coordination  authorities  who  responded  (respondents) 
agreed/strongly  agreed  that  the  technical  support  increased the  sharing  of  knowledge  and  good practices 
between beneficiary  authorities in their country and institutions from other countries. Similarly, 77% (17 
out  of  22)  of  the  respondents  agreed/strongly  agreed  that  the  technical  support  increased  mutual  trust 
between  beneficiary  authorities  in  their  country  and  the  Commission,  with  another  88%  (15  out  of  17) 
agreeing/strongly  agreeing  that  the  technical  support  helped  to  address  cross-border  or  Union-wide 
challenges in their country. 

41 

 
 
 
 
 
Beneficiary authorities not only shared good practices and lessons learned, but also quite 
actively used them after the end of technical support projects. According to the targeted 
survey results, 88% of beneficiary authorities (105 out of 119) used to a large or to some 
extent the good practices and lessons learnt from the specific technical support projects 
after  their  end.  Some  beneficiary  authorities  participating  in  the  focus  groups  also 
reported  that  they  benefitted  from  opportunities  to  collaborate  and  share  knowledge 
between  the  SRSP-funded  projects  in  the  same  Member  State,  particularly  where  the 
coordinating authority played  a facilitation role  (e.g., by organising annual  meetings of 
the SRSP-funded projects). 

the 

internal 

restructuring  of 

In  some  cases,  the  SRSP-funded  projects  directly  contributed  to  the  implementation  of 
EU  priorities  and  policies,  as  showcased  in  Latvia  where  several  different  projects 
facilitated 
the 
implementation  of  the  national  reforms  related  to  EU  policies.  Similarly,  Croatia 
implemented  SRSP-funded  projects  in  the  area  of  curriculum  reform.  This  was  done 
within the context of a broader EU reform policy framework in which the Commission 
provided support to  EU  Member States to  upgrade their education systems  in  line with 
the European Education Area’s strategy.  

institutions  and  procedures  and 

In  terms  of  sharing  good  practices  among  Member  States,  almost  all  the  projects 
analysed in the case studies included the use of good practices from other Member States. 
Sharing  of  good  practices  either  occurred  through  direct  channels  of  communication 
amongst themselves developed during SRSP projects or were facilitated by the European 
Commission and occurred through specific dedicated platforms (e.g., in case of the TJTP 
projects,  a  Just  Transition  Platform  was  set  up  as  a  single  access  point  to  provide 
beneficiaries with opportunities to self-reflect and share good practices73). In some cases, 
SRSP-funded projects contributed to the sharing of good practices among institutions of 
the same Member State.  

In  general,  the  case  study  evidence  indicates  that  one  of  the  key  factors  behind  the 
establishment of cooperation ties among individual Member States was similarities in the 
specific national  reform  priorities and similar challenges faced by specific EU Member 
States. 

The  number  of  multi-country  projects  addressing  the  reform  needs  of  at  least  two 
Member  States  reflects  the  cross-border  or  Union-wide  impacts  of  the  SRSP.  The 
available evidence shows that the use of multi-country projects in the SRSP (2017-2020) 
was  limited;74  the  SRSP  did  not  provide  explicitly  the  possibility  of  submitting  multi-
country  requests  from  different  Member  States,  this  was  rather  a  limiting  factor  in  the 
programme’s  practice.  The  results  of  both  desk  research  and  stakeholder  consultations 
show a need to  enhance  the multi-country dimension  and  call  for a more active role of 

73 Interviews with stakeholders also indicate that some delivery modes (e.g. TAIEX focusing on peer-to-
peer  cooperation)  were  particularly  well-suited  for  the  sharing  of  good  practices  and  the  exchange  of 
information between different national authorities.  

74  During  the  period  2017-2020,  there  were  only  9  multi-country  SRSP-funded  projects  out  of  826  total 
projects covered in the evaluation (i.e., around 1%). 

42 

 
 
the  Commission  in  facilitating  multi-country  projects  and  sharing  technical  support 
outputs across borders.  

Multi-country  projects  represent  only  one  type  of  multi-beneficiary  projects  supported 
under  the  SRSP  2017-2020.  The  programme  also  provided  the  opportunity  to  form 
projects involving multiple beneficiary authorities from the same country (e.g., different 
ministries),  as  well  as  multi-regional  projects  (e.g.,  projects  involving  regional 
institutions from the same country or regional institutions across different countries). The 
SRSP  thereby  provided  the  framework  not  just  for  cross-border  cooperation  and  good 
practice exchange, but also for cross-regional cooperation. 

Some stakeholders suggested specific measures for a more active role of the Commission 
in facilitating collaboration between potential beneficiary authorities in different Member 
States with  similar needs and priorities;  for  example  by facilitating the  networking  and 
exchange of information between Member States about their common challenges, reform 
needs and good practice examples, by using the newly created coordinating authorities’ 
network, which could facilitate more direct exchange of experience and knowledge about 
common challenges between Member States.  

These  actions  could  create  more  favourable  conditions  for  Member  States  to  identify 
synergies and complementarities, leading to the set-up of multi-country technical support 
projects while fully respecting the demand driven SRSP approach. Cooperation between 
the  Member  States  outside  the  programme’s  framework  could  also  be  facilitated  by 
further  encouraging  the  establishment  of  communities  of  practice  involving  experts  in 
interested Member States in specific areas, including after project completion. 

The results and impacts of multi-country projects should also be closely monitored by the 
programme  in  parallel  to  the  above  efforts.  The  impacts  of  the  multi-country  projects 
(particularly, as compared to single-country projects) were not systemically assessed yet, 
as indicated by the evaluation report. 

Evidence  from  the  consultation  of  stakeholders  concludes  overall  that  the  SRSP  had 
significant  cross-border  and  Union  wide  impacts,  in  particular  in  terms  of  the 
contribution to the consistent and coherent implementation of the Union law, promotion 
of  European  values  and  contributing  to  the  sharing  of  good  practices  between  the 
Member States. At the same time, the use of multi-country projects was very limited in 
the  SRSP.  This  suggests  that  the  Commission  could  play  a  more  proactive  role  in 
promoting applications to participate in and in facilitating multi-country projects. 

4.3 Is the intervention still relevant? 

This  section  addresses  the  relevance  of  SRSP  actions  for  enhancing  the  administrative 
and institutional capacity of Member States to design and implement the reforms needed 
to  tackle the challenges  faced;  as well as the  relevance of the  actions  funded under the 
SRSP to address the needs of beneficiary authorities and Member States. 

Relevance of SRSP actions for enhancing the administrative capacity of Member States  

43 

 
 
The  targeted  survey  results  indicate  that  at  the  moment  of  requesting  technical  support 
beneficiary  authorities  faced  the  challenges  that  were  also  set  out  in  the  ex-ante 
evaluation  of  the  SRSP:  limited  administrative  and  institutional  capacity,  challenges  in 
the design  and implementation  of structural  reforms in  line with  the Union’s economic 
and social goals and challenges in the application and implementation of Union law (see 
figure 18).  

The  incidence  of  these  key  challenges  faced  by  the  beneficiary  authorities  generally 
remained similar to the results of the mid-term evaluation.  

FIGURE  19:  CHALLENGES  FACED  BY  BENEFICIARY  AUTHORITIES  AND  ADDRESSED  BY 
TECHNICAL SUPPORT 

Limited administrative/institutional capacity

72%

68%

90%

Challenges in the design and implementation of
structural reforms in line with the Union's
economic and social goals

30%

37%

68%

Other - write in

13%

16%

14%

Challenges in the application and
implementation of Union law

11%

13%

23%

Beneficiary authorities (N=132)

Technical support providers (N=106)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

National coordinating authorities (N=22)

Source: the targeted survey of beneficiary authorities, coordinating authorities and technical support providers, May-
June 2022. 

All stakeholders viewed very positively the relevance of the SRSP’s design and structure 
for achieving its goals, according to the targeted survey.75  

94%  of  surveyed  beneficiary  authorities  and  coordinating  authorities  (126  out  of  134) 
also agreed/strongly agreed that the objectives of technical support projects corresponded 
to the key reform goals  of their country. There were no significant differences between 
different groups of stakeholders in terms of their views on the relevance of the SRSP’s 
design  and structure.  This  highly positive assessment of the programme’s structure and 
design  was very similar  to  the situation found during the mid-term evaluation of 2019, 
when 97% of survey respondents agreed that the SRSP is a suitable instrument to provide 
technical support. 

75 Around 98% of the surveyed beneficiary authorities and coordinating authorities (149 out of 152) agreed 
or strongly agreed that the SRSP was a suitable instrument to provide technical support, around 96% (144 
out of 150) agreed that their project design (identification of the problem, definition of the objectives and 
results  to  achieve,  definition  of  the  activities  to  carry  out)  was  appropriate.  Similarly,  technical  support 
providers very positively assessed the design and structure of the SRSP, with 98% of them (101 out of 104) 
agreeing  the  SRSP  was  a  suitable  instrument  to  provide  technical  support  and  97%  of  them  (101  out  of 
105) agreeing that the project design was appropriate. 

44 

 
 
 
Similarly,  data  from  the  satisfaction  questionnaires  of  DG  REFORM  show  that  the 
beneficiary  authorities  very  positively  assessed  the  DG  REFORM’s  help  in  designing 
projects  adequately  to  address  their  reform  needs  (identification  of  the  problem, 
definition  of  the  objectives  and  results  to  be  achieved,  definition  of  the  activities  to  be 
carried out).  

DG REFORM policy officers also positively assessed their own contribution in terms of 
designing  projects  in  a manner  that  could  adequately  address  the  needs  of  the  Member 
State concerned (average assessment score 8.6; N=382).  

The relevance of project design for addressing the needs of the Member States depended 
on the quality of technical support requests submitted by the Member State, as suggested 
by  the  evaluation  report  (analysis  based  on  the  SRSP  2017-2020  monitoring  data):  the 
relevance was higher for projects where technical support requests were more mature to 
allow  for  speedier  delivery  of  the  support,  as  well  as  for  the  projects  where  measures 
requested by the Member States were targeted enough for addressing the reform needs.76  

Looking at the relevance of the SRSP for addressing the specific types of reforms needed 
by  the  Member  States,  the  analysis  of  the  circumstances  of  the  selected  requests  (see 
Table 2) indicated that the requests for technical support were mostly relevant to address 
the economic governance process, including CSRs, EU priorities and reforms initiated by 
Member States.  

According  to  the  results  of  the  exploratory  interviews,  this  achievement  was  made 
possible through the closer alignment of the SRSP with the European Semester process 
and  by  considering  the  technical  support  requests  related  to  CSRs  during  the  selection 
process.  However, according  to  these interviews, there has been no assessment on how 
many  CSRs  have  been  addressed  by  Member  States  due  to  the  provision  of  technical 
support. 

The available evidence indicates that overall, the SRSP was a highly relevant instrument, 
well-tailored to enhancing the administrative capacities of Member States to design and 
implement the reforms needed to address the challenges faced.  

Relevance of SRSP actions to  address the needs of beneficiary authorities and Member 
States 

All groups  of stakeholders  involved in SRSP-funded projects  very positively  assess the 
relevance  of  technical  support  to  the  needs  of  Member  States.  96%  of  surveyed 
beneficiary  authorities  and  coordinating  authorities  (145  out  of  150)  agreed  or  strongly 

76  There  is  a  significant  statistical  relation  between  the  assessment  of  the  project's  design  in  terms  of  its 
relevance  for  addressing  the  needs  of  the  Member  State  and  the  assessment  of  the  quality  for  technical 
support requests by the policy officers. More specifically, there is a strong positive correlation between the 
assessment  by  policy  officers  of  the  project  design's  relevance  for  the  Member  States’  needs  and  the 
assessment  of  the  maturity  (to  allow  a  quick  delivery  of  the  support)  of  the  requests  for  the  technical 
support submitted by the Member States. Similarly, there is a strong correlation between the assessment of 
the project design's relevance for the Member States needs and the assessment of the measures requested 
by the Member States (i.e., whether they were targeted enough for addressing the reform needs). 

45 

 
 
 
agreed  with  the  statement  that  their  project  addressed  the  key  reform  needs  of  the 
beneficiary institution. There were no significant variations in the distribution of answers 
among these groups of stakeholders.  

the  European  Commission  acted  as  a  contracting  authority  during 

Since 
the 
implementation  of  the  SRSP,  it  was  responsible  for  the  selection  of  technical  support 
providers.  However,  in  their  requests  for  technical  support,  BAs  were  able  to  provide 
suggestions  on  the  delivery  mode  and  the  technical  support  provider.  60%  of  surveyed 
BAs (78 out of 130) indicated that such suggestions were accepted to a large extent or to 
a  moderate  extent,  with  only  5  beneficiaries  out  of  130  (6%  of  surveyed  beneficiary 
institutions)  indicating  that  such  suggestions  were  not  considered  at  all.  Nevertheless, 
some respondents highlighted the need for a closer involvement of beneficiary authorities 
during the selection of technical support providers.  

Different sources of evidence also clearly indicate that the technical support provided by 
the  SRSP  remained  highly  relevant  for  the  needs  of  both  EU  Member  States  and  their 
beneficiary  authorities.  The  evidence77  on  the  increasing  number  of  infringement  cases 
demonstrated  that  technical  support  aiming  at  a  more  effective  application  and 
implementation of Union legislation could still be needed. However, since individual EU 
Member  States  did  not  initiate  many  technical  support  requests  in  this  field  during  the 
implementation  of  the  demand  driven  SRSP,  the  number  of  technical  support  projects 
addressing specific challenges in the application and implementation of EU law remained 
rather low.  

Also,  some  deterioration  in  the  annual  implementation  rate  of  CSRs  during  the  period 
2015-2019 indicates the continued relevance of the technical support aimed at addressing 
the challenges identified in the context of the economic governance processes. Although 
the increasing number of technical support projects addressed these challenges (e.g., 62% 
of  such  projects  in  2019),  such  changes  were  expected  to  arise  later  due  to  the  long 
logical chain between the activities of technical support and the expected results/impacts 
of the programme.  

The  growing  need  for  the  technical  support  provided  by  SRSP  was  showcased  by  the 
increasing  number  of  requests  from  the  Member  States,  as  well  as  by  the  increasing 
budget of the projects, according to the evidence analysed in the evaluation report. These 
trends indicated a growing interest from the Member States, as well as the Programme’s 
lasting relevance for designing and implementing significant reforms. 

The  demand  for  technical  support  varied  among  different  Member  States,  with  the 
highest  overall  numbers  of  SRSP-funded  projects  found  in  some  of  the  Southern 
(Greece),  as  well  as  Central  and  Eastern  European  countries  (Romania  and  Croatia). 
Fewer projects were supported in the Nordic countries, Luxembourg, Germany, and the 
Netherlands. 

77  1,786  cases  at  the  end  of  2020  compared  to  1,347  cases  at  the  end  of  2014,  European  Commission 
(2021), Commission Staff Working Document: General Statistical Overview Accompanying the document 
‘Report  from  the  Commission  Monitoring  the  application  of  European  Union  law  2020  Annual  Report’. 
https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52021SC0212&rid=10  

46 

 
 
Despite the smaller number of SRSP-funded projects, qualitative evidence shows that the 
support provided in these countries was highly relevant for their national reform needs: 
e.g.,  in  the  case  of  Luxembourg,  technical  support  directly  contributed  to  the 
modernisation of the national customs system.  

Evidence from the interviews with stakeholders shows that one of the key reasons for a 
limited number of requests for technical support in some Member States was the timing 
of  evaluation  and  selection  procedures  in  the  programme,  which  did  not  always  match 
with the timing of the national reform agenda. Another reason for the limited number of 
requests  in  some  Member  States  was  a  lack  of  awareness  of  the  SRSP’s  benefits  and 
functioning among some of potential beneficiary institutions.  

The  interviews  and  focus  group  discussions  generally  confirmed  the  high  relevance  of 
technical  support  to  the  needs  of  Member  States  and  beneficiary  institutions.  Some 
respondents,  however,  noted  that  the  increased  focus  of  the  SRSP  on  larger  projects 
limited access to the programme for smaller beneficiary authorities.  

The  case  study  evidence  also  supports  the  finding  that  the  SRSP  actions  were  highly 
relevant  for  the  needs  of  beneficiary  authorities,  with  a  large  majority  of  the  projects 
analysed under the case  study programme directly addressing the needs of the Member 
States  beneficiary  authorities.  Overall,  most  of  the  projects  covered  under  the  case 
studies  were  highly  relevant  for  addressing  the  most  important  national  issues78.  For 
example,  all  the  SRSP-funded  projects  analysed  in  the  Italian  case  study  focused  on 
important national reform priorities and structural challenges. These projects were highly 
relevant and specifically designed to address the needs of Italian beneficiary authorities.79  

BOX  5  -  The  provision  of  technical  and  other  support  to  EU  Member  States  in 
the field of accounting reforms  

In  the  field  of  revenue  administration  and  public  financial  management  (where  140 
technical  support  projects  were  implemented),  the  SRSP  provided  technical  support 
for  the  Member  States  to  reform  public  financial  and  asset  management,  budget 
processes,  debt  management  and  revenue  administration.  Within  this  field,  several 
projects  specifically  supported  Member  States  to  reform  their  public  sector  financial 
accounting and reporting systems in line with the European Public Sector Accounting 
Standards (EPSAS).  

The EPSAS addresses the lack of common accounting standards for the public sector 
in  the  EU  (unlike  the  private  sector,  where  such  standards  are  in  place).  It  aims  to 
increase the transparency and comparability of public sector financial accounting and 
reporting  between  and within  EU Member States by developing  and implementing a 
harmonised European accounting framework. This will inform governance, policy and 
decision-making and the management of public finances. 

78 Case studies’ projects were not chosen on the basis of circumstances of requests. 
79  For  instance,  Italy’s  project  (‘Support  for  the  Implementation  of  the  Accrual  IPSAS/EPSAS-based 
Accounting  in  the  Italian  Public  Administration’)  on  accounting  reforms  related  to  the  European  Public 
Sector Accounting Standards (EPSAS) and International Public Sector Accounting Standards (IPSAS) was 
closely  in  line  with  national  reform  priorities.  Similarly,  the  SRSP-funded  project  in  Ireland  (‘Financial 
Reporting Reforms’) closely corresponded to the pending reform needs of the country. 

47 

 
 
standards,  an  operational  manual 

For example, one project in Italy supported the drafting of a conceptual framework for 
the  accrual  accounting 
accounting 
implementation and new accounting policies, as well as delivering training on accrual 
accounting.  The  project  also  supported  reform  in  line  with  Council  Directive 
2011/85/EU  on  requirements  for  budgetary  frameworks  of  the  Member  States  and 
addressed CSRs within the European Semester. Two projects in Ireland included study 
visits to best practice examples of reforms in Italy, Sweden and the UK. 

for 

These  projects  supporting  the  reform  of  public  sector  financial  accounting  and 
reporting  systems  helped  bring  about  consensus  in  favour  of  reform  and  build 
capacity, as well as supporting the introduction of the reforms themselves. The outputs 
of  the  project  in  Italy  (accrual  accounting  conceptual  framework,  manual  and 
accounting  standards)  have  been  used  as  reference  by  the  Italian  Standard  Setter 
Board.  It  also  strengthened  project  management  culture  within  the  Ministry  of 
Economy  and  Finance.  The  projects  in  Ireland  provided  the  Department  of 
Expenditure  and  Reform  with  crucial  insights  from  best  practices  in  other  countries. 
This  helped  prepare  the  way  for  the  introduction  of  EPSAS,  with  the  accruals 
accounting system due to be put in practice after the end of the project. 

Sources:  the  European  Public  Sector  Accounting  Standards  (EPSAS);  SRSP  ex-post 
evaluation external independent study. 

Finally, qualitative evidence also indicates that SRSP is perceived as highly relevant in 
the  context  of  European-wide  challenges  and  trends  that  will  likely  define  the 
European  policy  agenda  in  the  future.  Overarching  challenges  –  such  as  climate 
change,  environmental  degradation,  digital  hyperconnectivity,  and 
technological 
transformation  –  pose  a  need  for  a  closer  strategic  cooperation  between  the  Member 
States, greater exchange of good practices and implementation of structural reforms (e.g., 
in  the  area  of  reducing  dependencies  on  fossil  fuels).  In  this  context,  the  SRSP  and  its 
successor programme will sustain its high relevance for the Member States and the whole 
of the EU. 

BOX  6 –  The provision of technical and other support to EU Member States in 
the field of sustainable growth & business environment 

In the field of sustainable growth and business environment (where the largest number 
of  technical  support  projects  was  implemented),  DG  REFORM  provided  technical 
support for the Member States in the field of actions to accelerate the twin climate and 
digital  transitions  and  the  economic  recovery,  including  the  reforms  and  investments 
that  strengthen  the  cohesion  and  competitiveness  of  the  economy.  During  the 
implementation of the SRSP, technical support to the ‘green’ projects was significantly 
increased, with around one in four projects addressing the goals of the European Green 
Deal as of 2020.  

Typical projects in the field of sustainability included advice on the reforms to increase 
energy  efficiency,  promote  climate  mitigation,  enable  a  just  transition,  boost  the 
circular  economy,  protect  the  environment,  and  expand  low-emission  transport.  In 
2020, DG REFORM launched the first dedicated call for technical support projects to 
support  the  preparation  of  Member  States’  TJTPs  (see  Annex  4  on  the  results  of  the 
case study programme). In the field of business environment, typical projects focused 
on  the  actions  to  reduce  market  barriers,  facilitate  foreign  investment  and  trade, 

48 

 
develop the digital economy, stimulate research and innovation, manage public–private 
partnerships and improve the governance of state-owned enterprises.  

The thematic field of sustainable growth and business environment was characterised 
by  the  frequent  adoption  of  a  strategy  or  a  new  law  (as  a  type  of  outcome).  Also, 
compared  to  other  thematic  fields  this  field  saw  relatively  more  changes  in  terms  of 
new legal acts adopted or existing legal acts modified, as well as improved application 
and  implementation  of  EU  law,  but  the  scope  of  legal  changes  was  only  moderate, 
largely  such  challenges  were  not  frequently  addressed  during  the  formulation  of 
requests for technical support or some resistance occurred during the implementation 
of  economic  reforms  (with  external  factors  usually  playing  a  significant  role  in  this 
sector).  

Sources: European Commission (2021), Annual Activity Report 2020: DG REFORM – 
Directorate-General  for  Structural  Reform  Support,  p.  14;  European  Commission 
(2020),  Directorate-General  for  Structural  reform  Support:  Growth  and  business 
environment.  https://commission.europa.eu/publications/annual-activity-report-2020-
structural-reform-support_en; SRSP ex-post evaluation external independent study. 

Overall,  the  evaluation  evidence  indicates  that  the  technical  support  provided  by  the 
SRSP  was  highly  relevant  to  address  the  needs  of  beneficiary  authorities  and  Member 
States.  The  evidence  also  shows  that,  despite  the  programme’s  contribution  in  the  last 
few years, beneficiary authorities still need the technical support provided by the SRSP.  

5.  What are the conclusions and lessons learned? 

5.1 Conclusions 

The  Structural  Reform  Support  Programme  was  adopted  in  2017  to  provide  technical 
support  to  national  authorities  and  help  Member  States  design  and  implement 
institutional, administrative and growth-sustaining structural reforms.  

Overall,  the  implementation  of  the  2017-20  SRSP  was  successful  in  terms  of 
effectiveness,  efficiency  and  coherence.  The  programme  was  also  found  to  be  highly 
relevant  in  addressing  the  needs  of  beneficiary  authorities  and  EU  Member  States  and 
had an EU added value (value above what the Member States could have achieved acting 
unilaterally).  

Effectiveness 

The  programme  successfully  achieved  its  general  objective,  namely,  to  contribute  to 
institutional, administrative and growth-sustaining structural reforms in the EU Member 
States.  Furthermore,  it  has  substantially  advanced  the  reform  agenda  in  EU  Member 
States. The actual achievement of different types of reforms also depended on the wider 
efforts made by EU Member States and not only on the results of other EU programmes 
and schemes. 

The  SRSP  contributed  most  to  achieving  its  specific  objectives  concerning  “the 
development  and  implementation  of  reform  policies”,  as  well  as  “the  definition  and 

49 

 
 
implementation  of  processes  and  methodologies”,  followed  by  the  specific  objective 
concerning “the design of reforms according to the priorities of national authorities”. 

Also,  the  programme  fully  achieved  its  expected  results;80  almost  all  programme 
stakeholders who participated in the consultation activities were highly satisfied with the 
results  and  the  success  of  the  technical  support  projects.  For  instance,  according  to  the 
targeted  survey  of  beneficiary  authorities,  94%  of  respondents  were  very  satisfied  or 
satisfied  overall  with  their  participation  in  the  SRSP  and  with  the  results  of  technical 
support projects. 

The  programme  contributed  substantially  to  the  design  and  implementation  of 
administrative, institutional and structural reforms in EU Member States, with potential 
benefits for European businesses and citizens/consumers. 

Where the projects have not generated the full intended results and impacts, this tends to 
reflect wider challenges in the reform process rather than any inherent weaknesses in the 
projects or in the provision of technical support. And more policy impacts were expected 
to  occur  in  future  because  some  one  third  of  all  evaluated  projects  were  still  ongoing 
during the ex-post evaluation and more time was necessary for the expected results and 
impacts  to  occur,  due  to  the  long  lag  between  the  technical  support  activities  and  the 
expected results/impacts.  

The SRSP contributed more to administrative reforms and capacity building than to the 
implementation  of other  structural  reforms.  In  fact,  administrative reforms are  easier to 
implement:  they  require  fewer  actors  (who  usually  come  from  the  public  sector),  less 
time to be implemented and bring more direct results.  

Other types of structural reform are more complex in nature, require the involvement of 
more  stakeholders  (including  the  private  sector)  and  take  more  time  for  their  results  to 
bear  fruits.  In  addition,  these  types  of  reforms  are  more  dependent  on  external  factors. 
Member  State  beneficiary  authorities  did  not  initiate  many  technical  support  projects 
aimed at advancing such reforms, especially in the initial rounds of the SRSP (2017 and 
2018).  

Overall,  communication  activities  were  rarely  included  in  project  design,  despite  an 
increase in the number of communication initiatives in the 2019 and 2020 rounds of the 
programme.  However,  engagement  and  communication  with  external  stakeholders  – 
when they happened – were crucial during the provision of technical support, because the 
effective  implementation  of  such  activities  increased  the  legitimacy  of  the  reforms  and 
their quality and sustainability, and reduced organisational resistance to change.  

Overall,  the  growing  complexity  of  technical  support  projects  (in  terms  of  number  of 
activities  and  outputs)  might  make  the  implementation  of  the  projects  and  the 

80 Based on the responses to the outcome questionnaires by the beneficiary authorities, the average  rating 
for  having  achieved  the  specific  outcomes  of  the  SRSP-funded  projects  was  8.2  out  of 10  (with  66%  of 
technical  support  projects  achieving  their  outcomes,  based  on  average  scores  8.0  and  above).  In  the 
targeted  survey  of  beneficiary  authorities,  31%  of  respondents  achieved  all  expected  results  and  65%  of 
respondents  achieved  some  expected  results,  both  during  the  implementation  of  the  technical  support 
projects and after their completion. 

50 

 
 
achievement of their expected results more difficult in future. This risk should be taken 
into account when designing any future multi-country projects, which are by their nature 
more complex than single-country projects.81 

Almost  all  consulted  stakeholders  were  highly  satisfied  with  the  quality  of  technical 
support  provided  on  the  ground.  The  choice  of  providers  was  crucial  to  the  success  of 
technical  support  projects  due  to  the  importance  of  having  adequate  professional  skills 
and a good understanding of the national  situation. This  reflects  the added value of the 
European Commission in choosing and contracting the best expertise on the market for 
each specific project.  

The strong involvement of the beneficiary authorities and senior management ownership 
of  the  projects  is  key  to  achieving  the  results.  Their  involvement  is  essential  not  only 
during  project  implementation,  but  also  in  the  pre-implementation  phase,  in  terms  of 
defining the project objectives and timelines and creating ownership and commitment to 
change. 

While  absorbing  technical  support  and  implementing  the  recommendations  made, 
beneficiary  authorities  sometimes 
lacked  adequate  administrative  capacity.  This 
hampered the smooth running of the technical support projects and the implementation of 
resulting  recommendations.  While  the  provision  of  technical  support  does  not  involve 
any  costs  for  the  beneficiary  authorities,  they  sometimes  lacked  the  sufficient  financial 
resources to implement project recommendations and relevant reforms. 

Efficiency 

The  SRSP  proved  successful  in  attracting  many  high-quality  requests  relative  to  the 
budget  available.  Moreover,  the  programme  was  able  to  support  projects  of  increasing 
size  and  budget,  thus  increasing  the  potential  for  projects  to  achieve  greater  impact. 
Growing  demand  for  technical  support  among  the  EU  Member  States  and  a  sufficient 
level of competition among the projects allowed high-quality proposals to be selected.  

The  programme’s  stakeholders  – 
including  beneficiary  authorities,  coordinating 
authorities, and providers – generally agreed that the financial contribution of the SRSP 
was  proportionate  to  the  needs  and  estimated  scope  of  technical  support,  and  they 
considered SRSP projects to be  a user-friendly and cost-effective instrument. The costs 
of controls on total annual payments by DG REFORM were comparable with other EU 
programmes,  such  as  Horizon  2020,  under  direct  management  mode.  DG  REFORM 
achieved good results in executing the SRSP budget. 

Most  notably,  the  assessment  and  selection  of  requests  for  technical  support  was 
completed on time and in line with the target set in the Financial Regulation, compared 
with other EU programmes. However, there is room for improvement in speeding up the 
preparation for the implementation phase, in particular the contracting process. As one of 
the selection criteria for funding a project under the SRSP was the urgency of the reform, 

81 On the other hand, multicountry projects were able to increase efficiency by providing synergies between 
the authorities involved. 

51 

 
 
Member  States  expect  the  project  to  be  operational  as  early  as  possible  to  ensure  the 
reforms remained relevant and were implemented on time. 

The  stakeholders  involved  in  the  programme  were  generally  positive  about  the  clarity, 
transparency  and  user-friendliness  of  the  SRSP’s  administrative  procedures  and  the 
proportionality  of  the  administrative  burden.  Areas  for  improvement  are  related  to  the 
need to further develop and refine project monitoring and evaluation practices, as well as 
communication activities. 

DG REFORM efficiently and effectively  coordinated the technical support given to the 
Member  States  with  other  Commission  departments.  According  to  the  outcome  of  the 
study,  in  2017-20,  DG  REFORM  cooperated  more  closely  with  other  DGs  and 
Commission  departments  while  disseminating  and  using  the  outputs  from  the  technical 
support once projects had been completed.  

Coherence 

The  SRSP  is  evaluated  to  be  a  coherent  policy  tool  with  a  clear  internal  logic,  well 
aligned  with  the  objectives  set  out  in  the  SRSP  Regulation,  with  no  major  gaps, 
inconsistencies  or  overlaps  in  the  internal  structure  of  projects.  The  design  of  the 
programme allowed for considerable flexibility when selecting project activities that are 
best tailored to specific project objectives and beneficiary needs.  

The  structure  and  activities  of  the  SRSP  were  adjusted  over  time  to  better  respond  to 
newly emerging challenges and to achieve its objectives in a more effective way, e.g., by 
introducing  dedicated calls  for requests that met  the newly  emerging needs of  Member 
States. However, the discrete nature of the projects means that additional effort could be 
required  to  disseminate  and  exploit  project  outputs  to  wider  audiences,  so  that  greater 
impact can be generated. 

Evidence  also  shows  that  the  SRSP  was  highly  consistent  with  other  EU  policies  and 
actions (including ESF, ERDF, COSME, Fiscalis), as well as those of the Member States 
themselves. This was because it addressed different aspects of similar objectives, issues 
and target groups. Having common themes with other EU-level programmes, the SRSP-
funded  projects  largely  complemented  these  interventions  by  providing  tailor-made 
technical support for national authorities in specific sectoral areas.  

In  terms  of  consistent  objectives,  the  SRSP  reinforced  wider  reform  agendas  of  the 
Members  States  and  the  EU,  including  by  helping  address  CSRs  made  as  part  of  the 
European  Semester  process.  Generally,  the  stakeholders  of  the  SRSP  highly  valued  the 
programme’s  contribution  to  the  Semester  process.  In  some  cases,  the  SRSP-funded 
projects  helped  better  define,  clarify  and  operationalise  the  CSRs,  as  well  as  providing 
specific guidelines for Member States to implement them. In other instances, the results 
of SRSP-funded projects were used to formulate these recommendations.  

Although overall there were sufficient mechanisms to ensure the linkages and integration 
between the SRSP and the European Semester process/CSRs, the study showed that there 
is insufficient monitoring on how the SRSP-supported projects contribute to meeting the 

52 

 
CSRs.  There  was  also  a  lack  of  references  to  SRSP  projects  in  the  European  Semester 
country reports.  

EU added value  

The SRSP delivered this by providing Member States with better access to international 
expertise than they would have enjoyed in the absence of the SRSP. Even in the limited 
cases when project benefits could have been realised without EU technical support, these 
would have been much smaller in scope or realised later in time. As a result, EU added 
value  was  offered  in  terms  of  the  strengthened  reform  processes  in  the  Member  States 
and  strengthened  contribution  to  achieving  EU  policy  objectives,  not  least  those  of  the 
European Semester. 

The SRSP had significant cross-border, EU-wide impacts going beyond single Member 
States. Such impacts included helping the Member States to implement EU policies and 
priorities nationally, including the promotion of European values, as well as helping them 
to share good practice  among themselves. The good practices and lessons learned were 
actively used by the beneficiaries after the end of the projects. 

Relevance  

The SRSP was found to be a highly relevant instrument, well-tailored to improving the 
administrative capacities of Member States to design and implement the reforms needed 
to address the challenges they faced.  

The beneficiary authorities and other stakeholders viewed positively the different aspects 
of the SRSP’s design and structure and its relevance for achieving the reform goals. At 
the  same  time,  the  objectives  of  technical  support  projects  largely  corresponded  to  the 
key reform goals  and  challenges faced by the Member States. The relevance of project 
design  for  achieving  the  reforms  goals  depended  on  the  quality  of  technical  support 
requests submitted by the Member States.  

The technical support provided by the SRSP was highly relevant to addressing the needs 
of  beneficiary  authorities  and  Member  States,  both  in  terms  of  the  implementation 
method  (i.e.  international  technical  support  providers  sourced  by  the  Commission)  and 
the nature of the support provided (i.e. being customised to the needs of each beneficiary 
authority). The programme – uniquely and in an effective and efficient manner – fulfils 
the need of Member States to access high-level international expertise when they are in 
the process of designing or implementing reforms.  

At the same time, where the SRSP was used  in an emergency capacity, it was not very 
suitable  for  coping  with  reform  or  capacity-building  needs  on  the  ground,  since  it  was 
restricted  by  the  timetable  laid  down  in  the  calls  for  requests.  Some  stakeholders 
suggested that the programme  introduce a possibility to  draw on a pre-selected pool  of 
experts to address urgent technical support needs in Member States.  

Despite having received the SRSP’s contribution over many years, beneficiary authorities 
still  need  the  technical  support  provided  by  the  programme.  The  growing  need  for 
technical support is also showcased by the increasing number of requests from Member 
States,  as  well  as  by  the  increasing  average  budget  for  the  projects.  Furthermore,  the 

53 

 
number  of  Member  States  who  requested  support  increased  steadily  during  the 
programme: from 16 in 2017 to 23 in 2018, 26 in 2019 and eventually all 27 in 2020. 

5.2 Lessons learned 

Following the ex-post evaluation, the main lessons learned include: 

Engagement of stakeholders 

•  Active involvement of external stakeholders was crucial during the provision of 
technical  support:  it  increased  the  legitimacy,  quality  and  sustainability  of  the 
reforms, and reduced organisational resistance to change. Key stakeholders were 
consulted  during  the  design  and  execution  of  successful  projects.  Effective 
communication  strategies  were  used  to  reach  many  target  groups  during 
implementation. 

•  Engagement  of  beneficiary  authorities  was  key  in  implementing  technical 
the  smoother 

the  reform  process, 

to  enable 

support  projects  and 
implementation of projects and their results.  

in 

•  Close  involvement  of  national  coordinating  authorities  was  important  in 
monitoring the implementation of the technical support projects, discussing their 
progress  with  the  Commission  and  national  stakeholders  and  taking  follow-up 
actions to use the outputs after project completion.  

Cross-country dimension 

•  Providing  access  to  international  expertise  for  Member  States  that  might  not 
otherwise  have  had  access  to  it  was  a  key  feature  of  the  programme.  Future 
programmes must retain this possibility.  

•  Exchanges  of  civil  servants  provided  a  basis  for  greater  use  of  peer-to-peer 
advice  within  national  administrations  in  future.  In  fact,  the  accumulated 
knowledge of civil servants in the Member States represented an enormous pool 
of expertise and skills.  

•  Disseminating  the  results  of  technical  support  projects  within  and  across 
Member  States  encouraged  the  sharing  and  replication  of  good  practice.  DG 
REFORM  was  able  to  play  a  more  active  role  in  clustering  similar  projects  or 
creating communities of practice and disseminating results effectively to the main 
relevant stakeholders.  

•  Transnational  dimension  of  the  programme  –  for  example  through  more 
projects that support several Member States and through better dissemination of 
results  to  wider  audiences  –  was  instrumental  in  strengthening  the  ‘EU  added 
value’. 

Facilitator role of the European Commission 

•  Reducing the complexity of technical support projects (in terms of number of 
activities,  outputs  and  outcome)  made  their  implementation  easier  and  more 
effective.  

•  The capacity to provide tailored solutions resulted in low administrative burdens 
for  beneficiary  authorities  and  ongoing  guidance  for  them,  and  helped  facilitate 
and legitimise the reforms. 

54 

 
•  The  Commission  played  a  valuable  role  in  facilitating  collaboration  between 
potential  beneficiary  authorities  in  different  Member  States  who  had  similar 
needs and priorities – including though joint requests for multi-country projects. 
None of this interfered with the programme’s demand-driven logic.  

Effective implementation of reforms and coordination with other EU processes 

•  Improved  monitoring  and  evaluation  contributed 

to  a  more  efficient 
implementation of the programme and a better dissemination and exploitation of 
technical support outputs.  

•  When  in  place,  the  monitoring  of  the  SRSP’s  contribution  to  the  European 
Semester  process  and  to  the  fulfilment  of  specific  CSRs  in  specific  national 
contexts  was  significant.  However,  the  provision  of  technical  support  by  DG 
REFORM  should  not  replace  the  Member  States’  responsibility  to  address  their 
CSRs.  

•  Technical support was crucial also in areas where the subject of the request is less 

frequent, such as assistance in applying and implementing EU legislation.  

Technical Support Instrument (TSI)  

In  2021  the  Technical  Support  Instrument  replaced  the  SRSP  as  main  programme  for 
delivering  technical  support  in  2021–27.  The  TSI  was  designed  on  the  basis  of  several 
recommendations  proposed  in  the  mid-term  evaluation  of  the  SRSP  and  factoring  in 
some of the conclusions of this ex-post evaluation, such as more active  involvement of 
stakeholders,  a tighter  focus  on the cross-country dimension and increased attention on 
monitoring and evaluation. 

55 

 
 
ANNEX I. PROCEDURAL INFORMATION 

1.  LEAD DG, DeCIDE PLANNING/CWP REFERENCES 

Directorate-General for Structural Reform Support.  

Decide planning reference: PLAN/2020/9192. 

2.  EXCEPTIONS TO THE BETTER REGULATION GUIDELINES 

N/A 

3.  ORGANISATION AND TIMING 

The  initiation  of  the  ex-post  evaluation  of  the  SRSP  was  approved  by  DG  REFORM 
senior  management  and  the  Cabinet  and  the  Inter-service  Steering  Group  (ISSG) 
members were appointed. The evaluation roadmap was published on 17 December 2020, 
marking  the  official  starting  date  of  the  evaluation.  The  procurement  procedure  for  the 
external  evaluation  was  initiated  on  30  June  2021  and  the  contract  with  the  external 
contractor was subsequently signed on 13 October 2021, for a duration of eleven months. 
The  public  consultation  through  the  website  of  the  European  Commission  took  place 
between March and May 2022. After the contractor submitted the final evaluation report, 
the  Commission  started  drafting  the  staff  working  document,  based  on  the  external 
evaluation and the analysis of its own reporting and procedural documents.   

DG  REFORM  chaired  the  ISSG,  which  comprised  of  representatives  of  the  Secretary 
General,  REGIO,  DIGIT,  CNECT,  EAC,  HOME,  ENV,  GROW,  SANTE,  MOVE, 
AGRI, EMPL, TAXUD, JUST, FISMA, NEAR, ECFIN, ENER, COMP, ESTAT, RTD, 
HR, JRC and OLAF. The ISSG held four meetings.  

4.  CONSULTATION OF THE RSB (IF APPLICABLE) 

N/A 

5.  EVIDENCE, SOURCES AND QUALITY – INCLUDING EXTERNAL EXPERTISE 

An external evaluation report served as a basis for the ex-post evaluation of the SRSP.  

The contractor used a variety of research methods and sources, including: desk research, 
data  extracts  of  the  Commission  internal  IT  system,  scoping  interviews,  targeted 
interviews,  online  targeted  consultation  and  case  studies.  In  line  with  the  Better 
Regulation guidelines, the evaluation also included a public consultation exercise.  

DG REFORM (then SRSS) internal IT system was not in place at the start of the SRSP, 
but  it  was  implemented  only  afterwards.  The  system  was  evolving  significantly  and 
many questions  /  categories of data were added  only  at  later stages and not  necessarily 
retroactively  completed.  Therefore,  information  was  to  some  extent  incomplete.  This 
deficiency  was  overcome  by  triangulating  the  information  based  on  other  data  sources, 
including the interviews, online consultation and case studies. Furthermore, the response 

56 

 
rate  to  the  online-targeted  consultation  was  sufficient  and  ensured  a  balanced 
representation of stakeholders.  

Additionally, the breadth and diversity of the Programme’s activities posed an important 
methodological  challenge,  in  that  it  would  not  have  been  possible  within  the  available 
resources  and  timeframe  to  cover  all  activities  in  the  detail  needed  to  draw  robust 
conclusions.  Moreover,  much  of  the  Programme’s  support  plays  a  contributing  role 
alongside other factors, such as the action of national administrations. This contributing 
role is difficult to assess without an in-depth qualitative research.  

Lastly,  given  that  many  of  the  evaluated  projects  were  still  ongoing  at  the  time  of  the 
exercise, results and impact could only be measured to a limited extent. For projects that 
were still on the ground, the likelihood of achieving the results was assessed only to the 
extent that information  was available. The case  studies confirmed that it is  too  early to 
expect considerable long-term impacts at this stage of the Programme.  

Despite  the  above-mentioned  limitations,  the  quality  and  the  representativeness  of  the 
collected evidence is considered satisfactory. The contractor was able to collect extensive 
and meaningful data that allowed drawing conclusions.  

57 

 
 
ANNEX II. METHODOLOGY AND ANALYTICAL MODELS USED 

Evaluation framework 

In  line  with  the  Better  Regulation  Guidelines,  the  ex-post  evaluation  provided  the 
relevant  evidence  to  assess  the  SRSP’s  performance  against  the  five  criteria: 
effectiveness,  efficiency,  relevance,  coherence  and  EU  added  value.  The 
evaluation  relied  on  quantitative  and  qualitative  indicators.  Measuring  the  first  group  of 
indicators  was  informed  by  the  analysis  of  data  from  the  internal  monitoring  system  of 
DG  REFORM,  the  satisfaction  and  outcome  questionnaires,  as  well  as  responses  to  the 
targeted survey and the Open Public Consultation (OPC). In these cases, the judgement 
criterion for success was more than two thirds of technical support projects or responses 
supporting  the  expected  outcome.  Measuring  the  second  group  of  indicators  was 
informed  by  the  analysis  of  literature  and  documents,  results  of  the  focus  group 
discussions,  interviews  and  case  studies.  This  share  of  evidence  was  mostly  used  to 
provide  specific  examples,  indicate  good  practices  and  lessons  learned  as  well  as  to 
explain  mechanisms  of  change  and  the  impact  of  external  conditions  on  the 
implementation  of  the  SRSP-funded  projects.  In  addition,  the  structured  expert 
assessment  of  each  project  under  the  case  study  programme  was  employed  with  more 
than  50%  of  the  projects  supporting  the  expected  outcome  as  the  main  judgement 
criterion  for  success.  The  evaluation  matrix,  which  includes  the  evaluation  questions 
pertaining to each of the evaluation criteria, the associated indicators/judgement criteria 
as well as methods and data sources used for their evaluation, is presented in  Annex 2: 
Evaluation framework.  

The adopted evaluation framework relied substantially on the theory of change presented 
in Annex 1: Theory of change of the SRSP. The theory of  change was developed on the 
basis  of  desk  research  (especially  the  intervention  logic  of  the  programme’s  mid-term 
evaluation, but also taking into account the results of recent academic research) and was 
validated  during  the  implementation  of  stakeholder  consultations.    The  vertical  logic  of 
the  programme  –  a  causal  chain  from  policy/programme  objectives  to  impacts  – 
remained very similar to the one used during the mid-term evaluation, but our theory of 
change  added  external  factors  that  might  affect  the  performance  of  the  programme 
(especially  its  expected  results  and  impacts),  considering  the  ex-post  nature  of  this 
evaluation and the requirements of the Technical Specifications.82 

Furthermore,  the  ex-post  evaluation  referred  to  different  types  of  results  (e.g.  those 
occurring at the individual level, organisation level and policy level) or impacts (in terms 
of changes in the institutional and administrative capacity of Member States’ authorities 
or achievement of relevant longer-term growth-sustaining reforms in EU Member States) 
(see  Annex  1:  Theory  of  change  of  the  SRSP  for  more  information).  When  possible,  the 
ex-post evaluation used the results found in the prior ex-ante and mid-term evaluations 
as the points of comparison (see section 1.2 for more information).  

Data collection and analysis  

A complex methodology was deployed to collect solid evidence and provide well-informed 
answers  to  the  evaluation  questions.  It  consisted  of  extensive  desk  research, 
stakeholder  consultation  activities,  in  particular  targeted  survey,  focus  group 
discussions, a wide-ranging interview programme and the OPC, as well as in-depth case 

82 European Commission, ‘Better Regulation’ toolbox 2021, Chapter 6 – How to carry an evaluation and fitness 

check, p. 389. 

58 

 
 
 
studies. To ensure the validity and reliability of evaluation findings, the evaluation relied 
on  a  combination  of  distinct  methods  and  triangulation  of  different  data  sources,  which 
was achieved by complementing perceptions of stakeholders (e.g., interview results) with 
objective data (e.g., monitoring information). Data-based, documentary and perception-
based  sources  as  well  as  quantitative  and  qualitative  techniques  were  synthesised, 
depending on the evaluation question and the respective strengths of data and methods 
applied.  

Desk research and literature review 

The  desk  research  exercise  consisted  of  two  key  elements:  (i)  a  literature  review  of 
publicly  available  documents  and  (ii)  an  analysis  of  internal  statistical  and  monitoring 
data received from DG REFORM. The entire desk research programme was supported by 
a computer assisted qualitative data analysis (content analysis using NVivo 12 software), 
which  helped  to  structure,  manage  and  analyse  the  collected  information  as  well  as  to 
make  connections  between  different  information  sources  and  draw  summarising 
conclusions. 

The literature review served as a source of contextual information for the assessment of 
the study questions and for benchmarking and comparing the progress of the SRSP over 
time. It considered four key categories of publicly available sources of information: 

-  Documents  directly  related  to  the  SRSP  and  its  implementation 
process: the SRSP Regulation, annual work programmes and annexes; annual 
monitoring reports; the ex-ante evaluation report and the mid-term evaluation 
report; and the project-specific documents of the programme (e.g. descriptions 
of the projects, requests for services, deliverables of the projects, if available); 
-  National-level documents relevant for the analysis of the SRSP and its 
projects: 
programmes, 
assessments/studies/reports  on  the  implementation  of  the  SRSP  in  individual 
Member States or the publicly available documents of beneficiary authorities; 

national-level 

strategies 

reform 

and 

-  EU-level  strategic,  policy  and 

for 
contextualising the SRSP, including the documents on EU priorities and the 
European Semester process; 
-  Relevant academic research. 

legal  documents 

relevant 

The documents focusing directly on the implementation of the SRSP (both previous mid-
term evaluation and national level documents on the SRSP’s implementation) served as a 
basis  for the descriptive analysis, were used for country  case studies and as one  of the 
sources  for  triangulation  of  evidence  to  answer  some  of  the  study  questions.  EU  and 
national-level  strategic  documents,  policy  documents  and  legal  acts  were  of  key 
importance in contextualising the SRSP implementation (e.g. identifying the relevance of 
SRSP  for  national  strategic  reform  priorities,  coherence  of  SRSP  with  other  EU  funding 
programmes  and  their  objectives).  In  addition,  these  documents  contributed  to  the 
evaluation of the role of technical support to Member States achievement of the relevant 
long-term  growth-sustaining  reforms  as  indicated  in  the  CSPs.  Finally,  recent  academic 
research  served  as  a  source  of  evidence  for  assessing  the  SRSP’s  implementation  and 
identifying  the  relevant  external  factors  that  affect  the  success  of  structural  reforms’ 
implementation.  

During the evaluation, the best use of monitoring data already collected by DG REFORM 
was  ensured  (i.e.  administrative  data,  satisfaction  and  outcome  questionnaires  filled  in 
by  the  project  stakeholders).  A  great  deal  of  internal  monitoring/administrative  data 
sources were employed for the evaluation purposes, including: 

59 

 
 
-  Data  from  the  internal  monitoring  system  for  monitoring  programme 
implementation.  Initial  descriptive  statistics  tables  on  the  SRSP  projects 
(2017-2020), disaggregated by such dimensions as lead unit, funding source, 
country, status, topic, eligible activities, delivery modes and expected outcome 
were  prepared  by  the  evaluation  team,  building  on  two  internal  monitoring 
datasets  from  JIRA.83  The  aggregated  data  facilitated  the  estimation  of  the 
state  of  play  in  the  implementation  of  the  programme  during  the  evaluation 
period  (see  section  2.  State  of  play  in  implementing  the  programme  during 
2017-2020 above).  

-  Ex-ante  and  mid-term  evaluations.  The  evaluation  drew  on  the  results  of 
the  ex-ante  evaluation  (published  in  November  2015)  and  the  mid-term 
evaluation84 (published in 2020 using monitoring data from 2019) as points of 
comparison.  

-  Feedback  mechanism:  satisfaction  and  outcome  questionnaires.  The 
results  of  satisfaction  questionnaires,  filled  in  by  the  Member  States 
beneficiary  authorities,  two  types  of  technical  support  providers  (providers  of 
procurement and providers of grants) and policy officers of DG REFORM as well 
as  outcome  questionnaires85  completed  by  Member  States’  beneficiary 
authorities  (January  2018  –  August  2022)  were  employed  to  triangulate  data 
on  some  of  the  evaluation  questions.  The  questionnaire  data  consists  of  both 
qualitative  responses  and  numeric  scores,  ranging  from  1  (standing  for 
“Completely disagree”) to 10 (standing for “Strongly agree”). The total number 
of  responses  to  the  satisfaction  questionnaire  was  1,000  (with  the  feedback 
provided  by  DG  REFORM  policy  officers  on  382  projects,  by  beneficiary 
authorities  on  309  projects,  and  by  technical  support  providers  on  309 
projects),  while  the  total  number  of  responses  to  the  outcome  questionnaire 
was  128.  It  was  not  possible  to  estimate  specific  response  rates  to  these 
questionnaires because the evaluator did not have data on the total number of 
invitations sent out by DG REFORM.  

-  Data  relevant  for  the  cost-effectiveness  analysis.  Data  regarding  the 
SRSP’s  budget,  the  number  of  DG  REFORM  staff  involved  in  the  management 
of the programme and contracting procedures were provided as inputs into our 
cost-effectiveness analysis.  

Stakeholder consultation 

To gather the views of key stakeholders and the data needed to inform responses to the 
evaluation questions, the stakeholder consultation combined a number of techniques for 
data  collection  and  analysis  (see  Table  4).  Given  the  wide  scope  of  the  SRSP,  the 
consultation 
including  European 
Commission officials, Member State beneficiary and coordinating authorities, providers of 
technical  support,  representatives  of  industries,  businesses,  workers’  and  other 

included  different  categories  of  stakeholders, 

83 Since the provider of the study received a separate monitoring dataset on the application of different delivery 

modes (sub-tasks) per project, it became necessary to merge it with the main monitoring dataset.  

84  European  Commission,  Ex-ante  Evaluation  Accompanying  the  document  Proposal  for  a  Regulation  of  the 
European Parliament and of the Council on the establishment of the Structural Reform Support Programme 
for the period 2017 to 2020 and amending Regulations (EU) No 1303/2013 and (EU) No 1305/2013; Ernst 
&  Young  (2018),  Mid-term  Evaluation  of  the  Structural  Reform  Support  Programme  (SRSP)  2017-  2020 
(2016 
17.  
https://doi.org/10.2887/656262 

Publications  Office 

European 

009/A). 

Union, 

ECFIN 

13, 

the 

p. 

of 

85 The data excludes information on projects implemented by PPMI or CSES. 

60 

 
 
 
organisations,  other  public  authorities,  research  centres  and  consultancies,  research 
institutions  as  well  as  the  general  public.  In  addition,  to  grasp  the  varying  experiences 
across  the  EU,  each  Member  State  that  received  technical  support  under  the  relevant 
SRSP  annual  cycles  was  represented  in  the  consultation  activities.  Overall,  the 
stakeholder  consultation  strategy  enabled  to  effectively  address  a  relevant  breadth  of 
stakeholders within varying geographical coverage. 

TABLE 3. SUMMARY OF CONSULTATION ACTIVITIES 

CONSULTATIO
N ACTIVITY 

DURATION 

TARGET AUDIENCE 

NUMBER OF 
PARTICIPANTS 

NUMBER OF 
MSS 
REPRESENTED 

TYPE OF 
DATA 
ANALYSIS 

Targeted 
survey 

19 May 2022 
– 7 June 
2022 

Open 
public 
consultati
on (OPC) 

1 March 
2022 – 24 
May 2022 

Interview 
programm
e 

26 January 
2022 – 6 
September 
2022 

MS beneficiary and 
coordinating 
authorities, technical 
support providers 

MS coordinating 
authorities, 
Industry/business/ 
workers’ and other 
organisations, public 
authorities, 
researchers/consultants
, general public 

DG REFORM officials, 
Representatives of 
other DGs and EC 
services involved with 
the SRSP, MS 
beneficiary and 
coordinating 
authorities, technical 
support providers  

260 

27 

Descriptive 
statistics, 
exploratory/ 
inferential 
analysis 

26 

16 

Descriptive 
statistics 

11086 

17 

Content 
analysis 

Focus 
group 
discussion
s 

22 June 
2022 – 27 
June 2022 

Member State 
beneficiary and 
coordinating authorities 

17 

12 

Content 
analysis 

Source: Independent evaluation study. 

The targeted survey was launched in mid-May 2022 and was open for almost a month, 
which  resulted  in  a  reliable  basis  of  260  responses  (compared  to  185  in  the  mid-term 
evaluation).  The  survey  design  and  questionnaires  were  customised  to  the  expertise  of 
different  stakeholder  categories,  in  particular  the  national  coordinating  authorities  (22 
responses),  beneficiary  authorities  (132)  and  technical  support  providers  (106).  Taken 
together,  representatives  from  all  Member  States  expressed  their  opinions  in  the 
targeted  survey.  Both  open-ended  and  closed  survey  questions  were  used  to  ensure 
comparability across groups of respondents and complementarity with the results of the 
mid-term evaluation, the satisfaction and outcome questionnaires, as well as the OPC.  

In  line  with  the  Better  Regulation  Guidelines,  the  OPC  was  launched  on  1  March  2022 
and  was  open  for  12  weeks.  The  total  number  of  responses  to  the  OPC  was  26,  which 
more than doubled the amount of responses collected in the OPC conducted for the mid-

86  While  some  interviews  involved  a  few  participants,  the  count  refers  to  the  total  of  number  of  interviews 

carried out.  

61 

 
 
 
 
term  report  (11).  The  OPC  attracted  stakeholders  from  16  Member  States,  including  a 
large  share  of  representatives  of  public  authorities,  as  well  as  research  institutions, 
business  organisations  and  EU  citizens.  However,  the  results  of  the  OPC  are  not 
statistically  representative  and,  therefore,  were  only  used  to  triangulate  information 
collected during the  evaluation process. A  synopsis report  summarising the consultation 
process  and  outcomes  was  published  on  the  European  Commission’s  dedicated  ‘Have 
your say’ website.87 

The  design  of  the  evaluation  methodology  allowed  for  continuous  cross-checking  and 
verification  of  findings.  This  was  mostly  achieved  through  the  implementation  of  the 
interview programme, which was divided in four parts: exploratory interviews with the 
officials of the European Commission (17), exploratory and follow-up interviews with the 
national  authorities  (8),  case  study  interviews  (83)  and  validation  interviews  (2).  The 
interview  programme  was  designed  to  capture  a  wide  variety  of  views,  including  those 
coming  from  the  services  of  the  European  Commission,  technical  support  providers  and 
the national coordinating or beneficiary authorities. Depending on the  question at hand, 
semi-structured interviews were used either to supplement other sources of evidence or 
to  gain  novel  insights  and  validate  findings  in  cases  when  other  data  sources  were 
scarce.  

Finally,  two  focus  group  discussions  were  implemented  to  get  more  detailed  insights 
on  the  management  and  implementation  of  the  SRSP  and,  in  particular,  the  dedicated 
call for the preparation of the Territorial Just Transition Plans (TJTPs). Both taking place 
at the end of June 2022, the focus group discussions provided a rich set of insights from 
12 Member States. Similarly to the interview programme, this information was used for 
validation  and  explanatory  purposes  and  led  to  the  collection  of  findings  from  the 
perspective of individual Member States.  

In close cooperation with DG REFORM, the design of stakeholder consultation was slightly 
adjusted during the ex-post evaluation to ensure that a solid evidence base is created in 
the  most  efficient  way,  without  overburdening  participants.  This  resulted  in  a  few 
changes  of  the  interview  programme  and  focus  group  discussions.  First,  we  only 
interviewed  the  representatives  of  those  national  coordinating  authorities  who  were  not 
involved  in  the  focus  group  discussions  and/or  represented  the  Member  State  that  had 
lower  numbers  of  requests  for  technical  support.  While  maintaining  the  geographical 
scope  of  qualitative  evidence,  this  decision  reduced  the  number  of  national-level 
interviews.  Second,  the  execution  of  validation  interviews  was  combined  with  the 
extensive feedback from the Inter-service Steering Group on the Final Report to ensure a 
reliable  verification  of  evaluation  results,  preliminary  conclusions  and  lessons  learned. 
Third, the initial selection criteria of participants of focus group discussions were modified 
to ensure the involvement of representatives from Member States that were not covered 
by  the  case  studies  and/or  less  represented  in  stakeholder  consultation  activities.  To 
increase  their  participation  rates,  it  was  decided  to  organise  a  single  session  per  focus 
group.             

As per the Better Regulation Guidelines, the evaluation refers both to the percentage and 
the  number  of  selections  of  a  particular  OPC  or  targeted  survey  response  out  of  total 
responses  to  a  particular  question  to  which  this  percentage  correspond.88  While 

87  Factual  Summary  Report,  summarising  results  of  the  Open  Public  Consultation  “Structural  reform  support 
https://ec.europa.eu/info/law/better-regulation/have-your-

programme 
say/initiatives/12790-Structural-reform-support-programme-final-evaluation/public-consultation_en 

evaluation“. 

final 

– 

88 For example, in the statement ‘According to the targeted survey of beneficiary authorities, as many as 30% 
(39 out of 131) of all evaluated projects were still ongoing during the ex post evaluation’ we refer to the 
results  of  the  targeted  survey  in  the  following  way:  131  (out  of  132,  excluding  “Do  not  know/cannot 

62 

 
 
 
presenting results of the quantitative consultation activities, N indicates the total number 
of responses to a particular OPC or targeted survey question. In addition, N refers to the 
sample of responses to the question of satisfaction and outcome questionnaires at hand. 

More  in-depth  methodological  information  and  results  of  the  stakeholder  consultation 
activities are presented in Annex 3: Synopsis of the consultation activities.  

Case studies and comparative analysis 

The  case  study  method  was  deployed  to  get  an  in-depth  analysis  of  how  technical 
support  was  provided  in  specific  EU  Member  States.  In  addition,  it  provided  an 
opportunity to better assess the medium- and long-term effects of SRSP-funded actions, 
the  causal  links  between  the  actions  and  results,  and  the  factors  that  have  driven  or 
hindered the achievement of the  expected results and impact. A total of 8 country case 
studies (including Ireland, Cyprus, Finland, Croatia, Italy, Latvia, Romania, Czechia) and 
1 horizontal case study of projects funded through the dedicated call for the preparation 
of  TJTPs  were  selected  for  the  case  study  programme  (see  Table  16  in  Annex  4).  The 
selection of countries for the case study programme integrated the feedback provided by 
DG  REFORM  while  ensuring  a  representative  sample,  well-balanced  in  terms  of 
geographical  distribution  and  the  thematic  fields  of  technical  support.  Taken  together, 
the nine case studies assessed the design and execution of 47 technical support projects.  

Each of the case studies followed the structure of the case study template, including the 
background information on the SRSP-funded projects, information on the reform context, 
a description of the design and execution of SRSP-supported projects (based on the five 
evaluation  criteria),  conclusions  and  a  standardized  assessment  of  each  project.  Taken 
together,  this  approach  ensured  comparability  of  the  case  studies  and  facilitated  the 
comparative  analysis.  The  latter  was  built  on  two  types  of  analysis  stemming  from  the 
case studies: (i) a structured assessment of each project by the country expert, based on 
pre-defined assessment scales; (ii) a qualitative content analysis of the text of each case 
study,  based  on  the  key  themes  as  per  evaluation  criteria.  As  a  result,  the  cross-case 
analysis  enabled  the  comparison  of  the  SRSP-funded  projects,  their  effectiveness, 
efficiency,  relevance,  coherence,  EU  added  value  and  lessons  learned,  as  well  as 
extracted  the  key  insights  and  generalisations  from  the  case  studies  (see  Annex  4: 
Comparative analysis of the case studies for a more in-depth methodology and the results 
of the comparative analysis).  

Reliability of findings and limitations 

Overall, the ex-post evaluation provides robust evaluation findings built on the 
high  quality  and  representative  data  collected  through  desk  research, 
stakeholder consultation activities and in-depth case studies. First, the evaluation 
was  supported  by  DG  REFORM,  providing  complete  and  comparable  internal  monitoring 
and  evaluation  data  on  the  evaluation  period.  Second,  the  consultation  activities  were 
highly representative in terms of stakeholder types, while the choice of case studies was 
well grounded in terms of geographical distribution and thematic policy areas.  Third, as 
indicated  in  the  evaluation  framework,  the  evaluation  criteria  were  broken  down  the 
general/operational  evaluations,  indicators  and  judgement  criteria.  The  majority  of 
consultation  activities  were  focused  on  gathering  data  related  to  all  five  evaluation 
criteria,  thus  allowing  for  data  triangulation  and  providing  the  suitable  mix  of  evidence 

answer” responses) beneficiary authorities responded to this targeted survey question, with 39 of them (or 
30%) claiming that their projects were still ongoing during the ex post evaluation.  

63 

 
 
 
for the evaluation questions at hand. Furthermore, different versions of the Report were 
reviewed by the DG REFORM team and the ISSG of the evaluation.   

However,  several  limitations  of  the  evaluation,  while  rather  marginal  or  mitigated 
during the study as explained below, should be mentioned. First, only 31% (40 out of 
129 responses) of beneficiary authorities indicated that their projects achieved all results 
by May-June 2022, according to the results of the targeted survey. The fact that some of 
the SRSP-funded projects were still ongoing at the time complicated the measurement of 
short-term,  medium-term  and  longer-term  results  during  the  evaluation  process.  In 
addition,  the  nature  of  such  project  outputs  as  recommendations,  roadmaps,  action 
plans,  handbooks  and  guidelines  determined  that  the  beneficiary  authorities  needed  to 
take follow-up actions after the provision of technical support.  

This situation was mitigated twofold. First, the study assessed the likelihood of achieving 
the intended results and impacts in the case of the still ongoing projects. Second, specific 
attention was paid to the projects that were already closed to find some “hard” evidence 
on  any  results  and  impacts.  Organising  ex  post  evaluation  somewhat  later  in  the 
implementation  process  could  allow  better  capturing  longer-term  impacts  of  the 
successor programme in the future.89  

Second,  some  drawbacks  of  the  quantitative  data  should  be  highlighted.  EU  Member 
States  that  had  the  lowest  numbers  of  requests  for  technical  support  and  consequently 
the  lowest  number  of  the  SRSP-funded  projects  were  less  represented  in  the  targeted 
survey and  OPC.  However, their experiences  were  relevant to grasp  possible drawbacks 
of  the  SRSP  that  discouraged  these  countries  and  their  beneficiary  authorities  from 
requesting technical support. Therefore, the study put additional efforts into ensuring the 
inclusion of representatives of Member States with lower participation rates in the focus 
group  discussions  or  interviews.  Also,  as  indicated  in  the  synopsis  report,  some  OPC 
questions received a large share of “do not know/cannot answer” responses and limited 
background  information  on  the  respondents.  Instead  of  treating  them  as  a  data 
limitation, the study used these responses to highlight issues that respondents were least 
aware of and cross-check them with the help of other sources of data.  

Third, while the evaluation was carried out at the programme level (by treating the SRSP 
as  the  main  unit  of  analysis),  the  analysis  of  the  monitoring  data,  stakeholder 
consultations  and  case  studies  focused  on  the  project  level.  The  complete  and 
comparable nature of the internal monitoring/evaluation data, relatively large samples of 
the targeted survey data and the in-depth analysis of 47 SRSP-funded projects under the 
case  study  programme  made  it  however  possible  to  draw  reliable  programme-level 
results  and  conclusions  on  the  basis  of  project-focused  information.  The  continuous 
triangulation  of  data  was  undertaken  to  ensure  that  the  specific  assessments  are 
applicable  beyond  the  individual  project  experience.  Therefore,  the  overall  findings  can 
be well generalised to the whole programme.  

Fourth,  the  thematic  field  of  sustainable  growth  and  business  environment  was 
represented in our case study programme through the horizontal case study on TJTP with 
17 projects implemented 17 Member States, thus well addressing the sub-field of climate 
actions.  However,  during  the  case  study  programme  we  did  not  analyse  in-depth  any 
projects  implemented  in  the  sub-field  of  business  environment  where  typical  SRSP-

89  According  to  the  SRSP  Regulation,  the  European  Commission  should  provide  an  independent  ex  post 
evaluation report by 31 December 2021. Planned start and finish dates of this evaluation were moved forward 
in the evaluation roadmap. The ex post evaluation was completed in the beginning of October 2022. 

64 

 
 
 
 
funded  projects  included  the  actions  to  reduce  market  barriers,  facilitate  foreign 
investment  and  trade,  develop  the  digital  economy,  stimulate  research  and  innovation, 
manage  public-private  partnerships  and  improve  the  governance  of  state-owned 
enterprises  (where  more  resistance  to  change  is  likely  to  occur  during  the  reform 
process).  Since  the  role  of  external  factors  is  usually  very  important  for  achieving  the 
intended results and impacts of the projects in the sub-field of business environment, the 
study  assessed  the  role  of  such  factors  (including  political  support  for  reforms)  by 
applying other evaluation methods (e.g., under all types of stakeholder consultations). 

65 

 
 
 
ANNEX III. EVALUATION MATRIX AND, WHERE RELEVANT, DETAILS ON ANSWERS TO THE EVALUATION QUESTIONS (BY CRITERION) 

1. Effectiveness (results) 

Effectiveness refers to the extent to which the SRSP has been successful in achieving its short- and medium-term results. In addition, the 
assessment  involves  the  underlying  mechanisms  of  change  and  the  external  conditions/factors  that  encouraged  or  hindered  the 
achievement  of  the  expected  results  of  the  SRSP.  The  assessment  of  the  SRSP  results  and  the  operationalisation  of  questions 
encompasses the individual, institutional and policy level results of the SRSP projects.  

Table 4. The operationalisation of the evaluation questions for assessing the effectiveness (results) of the SRSP 

STUDY/ OPERATIONAL QUESTION  

INDICATORS AND JUDGMENT CRITERIA 

DATA SOURCES 

to 

1.  To  what  extent 
have  SRSP  actions 
the 
contributed 
results  expected  and 
obtained  in  terms  of 
and 
design 
implementation 
of 
institutional, 
administrative 
and 
structural  reforms  in 
the  Member  States  in 
line  with  the  general 
and specific objectives 
of 
the  programme, 
specifically? 

in 

1.1.  To  what  extent 
actions 
have  SRSP 
to 
contributed 
national 
supporting 
the 
authorities 
design/implementation 
reforms 
of 
their 
according 
into 
priorities, 
account 
initial 
conditions and expected 
socioeconomic 
impacts? 

to 
taking 

their 

Quantitative: 

-  Perception of stakeholders on the extent to which the 
objectives  their  SRSP-funded  projects  corresponded 
to  the  key  reform  goals  of  beneficiary  countries 
(successful if above 70% assessed positively); 

-  %  of  technical  support  projects  that  produced  the 
expected  outcomes  after  the  projects’  completion 
(successful if above 70%; with the breakdown of data 
according  to  sectoral  categories,  types  of  specific 
objectives, etc.); 

-  Perception  of  different  stakeholders  on  the  extent  to 
the 
which  SRSP-funded  projects  contribute 
achievement  of  specific  changes  in  institutions  and 
sectoral  areas  (successful  if  more  than  two  thirds  of 
respondents assess that SRSP actions contributed to a 

to 

66 

 
 
 
 
 
 
 
 
 
 
 
 
STUDY/ OPERATIONAL QUESTION  

INDICATORS AND JUDGMENT CRITERIA 

DATA SOURCES 

high extent and to some extent). 

Qualitative: 

-  Examples of SRSP activities and outputs contributing 
to the achievement of specific results and examples of 
specific results that are related to the implementation 
of reforms that are national priorities. 

Quantitative: 

-  %  of  technical  support  projects  that  produced  the 
expected  outcomes  after  the  projects’  completion 
(successful if above 70%; with the breakdown of data 
according  to  sectoral  categories,  types  of  specific 
objectives, etc.); 

- 

-  %  share  of  projects  the  specific  outcome  of  which 
included adoption of (new) procedures and actions to 
enhance the implementation of reform;  
the  feedback  from  beneficiary  authorities,  support 
providers  and  other  stakeholders  on  the  extent  to 
which  the  project  results  improved  the  capacities  for 
reform/policy 
and 
implementation (successful if more than two thirds of 
respondents assess that SRSP actions contributed to a 
high extent and to some extent). 

development 

formulation, 

67 

capacity 

1.2.  To  what  extent 
have  SRSP 
actions 
to 
contributed 
supporting  the  national 
authorities in enhancing 
to 
their 
formulate,  develop  and 
implement 
reform 
policies  and  strategies 
in  pursuing  an 
and 
approach 
integrated 
consistency 
ensuring 
between 
and 
goals 
means  across  sectors? 
the 
To  what  extent 
SRSP  actions  increased 
the 
of 
capacity 
beneficiary  authorities 

Desk 
research/literature 
review: 

-  Analysis  of  the 
the 
from 
data 
internal  monitoring 
system (JIRA); 

-  Satisfaction  and 
outcome 

 
 
 
 
 
 
 
 
 
 
 
STUDY/ OPERATIONAL QUESTION  

INDICATORS AND JUDGMENT CRITERIA 

Qualitative: 

DATA SOURCES 
questionnaires 

to  formulate,  develop 
and  implement  reform 
policies and strategies? 

1.3.  To  what  extent 
actions 
have  SRSP 
to 
contributed 
supporting the efforts of 
national  authorities  to 
define  and  implement 
appropriate 
processes 
and  methodologies  by 
taking 
account 
into 
good  practices  of  and 
lessons learned by other 
countries  in  addressing 
similar situations? 

-  Examples/qualitative  evidence  on  adopted  initiatives 
that  increased/are  likely  to  increase  capacity  of 
beneficiary  authorities  to  formulate,  develop  and 
implement  reform  policies  and  strategies/pursue 
integrated approach. 

-  Analysis  of  the 
secondary literature 
articles, 
(studies, 
impact 
assessments). 

Quantitative: 

-  %  of  technical  support  projects  that  produced  the 
expected  outcomes  after  the  projects’  completion 
(successful if above 70%; with the breakdown of data 
according  to  sectoral  categories,  types  of  specific 
objectives, etc.); 

-  number  and  %  share  of  projects  that  resulted  in 
improved internal working procedures, methodologies 
and  processes  (by  taking  into  account  successful 
experiences  and  good  practices  from  other  EU 
Member States); 

-  perception  of  national  authorities,  support  providers 
and  other  stakeholders  on  the  extent  to  which 
and 
organisational 
methodologies  were  improved  (successful  if  more 
than  two  thirds  of  respondents  assess  that  SRSP 
actions  contributed  to  a  high  extent  and  to  some 

procedures 

processes, 

Stakeholder 
consultation: 

- Targeted survey; 

- OPC; 

- Focus groups; 

- Interviews. 

Case studies 

68 

 
 
 
 
 
 
STUDY/ OPERATIONAL QUESTION  

INDICATORS AND JUDGMENT CRITERIA 

DATA SOURCES 

extent). 

Qualitative: 

-  Examples/qualitative evidence on the cases where the 
SRSP-funded  projects  resulted  in  Member  States 
implementing 
and 
methodologies based on good practices of and lessons 
learned by other countries.  

appropriate 

processes 

1.4.  To  what  extent 
actions 
have  SRSP 
contributed  to  assisting 
the  national  authorities 
the 
enhancing 
in 
efficiency 
and 
effectiveness of human-
resource  management, 
inter 
by 
alia, 
strengthening 
professional  knowledge 
and  skills  and  setting 
out 
clear 
responsibilities? 

Quantitative: 

-  %  of  technical  support  projects  that  produced  the 
expected  outcomes  after  the  projects’  completion 
(successful if above 70%; with the breakdown of data 
according  to  sectoral  categories,  types  of  specific 
objectives, etc.); 

related 

-  Number  and  %  share  of  SRSP-funded  projects  the 
specific  objectives  of  which  address  support  to  more 
efficient and effective human resource management; 
-  %  share  of  projects  the  specific  objectives  of  which 
are 
to  organisational  change,  change 
management, improved human resource management; 
-  %  share  of  beneficiary  authorities  that  agree  that 
SRSP-funded projects improved their human resource 
management  (successful  if  above  70%;  with  the 
breakdown  of  data  according  to  sectoral  categories, 

69 

 
 
STUDY/ OPERATIONAL QUESTION  

INDICATORS AND JUDGMENT CRITERIA 

DATA SOURCES 

types of specific objectives, etc.). 

Qualitative: 

-  Examples/qualitative  evidence  on  the  cases,  when 
SRSP-funded 
adopting 
measures/initiatives  enhancing  the  efficiency  and 
in 
effectiveness  of  human-resource  management 
national authorities. 

projects 

resulted 

in 

driving 

2.  What  factors  have 
been 
or 
the 
hindering 
achievement  of  short 
term 
and  medium 
results 
SRSP 
of 
funded  actions  and 
why? 

2.1.  What  were  the  key 
success  factors  driving 
the 
of 
achievement 
short  and  medium-term 
results  of  SRSP  funded 
actions and why? 

Qualitative: 

-  Mechanisms of change (e.g. timing and sequencing of 
reforms; strong leadership and ownership of reforms; 
effective  consultation  and  communication)  leading 
from  SRSP  activities  and  outputs  to  short-term  and 
mid-term results; 

-  External  conditions  (e.g.  initial  economic,  social  and 
institutional  conditions; 
the  election  cycle  and 
political  support  for  reforms;  the  scope  of  reforms) 
favourably  influencing  the  achievement  of  medium 
and  longer-term  results  (favourable  if  more  than  two 
thirds  of  respondents  assess  that  these  conditions 
materialised to a high extent and to some extent). 

Stakeholder 
consultation: 

- Targeted survey; 

- OPC; 

- Focus groups; 

- Interviews. 

Case studies 

70 

 
 
 
 
STUDY/ OPERATIONAL QUESTION  

INDICATORS AND JUDGMENT CRITERIA 

DATA SOURCES 

2.2.  What  were 
the 
obstacles 
and 
challenges 
hindering 
of 
achievement 
the 
short- and medium-term 
results  of  SRSP  funded 
actions and why? 

Qualitative: 

-  Lack  of  mechanisms  of  change  (e.g.  timing  and 
leadership  and 
reforms;  strong 
sequencing  of 
ownership  of  reforms;  effective  consultation  and 
communication)  leading  from  SRSP  activities  and 
outputs to short-term and mid-term results; 

-  External  conditions  (e.g.  initial  economic,  social  and 
institutional  conditions; 
the  election  cycle  and 
political  support  for  reforms;  the  scope  of  reforms) 
unfavourably influencing the achievement of medium 
and  longer-term  results  (unfavourable  if  more  than 
two thirds of respondents assess that these conditions 
materialised to a limited extent or not at all). 

3.  To  what  extent  did 
SRSP  funded  actions 
have  ownership  and 
commitment  from  the 
States 
Member 
beneficiary 
to 
authorities 
the 
implement 
outputs and to follow-
up  on  the  relevant 
for 
reform  process 

3.1.  To  what  extent  did 
the  Member 
States 
beneficiary  authorities 
have the ownership and 
commitment 
to 
implement  the  outputs 
funded 
of 
actions? 

SRSP 

Quantitative: 

-  Number  and  %  share  of  SRSP-funded  projects  the 
outcomes of which included adoption of a strategy, a 
new  law  /act  or  modification  of  an  existing  one, 
adoption of (new) procedures and actions to enhance 
the  implementation  of  reforms,  as  well  as  the 
achievement of other expected results; 

-  Perception of different  groups of stakeholders on the 
extent  to  which  national  ownership  of  reforms 
affected  the  delivery  of  the  outputs  and  results  of 

Desk 
research/literature 
review: 

-  Satisfaction  and 
outcome 
questionnaires 

71 

 
 
 
 
STUDY/ OPERATIONAL QUESTION  

INDICATORS AND JUDGMENT CRITERIA 

which  support  under 
the  programme  was 
requested? 

SRSP-funded  projects  (successful  if  more  than  two 
thirds  of  respondents  assess  that  it  had/will  have  a 
moderate positive effect). 

Quantitative: 

3.2.  To  what  extent  did 
the  Member 
States 
beneficiary  authorities 
have the ownership and 
commitment  to  follow-
relevant 
the 
up  on 
for 
process 
reform 
which support under the 
programme 
was 
requested? 

-  %  share  of  stakeholders  reporting  follow-up  actions 
taken  in  their  country  after  the  project  end  and 
building  upon 
the  project  outputs  and  results 
(successful  if  more  than  one  third  of  respondents 
assess that follow-up actions were taken). 

Qualitative: 

-  Number  of  projects  assessed  under  the  case  studies 
where  all  or  some  of  project  outputs  have  been 
achieved  (successful  if  subsequent  reform  progress 
has been made in more than 50% of the cases). 

DATA SOURCES 
Stakeholder 
consultation: 

- Targeted survey; 

- Focus groups; 

- Interviews. 

Case studies 

Source: independent evaluation study. 

72 

 
 
 
 
 
 
 
 
 
 
 
 
2. Impact  

Evaluation  of  the  impact  of  the  SRSP  focuses  on  the  assessment  of  longer-term  outcomes  of  programme  activities,  including  the  key  two  types:  1) 
structural changes in the institutional and administrative capacity of Member States’ authorities to prepare and implement reforms, as well as to apply EU 
law in an effective way; and 2) achievement of relevant longer-term growth-sustaining reforms in EU Member States. The assessment focused not only 
on the impacts and the extent to which these impacts occurred as a result of SRSP, but also on the mechanisms of influence linking the SRSP outputs and 
results to the structural changes in Member States, as well as the external conditions that either facilitated the achievement of long-term SRSP impacts 
(success factors) or hindered their achievement (external obstacles/challenges).  

Table 5. The operationalisation of the evaluation questions for assessing the impacts of the SRSP 

STUDY/OPERATIONAL QUESTION 

INDICATORS AND JUDGMENT CRITERIA 

Quantitative: 

in 

the 

4. To what extent and in what ways has 
the  SRSP  contributed  to  structural 
changes 
institutional  and 
administrative  capacity  of  Member 
to 
States’ 
prepare  and  implement  reforms  and/or 
adequately apply EU law? 

beneficiary 

authorities 

in 

the 

to  structural  changes 

-  Perception  of  different  types  of  stakeholders  on  the  extent  to  which  the 
SRSP  contributed 
institutional  and 
administrative  capacity  of  Member  States’  beneficiary  authorities  to 
prepare  and  implement  reforms  and/or  adequately  apply  EU  law 
(successful  if  more  than  two  thirds  of  respondents  assess  that  SRSP 
actions contributed to a high extent and to some extent, as well as at least 
to  the  level  of  achievements  reported  in  the  mid-term  evaluation  of  the 
SRSP,  e.g.  42.1%  in  the  case  of  application  and  implementation  of  EU 
law). 

Qualitative: 

-  List of concrete structural changes in the institutional and administrative 

73 

DATA 
SOURCES 

Desk 
research/litera
ture review: 

-  Satisfaction 
outcome 
and 
questionnaires; 

Stakeholder 

 
 
 
 
 
capacity  of  Member  States’  beneficiary  authorities  that  at  least  partly 
resulted/will  likely  result  due  to  SRSP-funded  projects  (e.g.  public 
administration reforms); 

-  Number  of  projects  assessed  under  the  case  studies  where  the  expected 
changes  due  to  the  implementation  of  SRSP-funded  projects  have 
occurred  totally,  moderately  or  will  probably  occur  before  the  end  of 
2022 (successful if the expected changes occurred/will probably occur in 
more than 50% of the cases). 

Quantitative: 

-  Perception  of  different  types  of  stakeholders  on  the  extent  to  which  the 
SRSP  contributed  to  the  realisation  of  the  relevant  longer-term  growth-
sustaining  reforms  (successful  if  more  than  two  thirds  of  respondents 
assess that SRSP actions contributed to a high extent and to some extent). 

Qualitative: 

consultation: 

Targeted 

- 
survey; 

- OPC; 

- Focus groups; 

- Interviews. 

Case studies 

Desk 
research/litera
ture review: 

-  Satisfaction 
and 
outcome 
questionnaires; 

-  Examples of concrete longer-term growth-sustaining reforms in Member 
States  that  at  least  partly  resulted/will  likely  result  due  to  the  SRSP-
funded  projects  (e.g.  labour  market  reforms,  the  liberalisation  and 
deregulation of product and services markets); 

-  Number  of  projects  assessed  under  the  case  studies  where  the  expected 
changes  due  to  the  implementation  of  SRSP-funded  projects  have 
occurred  totally,  moderately  or  will  probably  occur  before  the  end  of 
2022 (successful if the expected changes occurred/will probably occur in 

Stakeholder 
consultation: 

Targeted 

- 
survey; 

- OPC; 

- Focus groups; 

74 

5. How did the technical support funded 
under  the  SRSP  contribute  to  the 
Member  States  achievement  of  the 
relevant  longer-term  growth-sustaining 
reforms as indicated in the Cooperation 
and Support Plans? 

 
 
more than 50% of the cases). 

Qualitative: 

6.  What 
factors 
have  been  driving 
or  hindering 
the 
achievement  of  the 
expected  impact  of 
SRSP 
funded 
projects and why? 

6.1.  What  were  the 
key  success  factors 
the 
driving 
of 
achievement 
impacts 
long-term 
funded 
SRSP 
actions and why? 

6.2.  What  were  the 
obstacles 
and 
challenges 
hindering 
achievement 
long-term 
of  SRSP 
actions and why? 

the 
of 
impacts 
funded 

-  Mechanisms  of  change  (e.g.  timing  and  sequencing  of  reforms;  strong 
leadership  and  ownership  of  reforms;  effective  consultation  and 
communication)  leading  from  SRSP  activities  and  outputs  to  long-term 
impacts; 

-  External  conditions  (e.g.  initial  economic,  social  and  institutional 
conditions; the election cycle and political support for reforms; the scope 
of reforms) favourably influencing the achievement of long-term impacts 
(favourable  if  more  than  two  thirds  of  respondents  assess  that  these 
conditions materialised to a high extent and to some extent). 

Qualitative: 

-  Lack  of  mechanisms  of  change  (e.g.  timing  and  sequencing  of  reforms; 
strong  leadership  and  ownership  of  reforms;  effective  consultation  and 
communication)  leading  from  SRSP  activities  and  outputs  to  long-term 
impacts; 

-  External  conditions  (e.g.  initial  economic,  social  and  institutional 
conditions; the election cycle and political support for reforms; the scope 
of  reforms)  unfavourably  influencing  the  achievement  of  long-term 
impacts (unfavourable if more than two thirds of respondents assess that 
these conditions materialised to a limited extent or not at all). 

Source: independent evaluation study. 

75 

- Interviews. 

Case studies 

Stakeholder 
consultation: 

Targeted 

- 
survey; 

- OPC; 

- Focus groups; 

- Interviews. 

Case studies 

 
 
 
3. Efficiency  

Evaluation of efficiency refers to the extent to which the desired effects have been achieved at a reasonable cost or as the optimal balance between the 
resources employed and the results achieved. The concept of efficiency also concerns the adequacy of management arrangements for the implementation 
of the programme (e.g., institutional set-up, human and financial resources, processes and procedures, tools), the administrative and regulatory burden 
imposed by the intervention and the potential for simplifications of the processes. Among other evaluation and analysis methods and tools, the ex-post 
evaluation employed cost-effectiveness analysis for answering the efficiency questions.  

Table 6. The operationalisation of the evaluation questions for assessing the efficiency of the SRSP 

STUDY/ 

OPERATIONAL QUESTION 

7.  To  what  extent  was  the  scale  of  the 
SRSP funded actions proportionate to the 
request  for  support  submitted  by  the 
Member  States  and  to  the  expected 
benefits from the reform projects? 

INDICATORS AND JUDGMENT CRITERIA 

DATA SOURCES 

Quantitative: 

-  Proportion  of  the  number  of  technical  support  requests  to  the 
number of selected technical support requests, evolution over time;  
-  Perception  of  stakeholders  on  the  extent  to  budget/financial 
contribution  of  the  SRSP  was  proportionate  to  the  needs  of 
technical support (successful if more than two thirds of respondents 
assess positively). 

Qualitative: 

-  Perception of beneficiaries and other stakeholders on the efficiency 
of different means of project implementation (such as direct grants 
to  international  organisations  and  other  reform  partners,  public 
procurement contracts, TAIEX, actions implemented under indirect 
management,  etc.);  identification  of  aspects  which  could  be 
improved (successful if more than two thirds of respondents assess 

Desk 
research/literature 
review: 

-  SRSP  Regulation, 
work 
annual 
programmes 
and 
annexes; 

-  Annual  monitoring 
reports 

Stakeholder 
consultation: 

- Targeted survey; 

76 

 
STUDY/ 

OPERATIONAL QUESTION 

INDICATORS AND JUDGMENT CRITERIA 

DATA SOURCES 

8.  How  cost-effective  were  the  actions 
funded under the SRSP? 

positively). 

Quantitative: 

-  The  ratio  of  total  estimated  management  costs  (administrative 
budget)  to  the  overall  size  of  the  managed  programme  funds 
(operational  budget),  evolution  over  time,  benchmarking  to  other 
centralised  EU  programmes  taking  into  account  the  inherent 
differences of the programmes; 

-  Financial  contribution  per  technical  support  reform  project  (within 
minimum  and  maximum  brackets  based  on  the  available  data), 
comparison  across  different  means  of  implementation,  evolution 
over  time,  comparison  with  similar  programmes  (to  the  extent 
possible taking into account the inherent differences of the support 
provided). 

Qualitative: 

-  Perception  of  stakeholders  on  possible  improvements  in  cost-

effectiveness of SRSP actions. 

77 

- Focus groups; 

- Interviews. 

Case studies 

Desk 
research/literature 
review: 

-  SRSP  Regulation, 
work 
annual 
and 
programmes 
annexes; 

-  Annual  monitoring 
reports 

Stakeholder 
consultation: 

- Targeted survey; 

- Focus groups; 

 
 
 
STUDY/ 

OPERATIONAL QUESTION 

INDICATORS AND JUDGMENT CRITERIA 

DATA SOURCES 

9.  How  efficient  is  the  SRSP  as  regards 
from  request 
the  process  duration 
submission  by  Member  States  to  the 
deployment  of  the  technical  support  on 
the  ground,  and 
to  completion  of 
technical support? 

- Interviews. 

Case studies 

Desk 
research/literature 
review: 

-  SRSP  Regulation, 
work 
annual 
programmes 
and 
annexes; 

-  Annual  monitoring 
reports 

Stakeholder 
consultation: 

- Targeted survey; 

Quantitative: 

-  Average duration of the evaluation and selection procedures of the 
technical  support  projects  (time  period  from  the  deadline  for 
submitting  the  technical  support  requests  to  the  approval  of 
respective financing decisions and the conclusion of the cooperation 
and support plans), evolution over time; 

-  The  timeliness  of  the  selection  of  requests  for  support  under  the 
SRSP  in  days,  evolution  in  time,  comparison  with  other  similar 
programmes. 

-  Perception  of  the  stakeholders  on  whether  project  implementation 
duration  was  sufficient  to  implement  corresponding  activities 
(successful 
thirds  of  respondents  assess 
positively); 

if  more 

than 

two 

-  Perception  of  the  stakeholders  on  the  timeliness  and  efficiency  of 
administrative  procedures  (evaluation  and  selection  of  requests; 
technical 
preparation  for 
support projects) (successful if more than two thirds of respondents 

implementation  of 

implementation; 

78 

 
 
 
 
 
STUDY/ 

OPERATIONAL QUESTION 

INDICATORS AND JUDGMENT CRITERIA 

DATA SOURCES 

assess positively). 

Qualitative: 

-  Perceptions of the stakeholders on the possible improvements in the 
the  SRSP  administrative  procedures 
for 

timeliness/efficiency  of 
(evaluation 
preparation 
implementation; implementation of technical support projects). 

selection 

requests; 

and 

of 

10.  How  time-efficient  was  the  budget 
execution 
to 
payments? 

commitments 

from 

Quantitative: 

-  The extent to which the means of delivery are marked by delays; 
-  Timeliness of execution of SRSP payment appropriations; 
-  The  extent  to  which  the  technical  support  providers  received 

payments in due time, evolution over time. 

Qualitative: 

-  Perception  of  stakeholders  on  possible  improvements  in  SRSP 

budget execution time-efficiency.  

79 

- Focus groups; 

- Interviews. 

Case studies 

Desk 
research/literature 
review: 

-  SRSP  Regulation, 
work 
annual 
and 
programmes 
annexes; 

-  Annual  monitoring 
reports; 

Stakeholder 
consultation: 

- Targeted survey; 

- Focus groups; 

 
 
 
STUDY/ 

OPERATIONAL QUESTION 

INDICATORS AND JUDGMENT CRITERIA 

DATA SOURCES 

11.  To  what 
extent was the 
governance  of 
SRSP 
the 
efficient  and 
how  efficient 
the 
was 
cooperation 
with 
Commission 
Services? 

other 

11.1.  How  efficient  was 
the 
and 
governance 
management  structure  of 
the SRSP?  

- Interviews. 

Case studies 

Desk 
research/literature 
review: 

-  SRSP  Regulation, 
work 
annual 
programmes 
and 
annexes; 

- 
SRSP  Actions 
relevant  documents 
in  accordance  with 
legal provisions; 

-  Other 
internal 
administrative  data 
(on staff numbers); 

Stakeholder 
consultation: 

Quantitative: 

-  Perception of the beneficiaries and other stakeholders on the clarity, 
transparency  and  user-friendliness  of  the  programme  management 
procedures  and  tools  through-out  project  cycle  (successful  if  more 
than two thirds of respondents assess positively); 
-  Average  number  of  DG  REFORM  staff 
management of SRSP-funded projects by year; 

involved 

the 

in 

-  Perception  of  stakeholders  on 

the 
administrative  burden  related  to  SRSP  administrative  procedures 
thirds  of  respondents  assess 
(successful 
positively). 

the  proportionality  of 

if  more 

than 

two 

Qualitative: 

-  The  extent  to  which  the  SRSP  governance  and  implementation 
functions are well-defined, detailed enough and clearly attributed to 
the involved management bodies; 

-  Perceptions  of  stakeholders  on  possible  improvements  to  the 
programme management/efficient administration of the SRSP. 

80 

 
STUDY/ 

OPERATIONAL QUESTION 

11.2.  How  efficient  was 
cooperation between SRSP 
with  other  Commission 
Services? 

INDICATORS AND JUDGMENT CRITERIA 

DATA SOURCES 

Quantitative:  

-  Perception  of  the  stakeholders  on  the  efficiency  of  cooperation 
between DG REFORM and other Commission Services (successful 
if more than two thirds of respondents assess positively). 

- Targeted survey; 

- Interviews. 

Qualitative: 

-  Clarity  and  sufficiency  of  the  formal  rules  establishing  the 
cooperation links  and practices between the SRSP governance and 
other Commission Services. 

-  Presence  of  information  flows,  co-operation  links  and  practices 
between the SRSP governance and other Commission Services. 

Source: independent evaluation study. 

81 

 
 
 
 
 
 
4. Relevance  

Evaluation of relevance concerns the relationship between the needs and problems in society and the objectives of the intervention and hence touches on 
aspects of project design.  

Table 7. The operationalisation of the evaluation questions for assessing the relevance of the SRSP 

STUDY QUESTION 

INDICATORS AND JUDGMENT CRITERIA 

DATA SOURCES 

the  SRSP 
the 

12.  To  what  extent  were  actions  funded 
under 
for 
enhancing 
and 
institutional capacity of Member States to 
design and implement the reforms needed 
to tackle the challenges faced? 

appropriate 
administrative 

Quantitative: 

-  Perception of stakeholders on the extent to which the objectives of 
SRSP-funded  projects  corresponded  to  the  key  reform  goals  of 
Member  State  (successful  if  more  than  two  thirds  of  respondents 
assess SRSP positively);  

-  Perception of stakeholders on whether and to what extent the SRSP 
design/structure  is  appropriate  in  addressing  key  challenges  of 
Member States to design and implement reforms (successful if more 
than two thirds of respondents assess SRSP positively). 

Desk 
research/literature 
review: 

-  Satisfaction  and 
outcome 
questionnaires 

Stakeholder 
consultation: 

Qualitative: 

-  Extent  to  which  the  specific  objectives  of  SRSP  correspond  to  the 
key current challenges in Member States/beneficiary institutions; 
-  Number  of  projects  assessed  under  the  case  studies  where  the 
support provided by SRSP corresponded to the key challenges faced 
by Member State (successful if assessed positively in more than 50 
% of the cases). 

- Targeted survey; 

- OPC; 

- Interviews. 

82 

 
 
 
 
STUDY QUESTION 

INDICATORS AND JUDGMENT CRITERIA 

DATA SOURCES 

the  SRSP  address 

13. To what extent did the actions funded 
under 
the  needs 
the  Member  States 
expressed  by 
beneficiary authorities? 

Quantitative: 

-  Extent to which stakeholders agree that the Member States still need 
the  support  from  SRSP  (successful  if  more  than  two  thirds  of 
respondents  assess  that  SRSP  actions  contributed  to  a  high  extent 
and to some extent); 

-  Extent to which beneficiary authorities agree that SRSP contributes 
to meeting their principal reform needs (successful if more than two 
thirds of respondents assess that SRSP actions contributed to a high 
extent and to some extent); 

-  The  number  of  total  infringement  cases  in  the  EU  (with  an 
intervention  into  the  application  and  implementation  of  Union 
legislation  still  necessary  if  this  number  is  not  well  below  1,347 
cases registered at the end of 2014). 

Qualitative: 

-  Number  of  projects  assessed  under  the  case  studies  where  the 
support provided by SRSP met the needs of beneficiary authorities 
(successful  if  the  needs  were  addressed  to  a  large  or  moderate 
extent in at least 70 % of the cases). 

Case studies 

Stakeholder 
consultation: 

- Targeted survey; 

- Focus groups; 

- Interviews. 

Case studies 

Source: independent evaluation study. 

83 

 
 
 
 
5. Coherence  

Evaluation of coherence refers to synergies or inconsistencies within the different elements of an EU intervention, or between different interventions (i.e., 
how well or not have different actions worked together). The ex post evaluation covered both the extent to which the actions funded under the SRSP were 
coherent  with  the  objectives  as  set  out  in  the  SRSP  Regulation  (internal  coherence)  and  the  extent  to  which  the  SRSP  was  coherent  with  other 
interventions  at  regional,  national  and  EU  level  (external  coherence).  For  this  purpose,  the  evaluation  relied  on  the  list  of  other  EU  programmes  and 
instruments that could be combined with the SRSP to produce synergy effects during the implementation of specific reforms on the ground.  

Table 8. The operationalisation of the evaluation questions for assessing the coherence of the SRSP 

STUDY QUESTION 

INDICATORS AND JUDGMENT CRITERIA 

the  SRSP  coherent  with 

14.  To  what  extent  were  actions  funded 
under 
the 
objectives  as  set  out 
the  SRSP 
Regulation? 

in 

Qualitative: 

-  Complementarities,  gaps,  inconsistencies  and  overlaps  between 

different specific objectives and actions of the SRSP; 

-  Perception  of  stakeholders  on  internal  consistency  between  the 
objectives  and  actions  of  SRSP/potential  improvements  in  the 
design/structure of the programme to better achieve its goals. 

84 

METHODS/DATA 
SOURCES 

Desk 
research/literature 
review: 

-  SRSP  Regulation, 
work 
annual 
and 
programmes 
annexes; 

- Internal monitoring 
system 
SRP 
(JIRA); 

of 

Stakeholder 
consultation: 

 
 
STUDY QUESTION 

INDICATORS AND JUDGMENT CRITERIA 

Quantitative: 

coherent 

15.  To  what  extent  was  the  SRSP 
other 
externally 
interventions at regional, national and EU 
level, such as technical assistance through 
structural and investment funds? 

with 

-  Perception  of  stakeholders  on 

the  extent 

to  which  SRSP 
complements  (by  funding  different  aspects  of  similar  activities, 
targeting  different  groups,  etc.)  other  EU/national/regional  level 
initiatives  (ESF,  ERDF  and  others)  (successful  if  more  than  two 
thirds of respondents assess that SRSP actions contributed to a high 
extent and to some extent). 

Qualitative: 

-  Overlaps/duplications, complementarities and synergies (in terms of 
objectives,  target  groups,  intervention  areas,  expected  impacts) 
between SRSP and other national/regional interventions in Member 
States. 

85 

METHODS/DATA 
SOURCES 

- Interviews. 

Stakeholder 
consultation: 

- Targeted survey; 

- OPC; 

- Interviews; 

Case studies 

Desk 
research/literature 
review: 

-  SRSP  Regulation, 
work 
annual 
and 
programmes 
annexes 

 
 
 
 
STUDY QUESTION 

INDICATORS AND JUDGMENT CRITERIA 

METHODS/DATA 
SOURCES 

Quantitative: 

-  % share of SRSP-funded projects answering to the country specific 
recommendations  issued  in  the  context  of  the  European  Semester 
(successful if more than two thirds of respondents assess that SRSP 
actions contributed to a high extent and to some extent); 

-  Perception  of  stakeholders  on  the  extent  to  which  SRSP-funded 
projects  contributed  to  the  implementation  of  the  EU  priorities  for 
2019-2024 (successful if more than two thirds of respondents assess 
the SRSP positively). 

Stakeholder 
consultation: 

- Targeted survey; 

- Interviews. 

Case studies 

Qualitative: 

-  Strength  of  the  links/correspondence  between  the  specific/general 
objectives  of  the  SRSP  and  the  key  EU  strategic  priorities  (e.g. 
European Union priorities for 2019-2024). 

16.  To  what  extent  has  the  SRSP  been 
consistent  with  and  integrated  in  the 
European  Semester  (e.g.  country-specific 
recommendations) and EU priorities? 

Source: independent evaluation study. 

86 

 
 
 
 
 
 
6. EU added value  

Evaluation of the EU added value looked for changes which it can reasonably be argued are due to the EU intervention over and above what could have 
been  expected  from  actions  by  the  Member  States  either  alone  or  cooperating  together  without  an  EU  dimension.  The  evaluation  of  EU  added  value 
brought together the findings of the other criteria, presenting the arguments on causality and drawing conclusions, based on the evidence to hand, about 
the performance of the EU intervention and whether it was still justified.  

Table 9. The operationalisation of the evaluation questions for assessing the EU added value of the SRSP 

STUDY/OPERATIONAL QUESTIONS 

INDICATORS/JUDGMENT CRITERIA 

DATA SOURCES 

17.  What  has  been  the  added  value 
resulting  from  the  SRSP  compared  to 
what  could  reasonably  have  been  possible 
for  Member  States  acting  at  a  local, 
regional  or  national  level  without  EU 
support? 

Quantitative: 

-  % share of national  authorities and other stakeholders indicating 
that  national  authorities  could  have  received  similar  technical 
support/implemented  activities  at  national/regional/local  level 
(successful  if  more  than  two  thirds  of  respondents  assess  SRSP 
positively). 

Stakeholder 
consultation: 

- Targeted survey; 

Qualitative: 

-  Number and share of projects assessed in case studies, where the 
EU technical support provided by the SRSP was indispensable for 
the achievement of project  benefit (successful if  more than 50% 
of the projects); 

-  Reasons  why  the  results  of  the  SRSP-funded  projects  could  not 

have been achieved at national/regional level. 

- OPC; 

- Interviews. 

Case studies 

87 

 
 
 
Quantitative: 

18.1.  To  what  extent  has  the 
SRSP  produced  cross-border 
or  Union-wide  impacts  in  the 
areas of intervention? 

-  %  share  of  SRSP-funded  projects  related  to  cross-border/union 
wide impacts (successful if assessed positively in more than 50% 
of the cases); 

-  % share and number of projects designed based on the technical 
support  requests  of  at  least  two  neighbouring  Member  States 
(successful if assessed positively in more than 50% of the cases). 

18. To what 
extent  did 
the 
expected 
EU  added 
value  of  the 
SRSP 
materialise? 

to 

18.2.  To  what  extent  has  the 
SRSP  contributed 
the 
implementation  of  Union  law 
and  policies,  including  the 
promotion 
European 
of 
values, including solidarity? 

18.3.  To  what  extent  has  the 
SRSP  contributed  to  sharing 
among 
practices 
good 
Member  States,  also  with  a 

Qualitative: 

-  Qualitative  evidence/specific  examples  of  SRSP  cross-border  or 

Union-wide impacts. 

Quantitative: 

-  %  share  of  the  projects  that  contributed  to  the  consistent  and 

coherent implementation of the Union law and policies. 

Qualitative: 

-  Qualitative 

evidence/specific 

cases  SRSP 
contributed  to  the  implementation  of  Union  law  and  policies  in 
Member States. 

examples  of 

Quantitative: 

-  %  share  of  the  projects  that  contributed  to  the  sharing  of  good 

practices; 

88 

Desk 
research/literature 
review: 

-  Data  from 
the 
internal  monitoring 
system  of  SRSP 
(JIRA); 

Stakeholder 
consultation: 

- OPC; 

- Interviews. 

Case studies 

 
 
 
 
 
 
to 

the  need 

increasing 
the 

the 
view 
reform 
of 
visibility 
better 
programmes, 
for 
identifying 
possible 
accompanying 
measures  and/or  sequencing 
of  reforms,  and  to  building  a 
Union-wide 
and 
network of expertise? 

platform 

between 

18.4.  To  what  extent  is  the 
Programme promoting mutual 
beneficiary 
trust 
Member  States 
the 
Commission  and  cooperation 
among Member States? 

and 

-  %  share  of  beneficiary  authorities  that  used  the  good  practices 
and  lessons  learnt  from  the  specific  technical  support  project(s) 
after their support was finished. 

Qualitative: 

-  Qualitative 

evidence/specific 

cases  SRSP 
contributed  to  the  sharing  of  good  practices  between  Member 
States/outputs based on sharing of good practices. 

examples  of 

Quantitative: 

-  %  share  of  the  projects  that  promoted  mutual  trust  between 

beneficiary Member States. 

Qualitative: 

-  Qualitative 

evidence/specific 

cases  SRSP 
contributed promoting mutual trust between beneficiary Member 
States  and  the  Commission,  enhanced  cooperation  between 
Member States. 

examples  of 

Source: independent evaluation study. 

89 

 
 
ANNEX IV. OVERVIEW OF BENEFITS AND COSTS AND, WHERE RELEVANT, TABLE ON SIMPLIFICATION AND BURDEN REDUCTION.  

90 

 
 
TABLE 10. Overview of costs and benefits identified in the evaluation90 

Citizens/Consumers  

Businesses 

Administrations 

[Other…] _ specify 

Quantita
tive  

Provide 
the 
monetar
y value 

the 

Mark 
cost/benefit, 
separate line: 

type 

of 
each  on  a 

Costs: 

compliance 

Direct 
(adjustment 
administrative 
regulatory charges) 

Type: 
Choose 
one-off  or 
recurrent 

costs 
costs, 
costs, 

(costs 
Enforcement  costs: 
associated  with 
activities 
linked  to  the  implementation 

Comment  Quantitative  

Comment  Quantitativ

Comment   Quantitativ

Comment 

e 

e 

[Cost or Benefit description]: 

Where 
no 
quantificatio
n 
is 
possible, 
please 
provide 
or 
ranges 
explain 
the 
reasons why 

the 

Provide 
monetary 
value 

Provide the 
monetary 
value 

Where 
no 
quantification 
is 
possible, 
please  provide 
or 
ranges 
the 
explain 
reasons why 

Provide 
the 
monetary 
value 

Where 
no 
quantificatio
n is possible, 
please 
provide 
or 
ranges 
explain 
the 
reasons why 

is 

no 

Where 
quantificati
on 
possible, 
please 
provide 
or 
ranges 
explain 
the 
reasons why 

90 Where there is a prior impact assessment, the table should contain as a minimum the costs/benefits identified in the IA with the information gathered on the actual cost/benefit. As available, the table 
should  include  the  monetisation  (€)  of  the  costs/benefits  based  on  any  quantitative  translation  of  the  data  (time  taken,  person  days,  number  of  records/equipment/staff  etc.  affected  or  involved 
represented in monetary value – see Standard cost model, for example). For all information presented, it should be included in the comments section whether it relates to all Member States or is drawn 
from a subset. An indication of the robustness of the data should be provided in Annex II on Methodology and analytical models used. 

91 

 
 
             
 
 
of  an 
such  as 
initiative 
monitoring,  inspections  and 
adjudication/litigation) 

Indirect 
(indirect 
costs 
compliance  costs  or  other 
indirect 
as 
costs 
transaction costs) 

such 

Benefits: 

Direct  benefits 
(such  as 
improved well being: changes 
levels,  safety, 
in  pollution 
health,  employment;  market 
efficiency) 

economic 

Indirect  benefits  (such  as 
benefits, 
wider 
macroeconomic 
benefits, 
social impacts,  environmental 
impacts)  

Source: independent evaluation study. 

92 

 
 
 
TABLE 2: Simplification and burden reduction (savings already achieved)  

Report any simplification, burden reduction and cost savings achieved already by the intervention evaluated, including the points of comparison/ where 
available (e.g. REFIT savings predicted in the IA or other sources).  

Citizens/Consumers/Workers 

Businesses 

Administrations 

[Other…] _ specify 

Quantitative   Comment 

Quantitative   Comment 

Quantitativ
e 

Comment  

Quantitativ
e 

Comment 

Title91  [Select  among:  (i)  direct  compliance  cost  savings  (for  example  adjustment  cost  savings,  administrative  cost  savings,  savings  from  regulatory 
charges);  (ii)  enforcement  cost  savings  (for  example  cost  savings  associated  with  activities  linked  to  the  implementation  of  an  initiative  such  as 
monitoring,  inspections  and  adjudication/litigation);  (iii)  indirect  cost  savings  (if  possible  -  for  example  indirect  compliance  cost  savings  or  other 
indirect cost savings such as transaction cost savings).  

Type:  One-off 
(select) 

/  recurrent 

the 

Provide 
estimated 
monetary  and 
quantitative 
value  

(point value or 
range) 

no 
If 
monetisation/quan
tification 
is 
please 
possible, 
explain  here 
the 
reasons. 
Qualitative 
analysis 
simplification 
benefits  provides 
important 
information 
and 
should  be  inserted 

on 

91  Each simplification/saving should be included on a separate line.  

93 

 
 
        
 
 
 
 
 
 
 
 
here.  

PART II: II Potential simplification and burden reduction (savings) 

Identify  further  potential  simplification  and  savings  that  could  be  achieved  with  a  view  to  make  the  initiative  more  effective  and  efficient  without 
prejudice to its policy objectives92. 

Citizens/Consumers/Workers 

Businesses 

Administrations 

[Other…] _ specify 

Quantitative  

Comment 

Quantitative  

Comment 

Quantitati
ve 

Comment   Quantitati

Comment 

ve 

Description:… 

Type:  One-off 
(select) 

/  recurrent 

Source: independent evaluation study. 

92  This assessment is without prejudice to a possible future Impact Assessment. 

94 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
ANNEX V. STAKEHOLDER CONSULTATION – SYNOPSIS REPORT 

The  synopsis  report  summarises  the  results  of  all  consultation  activities  carried  out 
during the ex-post evaluation of the SRSP by providing a qualitative and quantitative 
analytical  overview  of  the  main  results.  The  purpose  of  this  Annex  is  to  inform 
policymaking on the outcome of all consultation activities and to inform stakeholders 
on how their input was considered.  

The  contributions  received  in  the  context  of  this  public  consultation  cannot  be 
regarded  as  the  official  position  of  the  Commission  and  its  services  nor  that  the 
contributions can be considered as a representative sample of the EU population.  

Outline of the consultation strategy 

The consultation strategy involved the following three phases:  

-  Phase  1:  Planning  the  stakeholder  consultation  and  establishing 
the consultation  strategy (setting out consultation objectives; mapping 
stakeholders;  and  creating  consultation  (OPC  and  targeted  survey, 
interview  and  focus  group)  questionnaires  that  are  linked  to  the  specific 
evaluation questions and indicators); 

-  Phase 2: Conducting consultation work (announcing the launch of the 
OPC  and  the  targeted  survey;  running  their  execution,  the  interview  and 
focus group programmes); 

-  Phase  3:  Informing  policymaking  (analysing  the  data  collected  during 
both  the  online  public  and  targeted  consultations;  providing  an  analytical 
synopsis  of  the  consultation  results;  triangulating  the  data  with  other 
sources;  linking  the  data  collected  to  specific  indicators  and  judgement 
criteria of evaluation questions). 

FIGURE  20:  INTERACTING  PHASES  AND  KEY  STEPS  OF  THE  STAKEHOLDER  CONSULTATION 
PROCESS 

Source: independent evaluation study. based on the Better Regulation Guidelines. 

95 

 
 
 
The purpose of the stakeholder consultation in this evaluation was to gather views of 
the  key  stakeholders  and  data  needed  to  inform  responses  to  the  evaluation 
questions.  Given  the  wide  scope  of  the  SRSP,  the  consultation  sought  to  include 
different  categories  of  stakeholders,  in  particular  the  European  Commission 
officials,  Member  State  beneficiary  and  coordinating  authorities,  providers  of 
technical  support,  representatives  of  industries,  businesses,  workers’  and  other 
organisations, other public authorities, research centres and consultancies, research 
institutions as well as the general public.  

TABLE  11.  CATEGORIES  OF  STAKEHOLDERS  CONSULTED  DURING  THE  EX-
POST EVALUATION 

STAKEHOLDER CATEGORY 

EXAMPLES 

METHOD OF 
CONSULTATION 

European Commission  

DG REFORM officials 
Representatives  of  other  DGs  and 
services  of  the  European  Commission 
involved with the SRSP 

Member  State  beneficiary 
authorities 

Ministries,  public  agencies,  regulatory 
authorities, other public bodies  

Member  State  coordinating 
authorities 

Government offices, ministries 

Technical support providers 

Industry/business/workers’ 
and other organisations 

Public authorities 

Research 
centres/consultancies 

Research/academia 

General public/individuals 

Organisations 
provided technical support  

and 

experts 

that 

for 

and 

other 

interests; 

Commission 

Multinational/global 
national 
small,  medium  and  large  enterprises; 
Business organisations; Trade Unions; 
Chambers  of  Commerce  or  other 
representatives of  the  labour  market; 
national  organisations  representing 
for-profit 
not-for-profit 
organisations 
EU  institutions  (apart  DG  REFORM 
staff 
and 
responsible 
the  programme), 
Agencies  and  other  bodies;  national 
governments;  national  parliaments; 
authorities; 
regional/local/municipal 
competent 
national 
authorities/agencies 
(except 
beneficiary authorities) 
Think-tanks  in  relevant  policy  areas; 
professional  consultancies,  law  firms 
assistance 
(except 
providers) 
Higher 
institutions; 
schools/institutes/educational centres 
EU citizens 

education 

technical 

survey, 

consultation 
interviews, 
interviews, 

consultation 
interviews, 
focus 

Targeted 
(exploratory 
case 
study 
validation interviews) 
Targeted 
(case  study 
targeted 
groups) 
Open  public  and  targeted 
consultation  (case  study 
targeted 
interviews, 
survey, focus groups) 
Targeted 
(case  study 
targeted survey) 

consultation 
interviews, 

Open public consultation  

Open public consultation 

Open public consultation  

Open public consultation 

Open public consultation 

Source: independent evaluation study. 

The  consultation  activities  included  both  the  online  public  consultation  and  the 
targeted  consultation  whose  results  are  presented  in  the  following  chapters  of  this 
Annex: 

1)  Online public consultation (OPC). Duration: 1 March 2022 – 24 May 2022, 
26 responses from 16 EU Member States. Results of the OPC are summarised 
in section 2.1 of the Annex 3. 

96 

 
2)  Targeted consultation of specific stakeholders employed three methods: 

a targeted survey, interviews and focus group discussions.  
-  Targeted survey. Duration: 19 May 2022 – 7 June 2022, 260 responses 
from all EU Member States. Results are summarised in section 2.2 of the 
Annex 3. 
Interview  programme.  Duration:  26  January  2022  –  6  September 
2022,  110  interviews  with  representatives  from  17  Member  States  and 
other key stakeholders (see Table 13). Results are summarised in section 
2.3 of the Annex 3. 

- 

-  Two focus group discussions. Duration: 22  – 27 June 2022, a total of 
17  participants  from  12  EU  Member  States.  Results  are  summarised  in 
section 2.4 of the Annex 3. 

Special attention was paid for the selection of participants of consultation activities. 
To  grasp  their  varying  experiences,  at  least  one  representative  from  each  Member 
State that  received SRSP funding  was involved not  only in the  OPC  or the targeted 
survey, but also in the interview programme or focus group discussions.  

TABLE  12.  DISTRIBUTION  OF  PARTICIPANTS 
ACTIVITIES 

IN  CONSULTATION 

COUNTRY 

PARTICIPATION 
IN THE OPC 

PARTICIPATION 
IN TARGETED 
SURVEY 

PARTICIPATION 
IN INTERVIEW 
PROGRAMME 

PARTICIPATION 
IN FOCUS 
GROUPS 

Bulgaria  
Greece  
Hungary  
Lithuania  
Poland  
Portugal  
Slovakia  
Spain  
Sweden  
Cyprus  
Croatia  
Czechia 
Finland 
Romania 
Ireland 
Italy 
Latvia 
Austria 
Belgium 
Denmark 
Estonia 
France 
Germany 
Luxembourg 
Malta 
Netherlands 
Slovenia 

Source: independent evaluation study. 

The activities of the online public and targeted consultation adhered to the principles 
(participation,  openness  and  accountability,  effectiveness  and  coherence)  and 
standards  (clear  content  of  the  consultation  process,  targeting  and  inclusiveness, 
adequate  publication,  time  limits  for  participation  and  acknowledgement  of 
feedback) of the Better Regulation Guidelines.  

97 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
When  referring  to  quantitative  data,  the  synopsis  report  presents  the  summary  of 
key  questions  analysed  only  in  a  descriptive  way.  As  per  the  Better  Regulation 
Guidelines,  the  evaluation  refers  both  to  the  percentage  and  the  number  of 
selections of a particular OPC or targeted survey response out of total responses to a 
particular question to which this percentage correspond. While presenting results of 
the quantitative consultation activities, N indicates the total number of responses to 
a particular OPC or targeted survey question. In addition, N refers to the sample of 
responses to the question of satisfaction and outcome questionnaires at hand. 

Main results of the consultation process  

1.1. 

Summarised results of the Open Public Consultation (OPC) 

Implementation of the OPC 

The OPC is a web-based consultation that was launched on 1 March 2022 and closed 
on  24  May  2022  on  the  dedicated  website  of  the  European  Commission  ‘Have  your 
say’.  The  consultation  was  running  in  all  EU  working  languages  for  12  weeks.  It 
consisted  of  two  main  structural  parts:  the  background  information  and  SRSP-
related  questions  regarding  the  key  issues  of  effectiveness,  efficiency,  relevance, 
coherence and EU-added value of the programme. The OPC aimed to gather views of 
a broad spectrum of stakeholders who are interested in the evaluation of the SRSP.  

The  invitation  to  participate  in  the  OPC  was  disseminated  through  social  media 
platforms  and  websites  of  both  DG  REFORM  and  the  contractor,  directly  contacting 
and  informing  stakeholders  from  all  EU  Member  States  that  could  be  interested  in 
this  OPC93  as  well  as  sharing  the  invitation  to  the  OPC  with  the  registered 
participants  of  DG  REFORM  online  events  and  country  experts  involved  in  the 
preparation of the case studies for further dissemination. 

Taken  together,  these  measures  resulted  in  26  responses  to  the  OPC.  Due  to  a 
limited  number  of  responses  and  their  quantitative  nature,  the  data  were  manually 
checked  to  avoid  cases  of  campaigning.  No  responses  were  discarded  while 
preparing  the  synopsis  report.  Descriptive  statistics  was  used  as  the  main  method 
for the data interpretation. However, it should be noted that the results of the OPC 
are  not  statistically  representative  and  are  only  used  to  triangulate  information 
collected during the evaluation process. 

Participants of the OPC 

Participants  of  the  OPC  represented  four  types  of  stakeholders:  academic/research 
institution,  public  authority,  company/business  organisation  and  EU  citizens  (see 
Figure 36).                        

93 A total of 126 stakeholders were contacted and informed about the OPC. 

98 

 
 
FIGURE 21: CATEGORIES OF THE OPC RESPONDENTS 

3

3

8

12

Academic/research institution

EU citizen

Public Authority

Company/Business organisation

Source: the results of OPC. 

Public  authorities  were  the  most  represented  category  among  the  four,  covering 
46% of respondents (12 out of 26). The vast  majority (75%; 9  out of 12) of these 
public  authorities  were  large-sized  (more  than  250  employees),  17%  (2  out  of  12) 
were medium-sized (50-249 employees) and 8% (1 out of 12) were small-sized (10-
49). The sample included  agencies (3 out of 12), local authorities (1 out of 12), 
regional authorities (1 out of 12) and other types of authorities (7 out of 12). 

The second most represented category – EU citizens – covered 31% of respondents 
(8  out  of  26),  while  academic/research  institutions  and  companies/business 
organisations  shared  the  smallest  amount  of  responses  (12%  (6  out  of  26)). 
Regarding academic/research institutions, all of the respondents in this category 
were  representatives  of  large-sized  (250+)  universities.  Variation  within  the 
companies/business  organisations  category  was  higher  in  terms  of  both  size 
and  type, 
large-sized  (250+) 
organisations. 

including  one  small-sized  (10-49)  and  two 

Sixteen countries were  represented (see Figure 37) in the OPC. However, their 
representation  was  not  equal  with  the  highest  share  of  participants  from  Romania 
(19%; 5 out of 26) and Italy (12%; 3 out of 26). Two responses were received from 
Croatia, Latvia, Malta, and Portugal, while one response was provided from Austria, 
Bulgaria,  Cyprus,  Finland,  Hungary,  Ireland,  Lithuania,  Netherlands,  Spain,  and 
Sweden. 

99 

 
 
FIGURE 22: GEOGRAPHICAL DISTRIBUTION OF THE OPC RESPONDENTS 

5

3

6

5

4

3

2

1

0

2

2

2

2

1

1

1

1

1

1

1

1

1

1

Source: the results of OPC. 

Some  participants  (46%;  12  out  of  26)  were  very  familiar  with  the  SRSP.  Within 
this  group,  two  respondents  were  not  involved  in  the  implementation  of  SRSP-
funded  projects,  but  the  rest  of  them  were  involved  in  different  countries  (Croatia, 
Romania,  Malta,  Hungary,  Austria,  Latvia,  and  Bulgaria).  Furthermore,  31%  (8  out 
of 26) of participants were moderately familiar with the SRSP (8 out of 26). While 
three of them were not involved in the implementation of the SRSP-funded projects, 
the  rest  of  them  were  involved  in  Portugal,  Finland,  Latvia,  Ireland,  and  Spain.  As 
little  as  15%  of  participants  were  slightly  familiar  with  the  SRSP  (4  out  of  26). 
While  half  of  them  were  not  involved  in  the  implementation  of  the  SRSP-funded 
projects, the rest of them was involved in Lithuania and Sweden. Finally, only 8% of 
participants  were  not  familiar  at  all  with  the  SRSP  (2  out  of  26)  –  none  of  them 
were involved in the implementation of the SRSP-funded projects. 

Relevance and coherence of the SRSP 

More  than  a  half  of  participants  of  the  OPC  considered  the  SRSP  as  the  most 
appropriate  means  for  identified  challenges  related  to  the  EU`s  economic 
governance  processes  and  Member  States`  limited  administrative  and 
institutional  capacity  to  design  and  implement  structural  reforms  (20  and  21 
respondents respectively found the SRSP appropriate to a moderate or high extent in 
these  areas).  However,  the  perception  on  the  suitability  of  SRSP  to  address  the 
challenge of inadequate application and implementation of EU legislation to 
achieve  the  EU`s  fundamental  goals  was  perceived  somewhat  lower  (with  14 
respondents finding the SRSP appropriate to a moderate or high extent in this area).  

The  majority  of  participants  were  not  able  to  evaluate  the  SRSP’s  suitability  for 
addressing  issues  in  such  thematic  areas  as  sustainable  development,  labour 
market,  social  protection,  and  migration,  skills,  education  and  training  or  financial 
sector and access to finance (i.e. nearly half of them responded ‘do not know/cannot 
answer’).  However, a  significant part of  respondents recognised the SRSP as highly 
or  moderately  appropriate  to  address  the  needs  of  EU  Member  States  in  designing 
and  implementing  reforms  in  the  horizontal  areas  of  governance  and  public 
administration  (77%,  20  out  of  26)  and  digitalisation  (58%,  15  out  of  26). 
Additionally, eight participants followed up on this aspect in the comments, pointing 
out  the  specific  challenges  that  should  be  addressed  by  SRSP  (e.g.  addressing 
specific  national/sub-national  challenges  in  EU  Member  States  as  well  as  doing  it 
more quickly and in a more flexible way). 

100 

 
 
The vast majority of the OPC participants were at least to some extent familiar with 
the different EU programmes94. Among six of them, participants were highly familiar 
with  the  instruments  of  EU  Cohesion  policy  (61%,  16  out  of  26)  and  Horizon 
2020 (42%, 11 out of 26), while to some smaller extent they were familiar with the 
LIFE programme (42%, 11 out of 26 including both moderate and high familiarity). 
In contrast, 30-38% of participants (8-10 out of 26) reported being not familiar at 
all with Fiscalis 2020, Customs 2020, and the Hercule III programme.  

When  comparing  the  complementarity  of  the  SRSP  (by  funding  different  aspects  of 
similar activities, targeting different groups, etc.) with different EU programmes and 
instruments95,  62%  of  participants  (16  out  of  26)  marked  EU  Cohesion  policy  as 
the  most  complementary.  It  was  followed  by  Horizon  2020  (12  out  of  26),  the 
LIFE  programme,  Customs  2020,  and  Fiscalis  2020  (8  out  of  26)  with  the 
Hercule  III  programme  being  the  least  complementary  one  according  to 
participants (7 out of 26). Half of the participants (13 out of 26) were not aware of 
the  complementarity  of  SRSP  actions  with  similar  national  and  regional 
programmes,  while  the  largest  share  of  the  rest  of  participants  considered  SRSP 
actions moderately (9 out of 26) complementary in this regard. 

Some  participants  shared  specific  examples  of  national  and  regional  programmes 
similar to the SRSP (e.g., SAMA and POCI in Portugal, the Action Plan for joining the 
Exchange  Rate  Mechanism  II  (ERM  II)  and  the  Banking  Union  in  Croatia).  When 
answering an open question, one participant stated that complementarity depended 
on  the  Member  State's  strategy  to  use  technical  support  and  to  coordinate  the 
implementation of different initiatives at national level. 

Efficiency of the SRSP 

In  general,  the  majority  of  participants  saw  SRSP  actions  as  moderately  efficient. 
More  than  half  of  them  perceived  SRSP  actions  as  user-friendly,  timely,  and  cost-
effective to a moderate and high extent compared to other EU actions (17 out of 
26 respondents) and national actions (18 out of 26). When it comes to comparison 
with  the  regional  and  local  actions,  half  of  respondents  (13  out  of  26)  perceive 
SRSP actions as moderately and highly efficient. 

Regarding  efficiency  in  the  implementation  of  technical  support  projects,  the 
majority of participants considered the sufficient duration of the projects (20 out 
of  26)  and  the  proportionality  of  administrative  burden  (19  out  of  26)  as 
moderately  and  highly  suitable  factors  in  making  the  SRSP`s  implementation 
efficient.  About  half  of  participants  perceived  the  proportionality  of  the  budget 
scale  of  SRSP  actions  to  the  needs  of  technical  support  requested  by  the 
Member  States  (16  out  of  26)  and  the  appropriateness  of  the  timeliness  of 
administrative procedures (14 out of 26) moderately and highly suitable.  

Effectiveness of the SRSP 

The  majority  of  respondents  (62-73%,  i.e.  16-19  out  of  26)  thought  that  SRSP 
actions  were  to  a  moderate  or  high  extent  successful  in  supporting  national 

94  EU  cohesion  policy  instruments,  Horizon  2020,  LIFE  programme,  Customs  2020,  Fiscalis  2020  and 

Hercule III programme. 

95  EU  cohesion  policy  instruments  (the  European  Regional  Development  Fund,  the  Cohesion  Fund,  the 
European  Social  Fund),  Horizon  2020,  LIFE  programme  (the  EU’s  funding  instrument  for  the 
environment and climate action), Customs 2020 (enables national customs administrations to create 
and  exchange  information  and  expertise),  Fiscalis  2020  (enables  national  tax  administrations  to 
create  and  exchange  information  and  expertise),  Horizon  2020  (the  EU  framework  programme  for 
research  and  innovation)  and  Hercule  III  programme  (combats  irregularities,  fraud  and  corruption 
affecting the EU budget). 

101 

 
 
authorities  in  the  design/implementation  of  reforms,  enhancing  their 
capacity  to  formulate,  develop  and 
implement  reform  policies  and 
strategies,  and  their  efforts  to  design  and  implement  appropriate  processes 
and  methodologies  by  taking  into  account  good  practices  of  and  lessons 
learned  by  other  countries.  While  these  objectives  share  a  rather  similar 
perception,  the  evaluation  was  rather  different  regarding  enhancement  the 
efficiency  and  effectiveness  of  human  resource management  –  50%  (13  out 
of  26)  of  respondents  thought  SRSP  actions  were  to  a  moderate  or  high  extent 
successful, but 38% (13 out of 26) of them did not have an opinion on this objective 
of a more specific nature.  

A total of six participants provided their comments on the usefulness of the outputs 
produced  by  the  SRSP-funded  projects.  While  most  respondents  saw  these  outputs 
as  very  useful,  a  few  participants  perceived  the  production  of  technical  support 
outputs as insufficient to overcome long-standing and deep-rooted challenges (such 
as limited administrative capacity or even excessive bureaucracy). 

The majority of respondents emphasised the high (38%, 10 out of 26) or moderate 
(38%,  10  out  of  26)  contribution  of  the  SRSP  to  the  improvement  of  internal 
working  procedures,  methodologies,  and  processes 
in  beneficiary 
authorities.  The  programme’s  contribution  to  other  outcomes  –  the  adoption  of 
new  strategies,  new 
laws, 
procedures, and actions to improve the way reforms are implemented in the 
Member  States,  as  well  as  on  any  organisational  change  in  beneficiary 
authorities  –  was  perceived  to  be  somewhat  smaller  because  some  of  the 
participants (from 8 to 11 out of 26) did not know about such outcomes.  

laws/acts,  or  amendments  to  current 

More than a half of respondents recognised the high (35%, 9 out of 26) or moderate 
(23%,  6  out  of  26)  contribution  of  the  SRSP  to  structural  changes  within  Member 
States`  beneficiary  authorities  in  preparing  and  implementing  reforms  and 
adequately  applying  EU  law.  The  programme’s  contribution  to  the  impact  of 
achieving  the  relevant  longer-term  growth-sustaining  reforms  as  indicated 
in the CSPs was slightly weaker (with 19% or 5 out of 26 respondents stating that 
the contribution was high and 35% or 9 out of them stating that it was moderate).  

When  it  comes  to  the  success  of  implementing  SRSP-funded  projects  at  national 
level,  56-60%  (15-16  out  of  26)  of  participants  saw  national  ownership  of 
reforms  and/or  cooperation  between  the  European  Commission,  beneficiary 
authorities,  and  providers  of  technical  support  as  the  most  influential  factors. 
The following factors were less frequently mentioned, but still very important during 
the implementation process:  political reform commitments (46%, 12 out of 26), 
national  economic  and  social  situation  (46%,  12  out  of  26),  timing  and 
sequencing  of  reforms  at  the  national  level  (50%,  13  out  of  26),  national, 
regional,  and  local  authorities  working  in  partnership  during  the  reform 
process (42%, 11 out  of 26),  consultation  with different  stakeholders  during 
the reform process (38%, 10 out of 26) and communication of reforms to the 
general  public  or  specific  target  groups  (42%,  11  out  of  26).  Finally,  four 
participants provided additional comments, e.g. mentioning the very high capability 
of SRSS/REFORM project officers, their networking skills and practical approach as a 
significant factor.   

EU value added of the SRSP 

A total of 38% of participants (10 out of 26) believed that Member states could have 
achieved  similar  results  in  the  design  and  implementation  of  reforms  only  to  a 
limited extent without the technical support they received through the SRSP, while 
4% (1 out of 26) thought it could not be possible at all.  

102 

 
The  perception  of  EU  added  value  varied  among  the  main  groups  of  stakeholders. 
Public authorities stressed the relevance of tailored solutions, reflecting the needs 
of  Member  States  as  well  as  an  opportunity  to  collaborate  with  valuable 
partners (e.g. the World Bank or OECD). Representatives of business organisations 
pointed to a comparatively low administrative burden and clear procedures for 
beneficiary  authorities,  ease  of  access  to  international  expertise  and  absence  of 
them.  Representatives  of  academic/research 
co-financing 
institutions  indicated  that  the  technical  support  provided  under  the  SRSP  offered  a 
continuity  to  the  reform  process,  improved  the  development  of  administrative 
capacity and internal reorganisation of institutions. Finally, based on the views of EU 
citizens, the SRSP offered specialised assistance whose delivery would not be 
possible  at  national  and  regional  level  (e.g.  developing  new  methodologies, 
roadmaps or guidelines).  

requirements 

for 

Measures for the improvement of the future programme 

in 

links  between 

the  administrative  burden 

technical  support  projects  and 

Simplification  and  reduction  of 
the 
administration  of  the  programme  in  the  future  as  one  of  the  most  important 
measures  by  vast  majority  -  65%  (17  out  of  26)  participants.  Additionally, 
the  European 
developing 
Semester/Recovery  and  Resilience  Facility  surveillance  process,  and  developing 
flagship  technical  support  projects  that  address  top  EU  priorities  and  the 
common needs of Member States were marked as important by 54% each (14 out of 
26). Furthermore, better support for the digital and green transition, as well as 
support  for  a  broader  range  of  technical  support  activities  on  the  national 
level  were  considered  important  by  38-46%  (10-12  out  of  26  participants).  In 
addition,  involving  more  regional  and  local  authorities  in  the  design  and 
implementation of the technical support and providing more support to multi-
country projects were selected by 46% (12 out of 26) of participants. Finally, the 
least  important  measure  according  to  participants  was  the  better  adaptation  of 
delivery modes  to  Member  States`  needs,  which  was  selected  by  only  27%  (7 
out of 27) of participants.  

Representatives  of  public  authorities  pointed  out  that  the  SRSP  should  better  take 
into  account  the  country-specific  recommendations  and  pay  more  attention  to 
the involvement of grassroots organisations (e.g. NGOs). Additionally, the need 
for  a  more  transparent  management  of  funds  was  highlighted.  Representatives 
of  business  organisations  stressed  that  sharing  of  good  practices  and  lessons 
learned/exchange  of  information  between  beneficiary  authorities  in  different 
Member states could be further improved. Also, they added some recommendations 
regarding  capacity-building  activities  for  beneficiary  authorities  (workshops, 
seminars, etc.), the potential involvement of niche-oriented advisory firms in the 
implementation  of  projects  (rather  than  using  the  already  pre-selected  companies) 
and  the  need  for  making  a  quicker  project  launch  (especially  in  case  of  urgent 
reform  needs).  Finally,  citizens  pointed  out  that  the  on-demand  nature  of  technical 
support be further improved, while beneficiary authorities should be involved in the 
selection  of  technical  support  providers  to  ensure  their  suitability  for  the 
project.  

1.2. 

Summarised results of the targeted survey 

Implementation of the targeted survey and data analysis 

The targeted survey was running from 19 May 2022 to 7 June 2022 on the Alchemer 
survey tool. The survey questionnaire was divided into two parts: (i) an introduction 
and  (ii)  questions  adjusted  to  the  experience  of  national  beneficiary  authorities, 
coordinating  authorities  and  technical  support  providers.  Questions  for  these  target 
groups  were  split  into  several  blocks  on  the  basis  of  key  evaluation  criteria:  the 

103 

 
effectiveness/impact,  efficiency,  relevance,  coherence  and  EU-added  value  of  the 
SRSP.  The  survey  questions  were  carefully  crafted  to  ensure  comparability  across 
the different groups of respondents and complementarity with the other consultation 
methods. 

The main purpose of the survey was to collect opinions on the overall functioning of 
the  SRSP,  the  quality  of  technical  support  received,  the  policy  goals  and  objectives 
met, and the results and impacts achieved throughout the evaluation period (2017-
2020).  DG  REFORM  directly  shared  invitations  to  participate  in  the  targeted  survey 
with the representatives of beneficiary authorities, coordinating authorities as well as 
technical support providers, involved in the preparation, design and implementation 
of SRSP-funded projects.  

To provide robust findings on the targeted survey results, the data was cleaned from 
empty  and  partial  responses.  The  data  analysis  was  based  on  two  pillars:  (i)  a 
descriptive  presentation  of  the  targeted  survey  results  and  (ii)  an  estimation  of 
statistical  associations  between  key  variables.  Since  most  of  the  targeted  survey 
responses  were  ordinal  or  nominal,  Cramer’s  V  was  used  to  measure  the  power  of 
association,  and  Fisher’s  exact  test  was  employed  to  identify  the  statistical 
significance  of  the  findings.  Some  nominal  variables  were  transformed  into  binary 
ones when there  was a need to report  on individual answers to the  multiple-choice 
question  sets.  While  providing  descriptive  analysis  of  the  targeted  survey  results, 
“Do not know/cannot answer” responses were omitted. 

Participants of the targeted survey and scope of the projects involved 

A total of 260 completed responses of the targeted survey were analysed. A total of 
51%  (132)  of  participants  were  representatives  of  the  beneficiary  authorities,  41% 
(106)  –  technical  support  providers,  and  8%  (22)  submitted  responses  as  the 
national coordinating authorities (Figure 38).  

FIGURE 23: SHARE OF THE TARGETED SURVEY RESPONDENTS 

8%

41%

51%

Beneficiary authorities (N=132)

Technical support providers (N=106)

National coordinating authorities (N=22)

Source:  the  targeted  survey  of  beneficiary  authorities,  coordinating  authorities  and  technical  support 
providers, May-June 2022. 

As  presented  in  the  figure  below,  the  representatives  of  the  beneficiary  authorities 
from all EU Member States except France participated in the targeted survey, while 
the  responses  of  national  coordinating  authorities  from  Austria,  Belgium,  Denmark, 
Ireland,  Latvia,  Lithuania,  and  Slovenia  were  absent.  The  number  of  respondents 
from  different  Member  States  closely  reflected  the  distribution  of  SRSP-funded 
projects across the countries.   

104 

 
 
FIGURE  24:  PARTICIPATION  OF  REPRESENTATIVES  OF  BENEFICIARY  AND  NATIONAL 
COORDINATING AUTHORITIES IN THE TARGETED SURVEY BY THE COUNTRY OF INSTITUTION 

1

1 1 3

1

12 12

10 10

9

8 8

14

12

10

8

6

4

2

0

1

7

1

1 1

6 6 6 6

1 1

1

5 5

4 4

2

2 2 2

2

1

1 1 1 1 1
1 1 1 1 1

1

Beneficiary authorities (N=132)

National coordinating authorities (N=22)

Source: the targeted survey of beneficiary authorities and coordinating authorities, May-June 2022. Note: 
numbers  in  brackets  refer  to  the  number  of  SRSP-funded  projects,  implemented  in  a  particular  Member 
State.  

Regarding  the  previous  experience  of  implementation  of  the  SRSP-funded  projects, 
72%  of technical support providers were involved in two and  more (76 out  of 106) 
projects, while more than a half of beneficiary authorities (51%, 67 out of 132) were 
involved only in one project. 

As  presented  in  the  table  below,  the  highest  share  of  the  targeted  survey 
participants was involved in the SRSP 2019 projects.  The distribution  of beneficiary 
authorities  and  technical  support  providers  in  the  SRSP-funded  projects  was  rather 
equal  through  the  period  of  evaluation  (2017-2020),  except  for  the  SRSP  2017. 
There  was  only  one  technical  support  provider  who  took  part  in  the  Article  11 
programme in 2018. 

TABLE  13.  PARTICIPATION  IN  THE  SRSP-FUNDED  PROJECTS  BY  YEAR, 
2017-2020 

BENEFICIARY AUTHORITIES 
(N=132) 

TECHNICAL SUPPORT PROVIDERS 
(N=106) 

SRSP 2017  

SRSP 2018  

SRSP 2019  

SRSP 2020  

Article 11 2018  

30% 

42% 

 49% 

38% 

 0% 

17% 

43% 

57% 

45% 

1% 

Source: the targeted survey of beneficiary authorities and technical support providers, May-June 2022. 

A  relatively  equal  share  of  beneficiary  authorities  and  technical  support  providers, 
who  responded  to  this  targeted  survey  question  delivered/received  support  in  the 
fields  of  governance  &  public  administration  as  well  as  financial  sector  &  access  to 
finance (Figure 40). A larger share  of the  contractors provided  support in the fields 
of  revenue  administration  &  public  financial  management,  sustainable  growth  & 

105 

 
 
 
business  environment,  labour  market,  education,  health  &  social  services,  and 
horizontal policies. 

FIGURE 25: SECTORAL AREAS OF THE SRSP TECHNICAL SUPPORT, 2017-2020 

60%

50%

40%

30%

20%

10%

0%

24%

18%

Revenue
Administration
& Public
Financial
Management
(N=24/25)

48%

42%

42%

27%

11%

15%

18%

20%

19%

Governance &
Public
Administration
(N=64/44)

Sustainable
Growth &
Business
Environment
(N=15/29)

Labour Market,
Education,
Health & Social
Service
(N=20/45)

Financial Sector
& Access to
Finance
(N=24/21)

5%

Horizontal
Policies
(N=7/20)

Beneficiaries (N=132)

Technical support providers (N=106)

Source: the targeted survey of beneficiary authorities and technical support providers, May-June 2022. 

The  majority  of  respondents  received  or  provided  technical  support  through  public 
procurement  (57%  of  beneficiary  authorities,  75  out  of  108,  51%  of  technical 
support  providers,  54  out  of  98).  A  significantly  lower  share  of  beneficiaries  (17%, 
23 out of 108) and technical support providers (28%, 30 out of 98) used grants. No 
more  than  15%  of  respondents  from  both  groups  used  the  TAIEX,  delegation 
agreements, Fiscalis 2020 and direct expertise of the Commission. 

Relevance and coherence of SRSP 

Regarding  specific  types  of  reforms  that  beneficiary  authorities  targeted  in  their 
requests  for  technical  support  under  the  SRSP  (2017-2020),  the  vast  majority  of 
reforms  were  undertaken  at  the  own  initiative  of  Member  States  (58%,  77  out  of 
132).  However,  a  large  share  of  reforms  was  also  related  with  the  context  of 
economic  governance  (43%,  57  out  of  132)  or  linked  to  the  Union  priorities  (31% 
and  41  out  of  132).  A  smaller  number  of  respondents  indicated  reforms  being 
related  to  the  Union  law  (17%,  22  out  of  132)  or  the  implementation  of  economic 
adjustment programmes (3%, 4 out of 132).  

The majority of beneficiary and coordinating authorities as well as technical support 
providers  (84%  –  98%)  agreed  or  strongly  agreed  that  the  SRSP  was  a  suitable 
instrument to provide technical support, addressed the key needs of beneficiary 
institutions, corresponded to the key reform goals of the Member State, the project 
design  was  appropriate,  and  the  modes  of  delivery  selected  for  the  project(s)  met 
their implementation needs. 

When  assessing  the  further  need  for  technical  support,  the  majority  of 
respondents  indicated  that  it  is  still  relevant  both  for  the  particular  institutions  and 
the  Member  States.  As  many  as  95%  of  representatives  of  national  coordinating 
authorities  (21  out  of  22)  reported  that  the  support  is  needed  to  a  large  (45%,  10 
out of 22) or to some extent (50%, 11 out of 22). This view was supported by 83% 
of  beneficiary  authorities  (100  out  of  121),  the  majority  of  which  indicated  that 
support  is  needed  to  some  extent  (61%,  74  out  of  121).  Finally,  the  views  of 
technical support providers were more positive: 39% and 43% of them claimed that 

106 

 
 
the support is still needed to a large or some extent respectively (38 and 42 out of 
97 respectively).  

There  were  no  clear  trends  when  assessing  the  complementarity  of  the  SRSP 
with the similar national and regional programmes. According to the targeted 
survey results, the largest share of national beneficiary authorities (72%, 35 out of 
49)  considered  that  SRSP  to  a  moderate/high  extent  complemented  the  support 
provided through the ESIF. Similarly, the majority of the beneficiary authorities that 
offered  an  opinion  on  this  issue  (69%,  34  out  of  49)  considered  that  the  SRSP 
complemented  the  interventions  enhancing  the  institutional  capacity  of  public 
authorities  and  stakeholders  and  efficient  public  administration  supported  by  the 
European  Social  Fund  and  the  European  Regional  Development  Fund.  At  the  same 
time,  the  respondents  claimed  that  the  SRSP  complemented  the  modernisation  of 
public  and  private  employment  services  (ESF)  and  investments  in  employment 
infrastructure by the ERDF only to a limited extent (40%, 14 out of 35).  

the  contribution  of 

the  SRSP-funded  projects 

When  assessing 
the 
implementation of the EU priorities for 2019-2024, the majority of beneficiary 
and  coordinating  authorities  agreed  on  their  moderate  and  high  contribution  to  the 
development of a strong and vibrant economic base (66%, 67 out of 116) and to a 
lesser extent – to building a climate-neutral, green, fair, and social Europe (58%, 54 
out  of  116).  The  smallest  share  of  beneficiaries  and  coordinating  authorities  (38%, 
29 out of 108) indicated that the projects moderately and highly contributed to the 
protection of citizens and freedoms.   

to 

Efficiency of the SRSP 

respondents 

the  preparation, 
The  majority  of 
implementation,  and  evaluation  of  SRSP-funded  projects  were  carried  out  in  a 
timely  and  efficient  manner  (see  the  figure  below),  but  their  agreement 
concerning preparation for implementation was somewhat weaker.  

(over  80%)  stated 

that 

FIGURE 26: TIMELINESS AND EFFICIENCY OF THE SRSP-FUNDED PROJECTS 

Preparation for implementation: time taken from the
submission of technical support requests to the start
of delivering technical support (N=116)

The implementation of technical support projects:
time taken from the conclusion of
contracts/agreements with technical support providers
to the completion of projects (N=116)

The evaluation and selection of technical support
requests: time taken from the submission of technical
support requests to the approval of respective
financing decisions (N=115)

54%

33%

11%

2%

66%

27%

5%

2%

63%

28%

8%

2%

To a large extent

To some extent

To a limited extent

Not at all

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

Source: the targeted survey of beneficiary authorities, May-June 2022. 

The absolute majority of respondents (92-97%, 74-77 out of 80) agreed or strongly 
agreed that the administration of technical support was efficient. However, 8% 
of  beneficiaries  disagreed  that  the  evaluation  procedures  and  processes  after  the 
completion  of  projects  was  clear,  transparent,  and  user-friendly  (10  out  of  119). 
Moreover,  7%  claimed  that  the  financial  contribution  of  the  SRSP  was  not 
proportional to the needs of technical support (8 out of 115).  

107 

 
 
Regarding  the  efficiency  of  the  SRSP  funded  projects  implementation  on  the 
ground,  the  majority  of  beneficiary  authorities  and  technical  support  providers 
(strongly) agreed that the support of DG REFORM officers was useful (89%, 118 out 
of 130 and 92%, 97 out of 102, respectively). Beneficiary authorities also (strongly) 
agreed  that  DG  REFORM  was  able  to  provide  support  from  the  start  of  technical 
support  to  its  end  (91%,  120  out  of  131)  and  risks  of  the  project(s)  have  been 
effectively managed (85%, 112 out of 127). In addition, technical support were seen 
as  having  the  required  expertise  and  skills  according  to  all  delivery  modes  (88%, 
109 out of 124), while the cooperation with them was treated as efficient (86%, 106 
out  of  122).According  to  the  providers  of  technical  support,  bureaucratic  resistance 
to change during the implementation of the project activities was one of their main 
inefficiencies (55%, 58 out of 94 (strongly) agreed on its occurrence). 

FIGURE  27:  THE  EXTENT  TO  WHICH  THE  RESPONDENTS  AGREED  THAT  THE  PROVIDERS  OF 
TECHNICAL  SUPPORT  HAS(VE)  HAD  THE  REQUIRED  EXPERTISE  AND  SKILLS  ACCORDING  TO 
THE DELIVERY MODES 

Public procurement (N=70)

Grants (N=21)

TAIEX (N=17)

49%

52%

53%

Delegation agreements (N=11)

82%

Fiscalis (N=1)

100%

36%

14%

1%

38%

35%

10%

12%

18%

Commission direct expertise (N=9)

67%

11%

22%

0%

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Strongly agree

Agree

Disagree

Strongly disagree

Source: the targeted survey of beneficiary authorities, May-June 2022. 

Effectiveness of the SRSP 

The  respondents  reported  some  differences  in  the  stages  of  the  policy  cycle 
the  SRSP-funded  projects.  Beneficiary  authorities  worked 
addressed  by 
predominantly  on  policy  development  (76%,  100  out  of  132)  and  policy 
implementation  (51%,  67  out  of  132).  A  fifth  of  them  (19%,  25  out  of  132)  were 
involved in the evaluation phase of the cycle during their projects. 

technical 

support  providers 

Both  beneficiary  authorities  and 
reported 
recommendations (73%, 96 out of 132 and 86%, 91 out of 106, respectively) being 
the  most  common  outputs  expected  to  be  delivered  through  the  technical  support 
projects. The other outputs included analyses and reports (68% of beneficiaries, 90 
out of 132, 83% of technical support providers, 88 out of 106), workshops, training, 
training  material  (65%  of  beneficiaries,  86  out  of  132,  84%  of  technical  support 
providers,  89  out  of  106)  as  well  as  action  plans  and  roadmaps  (58%  of 
beneficiaries, 77 out of 132, 69% of technical support providers, 73 out of 106). The 
least  common  outputs  were  legislative  proposals  (20%  of  beneficiaries,  27  out  of 

108 

 
 
132, 29% of technical support providers, 31 out of 106) and terms of reference (9% 
of beneficiaries, 12 out of 132, 16% of technical support providers, 17 out of 106).  

Similarly,  the  beneficiary  authorities  referred  to  the  results  of  the  analysis  and 
reports  used  (90%,  114  out  of  124),  new  knowledge  and  skills  maintained  (93%, 
107 out of 120) as well as recommendations implemented (80%, 101 out of 121) as 
deliverables of the projects that were to some or large extent  used in their work. 
In terms of the actual delivery of outputs, 80% of beneficiary authorities (106 out 
of  130)  and  64%  of  technical  support  providers  (68  out  of  103)  indicated  that  all 
project outputs were delivered. 

Regarding  the  factors  that  positively  or  negatively  affected  the  delivery  of 
technical support outputs and results, answers of all the groups of respondents 
were  analysed  together  (Figure  42)96.  The  cooperation  between  the  EC,  beneficiary 
authorities  and  technical  support  providers,  consultation  with  different  stakeholders 
during  the  reform  process  as  well  as  partnership  with  national,  regional  and  local 
authorities  were  named  as  factors  with  a  moderate  or  strong  positive  impact.  In 
terms  of  factors  with  the  moderate  or  strong  negative  effect,  the  COVID-19 
pandemic, time and sequencing of  reforms as well as political reform commitments 
were distinguished. 

96 Three answer options specific for different groups of participants, namely, objectives and the scope of 
the  reforms  initially  planned,  the  quality  of  the  technical  support  request  design  and  bureaucratic 
resistance to change during the implementation of the project(s) activities, were excluded to ensure 
comparability of responses across the groups. 

109 

 
 
FIGURE  28:  FACTORS  AFFECTING  THE  DELIVERY  OF  OUTPUTS  AND  RESULTS  OF  THE  SRSP-
FUNDED PROJECTS 

COVID-19 crisis (N=213)

7%

6%

31%

41%

15%

Communication of reforms to the general public or
specific target groups (N=193)

12%

43%

41%

3%

1%

Consultation with different stakeholders during the
reform process (N=224)

46%

35%

13%

4%

1%

Working of national, regional and local authorities in
partnership during the reform process (N=212)

34%

34%

25%

6%

1%

Cooperation between the European Commission,
beneficiary authorities and providers of technical
support (N=234)

58%

25%

14%

2%

1%

Timing and sequencing of reforms at national
level/election cycle (N=205)

16%

34%

27%

18%

5%

National economic and social situation (N=209)

11%

26%

47%

14%

2%

National ownership of reforms (N=213)

33%

27%

26%

10%

5%

Political reform commitments (N=200)

28%

24%

29%

16%

4%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

A strong positive effect
No effect
A strong negative effect

A moderate positive effect
A moderate negative effect

Source:  the  targeted  survey  of  beneficiary  authorities,  coordinating  authorities  and  technical  support 
providers, May-June 2022. 

The  results  of  the  targeted  survey  highlight  some changes  observed  in  sectoral 
policy  areas  because  of  implementing  the  projects.  According  to  beneficiary 
authorities, the projects resulted in the production of new information or knowledge 
(89%,  92  out  of  103),  improved  strategies  and  reform/policy  documents  (79%,  81 
out of 100) and issuing or modification of ‘soft’ instruments (67%, 68 out of 101) to 
some or a large extent. Such changes as better application of economic instruments 
(taxes,  charges,  fees,  etc.)  (31%,  23  out  of  74)  or  improved  application  and 
implementation of EU law (43%, 35 out of 82) were less common.  

While  responding  to  an  open  question  on  the  contribution  of  the  SRSP-funded 
projects  to  reforms  and  policies  in  their  country,  beneficiary  and  coordinating 
authorities  highlighted  the  production  of  practically  applicable  tools  (guidelines, 
roadmaps, action plans, etc). They mentioned the relevance of the received support 
for  specific  policy  fields,  resulting  in  an  improvement  of  legislation,  regulations  and 
strategic  documents,  an  increased  administrative  capacity  and  a  better  quality  of 
public services. 

The submission of the request for technical support for a follow-up-project or a new 
stage of the larger project was named as the most frequent activity (36%, 41 out of 
114)  among  follow-up  actions  taken  after  the  project  end  and  building  upon  the 
project  outputs  and  results.  Overall,  the  beneficiary  authorities  were  very  satisfied 
with the SRSP-funded projects (65%, 79 out of 121 were very satisfied, 29%, 35 out 

110 

 
 
of  121  were  satisfied).  Approximately  a  half  of  beneficiaries  (49%,  54  out  of  111) 
would consider requesting support from the TSI and 46% have already requested for 
it (51 out of 111).  

While  answering  to  the  open  questions  on  the  implementation  of  the SRSP  and 
the possible improvements of the future programme, significant overlaps were 
noted in the opinion of beneficiary authorities, coordinating authorities and technical 
support  providers.  All  these  stakeholders  raised  the  issue  of  project  impact  and 
sustainability  that  depend  on  national-level  political  commitment  and  continuous 
monitoring  and  provision  of  resources  by  the  EC.  In  addition,  the  need  for  closer 
collaboration  across  countries  and  projects  was  stressed  with  an  aim  to  learn  and 
transfer  professional  experience.  Looking  from  the  administrative  perspective,  the 
period  between  the  submission  of  requests  for  services  and  the  start  of  project 
activities was named as too long. Furthermore, since the selection of technical 
support  providers  was  seen  as  insufficiently  transparent,  a  broader 
selection,  a  clearer  criterion  of  selection  and  a  greater  inclusion  of 
beneficiary authorities were seen as possible areas for improvement in this 
process. 

EU value added of the SRSP 

The  SRSP-funded  projects  had  a  significant  EU  added  value.  A  total  of  88%  of 
beneficiaries  claimed  that  they  used  good  practices  and  lessons  learnt  from  the 
SRSP-funded projects to a large or to some extent (105 out of 122), while 12% (14 
out  of  122)  indicated  not  using  the  good  practice/lessons  learned  at  all  or  using 
them only to a limited extent. At the same time, 80% of respondents (44-45 out of 
111)  indicated  that  a  similar  support  would  have  been  either  not  available  or 
available to a limited extent without the SRSP funding (Figure 43). 

FIGURE 29: EU VALUE-ADDED OF THE SRSP 

To a large extent (N=71/3)

3%

60%

To some extent (N=34/19)

17%

29%

To a limited extent (N=8/44)

Not at all (N=6/45)

7%

5%

40%

41%

0%

10%

20%

30%

40%

50%

60%

70%

Usage of practices and lessons from the SRSP projects (N=119)

The extent to which the support could have been available without SRSP (N=111)

Source: the targeted survey of beneficiary authorities, May-June 2022. 

1.3. 

Summarised results of interviews 

Implementation and participants of interviews  

Interviews with EU- and national-level stakeholders were  conducted to complement 
findings from desk  research, the  OPC and the  targeted survey  of stakeholders.  The 
interview programme was divided into four parts: (i) exploratory interviews with the 
European  Commission  officials,  (ii)  exploratory  and  follow-up  interviews  with  the 
national  authorities,  (iii)  case  study  interviews,  and  (iv)  validation  interviews.  The 
interviews  were  conducted  following  a  standardised  questionnaire,  including 

111 

 
 
questions  on  the  evaluation  criteria  and  adjusted  to  the  experience  of  individual 
stakeholders.  Each  interview  was  recorded,  interview  notes  and/or  transcripts  were 
prepared based on the recording. A total of 110 interviews were conducted (see the 
table below for the total number of interviews based on their type).   

TABLE 14. PARTICIPANTS OF THE INTERVIEW PROGRAMME 

INTERVIEW TYPE  NUMBER OF INTERVIEWS 

PARTICIPANTS 

Exploratory 
interviews  at 
the EU level 

Exploratory 
and 
follow-up 
interviews  at 
national level 

17 interviews 

8 interviews 

Case 
interviews 

study 

83 interviews 

DG  REFORM  staff,  including  (deputy)  heads  of 
units,  advisers,  dedicated  country  coordinators 
and project managers 

Representatives  of  other  Commission  services 
related  to  the  SRSP  and  involved  in  the  Inter-
service Steering Group 

Interviews  outside  the  scope  of  the  case  study 
programme with: 

Member States’ authorities that requested and/or 
received 
technical 
(beneficiary 
support under the SRSP 

authorities) 

National  coordinating  authorities  in  individual 
Member States 

Member States’ authorities that requested and/or 
received 
technical 
(beneficiary 
support under the SRSP 

authorities) 

National  coordinating  authorities  in  individual 
Member States 

Other national authorities that participated in the 
implementation of technical support projects  

Technical support providers 

DG REFORM staff  

Interested  stakeholders  (subjects  concerned  by 
the  reform,  including  politicians,  civil  servants, 
social partners, civil society organisations, etc. 

Professional and academic experts 

Validation 
interviews 

Total: 

2 interviews 

Officials of the European Commission 

110 interviews 

Source: independent evaluation study. 

The  interview  programme  was  designed  to  embrace  a  wide  variety  of  views, 
including  those  coming  from  the  beneficiary  or  national  coordinating  authorities  as 
well  as  from  the  policy  officers  and  other  officials  of  the  EC.  Depending  on  the 
question  at  hand,  the  interview  programme  was  used  either  to  supplement  other 
sources  of  evidence  or  as  the  main  information  source  in  cases  when  other  data 
sources were scarce. 

Relevance and coherence of the SRSP 

The  initial  results  of  the  interview  programme  point  to  the  importance  of  dedicated 
calls.  According  to  the  participants  of  exploratory  and  case  study  interviews,  the 
introduction of these calls allowed providing more targeted support, addressing both 
the urgent needs of Member States as well as the EU priorities and country-specific 
recommendations  of  the  European  Semester.  The  opportunity  to  receive  timely 
support (more often than through the annual call) was also highlighted. 

112 

 
 
 
The  evaluation  evidence  also  suggests  that  there  are  sufficient  mechanisms  to 
ensure  the  linkages  and  integration  between  the  SRSP  and  the  European  Semester 
process/CSRs. For instance, the projects addressing CSRs are prioritised  during the 
evaluation of requests for technical support. However, there is insufficient tracking if 
specific  CSRs  are  fulfilled  due  to  the  results  of  SRSP  projects  implemented  at 
national level. 

Case study interviews revealed that the SRSP funded projects  were complementary 
and had high level synergies with other projects, programmes, and/or interventions 
occurring at the level of the Member State or EU. In some Member States the SRSP 
funding  was  necessary  to  implement  other  projects  or  achieve  access  to  funding 
mechanisms,  while 
the 
implementation of reforms or broadening of their scope. 

follow-up  actions  were 

the  others 

taken 

for 

in 

Efficiency of the SRSP 

Participants of the exploratory, follow-up and case study interviews highlighted that 
the  administrative  burden  of  the  SRSP-funded  projects  was  similar  or  even  lower 
compared  to  the  other  EU  programmes.  The  bilateral  contract  between  the 
technical  support  providers  was  mentioned  as  reducing 
Commission  and 
administrative  burden  for  the  beneficiary  authorities  and  allowing  them  to  dedicate 
necessary  resources  on  the  actual  implementation  of  technical  support  projects. 
However,  some  drawbacks  were  mentioned,  including  a  long  time  period  between 
the  submission  of  requests  for  technical  support  and  the  beginning  of  project 
implementation (sometimes taking up to one year of time).  

Whereas  the  budget  of  the  programme  steadily  increased  over  the 
evaluation  period  2017-2020,  the  human  resources  in  DG  REFORM  did  not 
keep  the  pace  with  this  development.  As  a  consequence,  DG  REFORM  staff 
needed to manage an increasing number and more complex projects. This increased 
the workload of responsible policy officers and, as noted by some of the participants 
of  case  study  interviews,  reduced  their  ability  to  provide  tailor-made  expertise 
during the provision of technical support on the ground. 

The cooperation between DG REFORM and other Commission services was positively 
assessed  by  the  majority  of  interviewees.  In  general,  there  were  well-functioning 
formal  and  informal  communication  channels  between  DG  REFORM  units  and 
corresponding  Commission  services  working  in  specific  sectoral  areas.  More 
specifically,  all  the  parties  involved  considered  the  mechanism  of  consulting  other 
DGs  as  very  positive  and  productive  when  assessing  technical  support  requests. 
Some  of  the  stakeholders,  however,  indicated  short  cycles  of  evaluation  and  tight 
deadlines given for other DGs during this process, which added to their workload.  

Some interviewees from beneficiary and national coordinating authorities highlighted 
the  need  for  a  closer  involvement  of  beneficiary  authorities  into  the  selection  of 
technical  support  providers,  which  would  ensure  a  better  match  between  the 
contractor, the aims  of  the project and the expectations of beneficiaries. Regarding 
the  role  of  beneficiary  authorities,  the  national  ownership  and  commitment  for  the 
reforms  were  mentioned  as  key  factors  of  successful  project  implementation, 
allowing  to  make  timely  decisions,  facilitate  engagement  of  relevant  stakeholders 
and ensure smoother communication between the parties involved. It was noted that 
in  cases  without  an  effective  involvement  of  political  authorities,  the  projects 
required stronger involvement and steering by the contractor or policy officers of DG 
REFORM. 

Effectiveness of the SRSP 

113 

 
The data collected during the evaluation evidenced the mixed effects of the COVID-
19  crisis  on  the  implementation  of  SRSP-funded  projects  during  the  evaluation 
period. On the one hand, some public health restrictions prevented organising some 
of the  key project activities that previously worked as a platform for  building trust-
based  relationships and networks of stakeholders. Also, the pandemic caused some 
changes to the project  contracts as amendments to the budgets became necessary 
(e.g.  due  to  the  re-allocation  of  funds  from  missions  or  physical  events  to  other 
activities). On the other hand, a sharp worsening of the economic conditions created 
more awareness on the need  for reforms in individual EU Member States. Also, the 
crisis  could  have  even  increased  the  level  of  flexibility  and  made  some  project 
activities  more  inclusive  for  some  groups  of  stakeholders  due  the  organisation  of 
online or hybrid events.  

These  interviews  also  provided  some  evidence  on  the  key  factors  influencing  the 
success  of  SRSP-funded  projects  in  terms  of  achieving  the  project  objectives  and 
implementing reforms in Member States. Among the internal factors/mechanisms of 
change the following were identified by a number of stakeholders interviewed: 

-  Strong  leadership  and  ownership  by  beneficiary  authorities  of  the 
reforms  and  outputs  developed  during  the  technical  support  projects: 
leadership  and  pro-active  involvement  of  beneficiary  authorities  is 
essential for the project outputs and results to be used in policy decisions; 
-  Proper  cooperation  and  communication  between  different 
stakeholders involved  in  the project:  it  is  crucial  to  ensure  that  there 
is  constant  communication  and  exchange  of  feedback  between  all  the 
parties  involved  in  the  project  implementation,  e.g.  between  the 
beneficiary authority, the team of experts at national level and the support 
provider and is motivated. For this purpose it is important to develop and 
ensure  the  functioning  of  regular  and  effective  communication  channels  – 
e.g.  attending  common  meetings,  exchanging  comments,  co-designing  of 
project deliverables, setting common priorities and goals etc.; 

-  Adequate project design and planning: this includes development of a 
clear  strategy  of  embedding  the  outputs,  milestones  for  reforms,  also 
timing  of  the  key  project  deliverables  and  activities  while  considering  the 
budget and allocation of resources; 

-  Presence  of  a  well-balanced  project  team  within  the  beneficiary 
Member  States:  this  entails  a  project  team  that,  on  the  one  hand, 
involves  technocrats/experts  who  would  not  leave  their  positions  after 
changes  related  to  political  cycle.  This  helps  maintain  organisational 
continuity  and  expertise  inside  the  institution.  On  the  other  hand,  the 
project  team  should  also  include  adviser(s)  or  other  officers  close  to  the 
Minister’s office who would ensure political support for the reform.    

Furthermore,  the  evaluation  evidence  also  confirmed  some  of  the  key  external 
factors  that  influence  the  success  of  technical  support  projects  –  most  importantly, 
the  electoral  cycle  and  political  support  for  the  reforms  planned  and  designed  in 
these  projects.  A  technical  support  project  is  more  likely  to  be  successful  if  it 
addresses reforms or particular measures that are an integral part of a wider reform 
programme within a specific sector or when it addresses a specific EU policy priority 
in a particular sector. Since the political cycle might create incentives for reforms or 
might disrupt their implementation, the implementation of reforms depends much on 
the role of civil servants.  

EU value added of the SRSP 

The majority of participants of the interview programme referred to a few examples 
of the EU added value. Sharing of good practices, exchange of know-how and other 
activities  allowing  to  exploit  the  synergies  across  the  projects  were  highlighted. 

114 

 
Some  interviewees  even  indicated  that  learning  from  the  experience  of  other 
Member  States  significantly  improved  the  implementation  of  reforms  in  their 
institution  or  country.  Our  evaluation  results  suggest  that  there  were  no  significant 
differences in this regard between the annual and dedicated calls. However, the role 
of policy officers and technical support providers as facilitators of this exchange was 
stressed, especially due to the lack of communication platforms (e.g., Just Transition 
Platform)  and  limited  cooperation  experience  of  the  Member  States.  On  the  other 
hand,  some  of  our  interviewees  assessed  the  SRSP-funded  projects  as  facilitating 
cooperation  among  Member  States.  Key  factors  that  encouraged  this  process 
included  similarities  in  specific  country  challenges  and  national  priority  reforms  as 
well as incentivising role of the policy officers of DG REFORM.  

In  addition,  participants  of  case  study  interviews  were  rather  sceptical  about  the 
possibility of their institution or  country to implement foreseen  reforms without the 
technical support  of SRSP. The programme provided  valuable financial resources to 
achieve  high-level  expertise,  which  was  considered  less  likely  to  be  available  if  the 
project was funded and managed exclusively at the national level. Furthermore, the 
role  of  the  EC  as  a  legitimising  actor  was  mentioned.  It  was  especially  relevant  in 
countries  overcoming  political  turbulences  or  projects  focused  on  contested  topics, 
when the endorsement of the European Commission allowed to gain support as well 
as ensure involvement of the relevant stakeholders. 

1.4. 

Results of focus group discussions 

Implementation and participants of focus group discussions 

Two  focus  groups  were  carried  out  on  22  June  2022  (on  management  and 
implementation  of  the  SRSP)  and  27  June  2022  (on  the  management  and 
implementation of the dedicated call for the preparation of TJTPs).  

-  22  June:  focus  group  with  national  coordinating  authorities  (Cas).  Nine 
representatives  of  eight  national  coordinating  authorities  participated  in  the 
focus  group  (Belgium,  Estonia,  Spain,  Croatia,  Hungary,  Malta,  Portugal, 
Slovakia) in addition to the evaluation team. 

-  27 June: focus group with eight representatives of six beneficiary authorities 
(Bas) of projects under the dedicated call (Bulgaria, Greece, Czechia, Croatia, 
Latvia, Slovakia) in addition to the evaluation team. 

Both  focus  groups  shared  the  same  agenda,  including  introduction,  presentation  of 
emergent findings, structured discussion and its summary. The purpose of the focus 
groups  was  to  test  the  findings  emerging  from  the  research  with  a  cohort  of 
individuals directly involved in implementing the programme. The participants were 
asked  to  comment  on  the  validity  of  the  findings,  provide  examples  from  their 
experience and suggest recommendations for the future. 

Efficiency of the SRSP  

The majority of participants agreed that dedicated calls helped to address the needs 
of  specific  geographical  areas  and  achieve  a  better  understanding  of  new  funding 
instruments.  In  addition,  some  participants  emphasised  that  dedicated  calls  were 
very  helpful  when  focused  on  new  areas  where  Member  States  had  no  prior 
experience.  For  instance,  since  the  regulation  establishing  the  Just  Transition  Fund 
was  new  to  Member  States,  the  support  of  the  Commission  provided  under  the 
dedicated  call  on  the  preparation  of  TJTPs  was  key  to  understand  it.  This  was  also 
the case  with the dedicated  call  related to the implementation  of the  Recovery and 
Resilience Plans giving access to the Recovery and Resilience Facility.  

115 

 
Transparency  and  clarity  in  the  prioritisation  and  selection  of  technical 
support  requests  submitted  by  the  Member  Sates’  authorities  could  facilitate  the 
role  of  coordinating  authorities.  Member  States,  particularly  those  with  a 
decentralised  system  of  government,  have  experienced  challenges 
in  the 
prioritisation of technical support projects under SRSP. Coordinating authorities from 
these countries faced competing requests from different bodies both at national and 
regional levels and therefore needed to state the basis on which requests have been 
put  forward  to  the  Commission  and  accepted  by  it.  One  possible  solution  advised 
during  the  focus  group  discussion  was  to  have  a  regulatory  basis  for  the 
prioritisation.  This  could  also  encourage  regional  governments  of  Member  States  to 
join forces and collaborate on projects. 

Some  coordinating  authorities  have  created  specific  systems  to  prioritise  the 
technical  support  projects.  Such  systems  assess  the  projects  against  their 
compliance  and  coherence  with  the  EU  and  national  priorities  and  define  different 
priority categories. In addition, the coordinating authorities are also usually involved 
in the programming of other funds (e.g. Cohesion Policy programmes) and therefore 
have  a  strategic  overview  of  programming  priorities  for  the  country.  Sometimes, 
there are discrepancies between the priorities defined by the coordinating authorities 
and the technical support projects  selected by  the Commission and the justification 
provided  is  not  always  sufficient.  Reconciling  these  two  approaches  would  help 
programme implementation. 

Overall,  the  submission  and  selection  of  technical  support  requests  was 
viewed  positively  by  coordinating  authorities.  Simplifying  and  ensuring  more 
coherence  is  crucial  especially  for  coordinating  authorities  with  limited  resources 
dedicated  to  the  programme.  Coordinating  authorities  have  a  key  role  as  they 
facilitate cooperation with the beneficiary authorities and provide peer to peer review 
and support. Participants of the focus groups indicated some areas for improvement 
of  the  technical  support  that  beneficiaries  receive,  including  the  use  of  some  pre-
filled  forms  for  the  submission  of  requests  and  the  further  simplification  of  the 
current procedures. 

Coordinating  authorities  and  beneficiary  authorities  reported  a  high  level  of 
satisfaction  with  the  selected  technical  support  providers.  The  Commission  was 
responsible  for  the  selection  of  technical  support  providers  and  overall,  participants 
agreed that the contractors selected by the Commission had the right competencies. 
There  was  a  general  agreement  that  this  is  crucial,  as  beneficiary  authorities 
required the high-quality expertise. 

In  practice,  the  selected  international  experts  had  a  smaller  role  during  the  project 
implementation compared to what was initially foreseen in the project proposal. This 
might  have  been  caused  by  subsequent  economic  considerations  by  the  technical 
support  provider  or  by  a  wrong  interpretation  of  the  terms  of  reference.  Therefore, 
the  Commission  was  expected  to  maintain  oversight  over  the  selected  providers  in 
order to avoid subcontracting of certain services to third entities that are not able to 
provide an equivalent level of expertise and quality.  

On the other hand, there were  several advantages associated with the involvement 
of local experts. Above all, local experts can discuss technical aspects of projects in 
their  native  language.  The  involvement  of  local  experts  can  also  facilitate  the 
engagement  with  local  stakeholders.  However,  several  participants  argued  that 
having  national  experts  is  helpful  provided  that  they  are  connected  internationally 
and ensure access to international experience. 

Furthermore,  coordinating  authorities  reported  that  the  process  of  agreeing  the 
CSPs is straightforward, however, some challenges might arise from the non-binding 

116 

 
nature  of  these  Plans.  The  only  binding  document  is  the  contract  between  the 
Commission and the providers. 

Collaboration  between  the  Commission,  technical  support  providers  and 
beneficiary  authorities  was  reported  to  be  positive,  especially  during  the 
preparation  of  the  TJTPs.  Organising  ‘tripartite  meetings’  between  these  actors  was 
mentioned  as  key  to  effective  preparation  and  implementation  of  projects,  because 
of  input  and  views  provided  by  other  DGs  of  the  Commission  (in  case  of  the 
dedicated  call,  they  were  helpful  for  the  preparation  of  TJTPs).  Overall,  the 
Commission  supported  the  MSs  from  the  preparation  of  the  request  for  technical 
support, to the finalisation and submission of the same. 

Effectiveness of the SRSP  

Generally,  the  need  to  continuously  involve  beneficiary  authorities  in 
decision-making was stressed by the participants. Involving beneficiary authorities 
in the drafting of the terms of reference, especially as regards the description of the 
data collection activities, might help ensure that the work of the provider meets the 
expectations of the beneficiary authorities. The beneficiary authorities reported that 
the  preliminary  collection  of  data  and  statistics  was  useful  and  important  for  the 
preparation  of the TJTPs and the selection  of the projects that  should be part  of it. 
However,  some  beneficiary  authorities  shared  concerns  about  the  performance  of 
the  contractors  on  this  specific  activity.  In  some  cases,  the  technical  support 
providers did not conduct additional interviews or in-depth research to fill some data 
gaps.  One  factor  which  could  explain  this  misapprehension  –  according  to  a 
participant – is a wrong interpretation of the terms of reference.  

One  more  issue  raised  by  the  coordinating  authorities  was  the  possibility  for  the 
Commission to make changes to the project during implementation without sufficient 
involvement  and  consultation  with  the  beneficiary  authorities.  In  some  cases,  this 
approach  has  determined  a  slippage  in  the  timing  allocated  during  the  project 
implementation.  In  addition,  this  approach  might  reduce  the  value  of  the  technical 
support for the beneficiary authorities.  

Events can be a useful means by which beneficiary authorities can raise awareness 
among stakeholders about the results of their projects and also about the TJTP in 
general.  Some  beneficiary  authorities  reported  benefits  from  organising  workshops 
to  do  this.  Some  countries  adopted  a  more  structured  approach  to  stakeholder 
engagement  than  others.  For  instance,  one  participant  from  Greece  reported  that 
the contractor was responsible for identifying key stakeholders of the TJTP (including 
ministries  and  other  relevant  stakeholders).  The  beneficiary  authorities  set  up  a 
formal partnership with these stakeholders and organised a series of online meetings 
and  workshops  to  explain  the  rationale  and  value  of  this  plan  and  how  it  would 
benefit the different regions.  

Overall,  involving  a  broad  range  of  stakeholders  was  seen  as  key  to  ensure  the 
effectiveness of TJTP projects. In Croatia, for instance, an ad hoc team was set up at 
the central government level, but also regional stakeholders were involved in several 
aspects  of  this  process  (including  the  preparation  of  the  projects  and  overall  TJTP 
structure  and  priorities).  Private  entities  were  also  involved  in  working  groups 
dedicated to the preparation of TJTP projects. 

It is important to mention that the SRSP-funded projects were able to move much of 
their  activity  online  and  thus  minimise,  although  not  completely  overcome,  the 
adverse  impact  of  the  pandemic  on  their  implementation.  The  COVID-19 
pandemic  caused  a  shift  in  the  project  delivery  mode,  from  in  person  to  online  for 
most  activities  in  most  projects.  From  the  design  perspective,  most  of  the  projects 
were  not  affected.  In  the  implementation  phase,  instead,  the  ownership  of  the 

117 

 
projects  has  been  weakened  in  some  cases  by  the  restrictions  and  the  lack  of  in 
person  meetings.  This  particularly  affected  projects  which  foresaw  trainings  and 
conferences.  In  particular,  the  majority  of  the  events  were  held  online,  and  it  was 
not easy to get people  involved and  share data and information  effectively through 
virtual  communication  channels.  The  online  exchange  had  some  drawbacks 
compared  to  the  in-person  interaction  and  required  more  coordination  efforts. 
However,  coordinating  authorities  reported  that  technical  support  providers  had 
generally maintained equivalent levels of service to the extent possible.  

Measures for the improvement of the future programme 

Focus  group  participants  made  a  number  of  suggestions  for  measures  that  could 
improve the future programme. 

-  Multi-country  projects:  the  coordinating  authorities  were  supportive  of 
the  idea  of  multi-country  projects.  According  to  them,  the  Commission 
could  take  a  more  active  role  and  encourage  collaboration  between 
potential  beneficiary  authorities  in  different  Member  States  with  similar 
needs and priorities, in order to support the preparation of a joint request 
for  a  multi-country  project.  To  this  end,  the  Commission  could  set  up  a 
platform/website where beneficiary authorities could share information and 
experiences on different projects and learn from each other; 

- 

-  Preselected  pool  of  experts:  Member  States  could  benefit  from  being 
able  to  draw  on  a  pool  of  experts  to  address  urgent  technical  support 
needs; 
Involving 
the  project 
implementation:  some  participants  of  the  focus  group  emphasised  that, 
based  on  their  experience,  involving  regional  and  local  authorities  as  part 
of  Steering  Committees  and  in  the  exchanges  with  the  technical  support 
providers was beneficial for the implementation of projects; 

local  authorities 

regional  and 

in 

- 

-  More  dedicated  calls  on  specific  themes:  the  coordinating  authorities 
reported  that  the  design  of  dedicated  calls  is  beneficial  to  help  Member 
States  address  their  new  or  evolving  needs.  Such  calls  could  be  launched 
as and when it is needed to address the urgent needs or priorities similar 
across the Member States; 
Increased  flexibility  in  implementation:  beneficiary  authorities  would 
benefit from the possibility to make changes during the implementation of 
technical support projects to address the major  changes in their needs or 
respond  to  changing  circumstances.  At  the  same  time,  the  need  for  a 
possibility  of  project  duration  extension  was  expressed.  With  hindsight, 
this  might  have  been  beneficial  when  the  JTF  regulation  was  introduced 
and  some  of  the  projects  under  the  dedicated  call  have  already  been 
finalised; 

-  Collaboration  and  knowledge  exchange  between  the  projects:  the 
coordinating authorities can play a key role in facilitating this collaboration 
and  sharing  of  knowledge.  Beneficiary  authorities  highlighted  benefits 
provided  by  annual  meetings  that  were  organised  by  the  coordinating 
authorities to encourage mutual learning and informal exchanges between 
SRSP-funded projects. 

Project status, activities and delivery modes of the programme 

Out  of  826  projects  included  in  the  scope  of  this  evaluation,  in  terms  of  the 
circumstances of the selected requests, most of them were related to implementing 
challenges  identified  in  the  context  of  economic  governance  processes,  with  the 
gradual increase of their share from 51% in 2017 to 62% in 2019, followed by some 
decrease of their share to 58% in 2020. 

118 

 
 
Table  15.  Circumstances  of  the  selected  requests  under  the  SRSP,  2017-
2020 

CIRCUMSTANCES  OF 
SELECTED REQUESTS 

THE 

governance 
Economic 
process, including CSRs 

SRSP 2017 

SRSP 2018 

SRSP 2019 

SRSP 2020 

51% 

55% 

62% 

58% 

EU priorities 

27% 

29% 

21% 

30% 

EU law 

Member  States’  own 
initiative 

7% 

11% 

adjustment 

Economic 
programmes 
Source: SRSP annual monitoring reports 2017-2020. 

4% 

6% 

7% 

3% 

5% 

8% 

4% 

2% 

5% 

5% 

Overall,  there  were  1,352  technical  support  activities  planned  for  implementation 
under the four rounds of the SRSP (in 2017, 2018, 2018 and 2020). The  number of 
annual  activities  increased  from  281  under  the  2017  round  to  339  under  the  2018 
round  and  to  432  under  the  2019  round,  with  some  decrease  in  the  volume  of 
activities  (to  316)  under  the  2020  round  due  to  the  selection  of  fewer  larger 
projects. Similarly, the average number of activities per project increased from 2017 
to 2020 and this increase continued across all the rounds – around 2.4, 2.8, 3.1 and 
3.2 activities per project respectively in each round. 

The monitoring system of DG REFORM contains 12 eligible activities of the SRSP. The 
analysis  of  the  SRSP  monitoring  data  indicates  that  the  programme  in  2017-2020 
most  often  planned  the  following  activities:  study,  research,  evaluation  (313  such 
activities  out  of  826  projects)  and  workshops,  conferences  and  seminars  (270  such 
activities).  Long  term  expert  mission(s)  (26  activities)  and  organisation  of  local 
operations  (27  activities)  were  the  least  frequently  used  activities  in  the  technical 
support projects under the 2017-2020 rounds.  

119 

 
FIGURE 30: NUMBER OF ACTIVITIES PLANNED FOR THE IMPLEMENTATION OF THE SRSP, 2017-
2020 

313

270

194

147

142

350

300

250

200

150

100

50

0

70

65

60

38

27

26

Source:  DG  REFORM  internal  monitoring  system  (N=1,352).  Note:  the  categories  of  ‘other  activities’  and 
‘none of the activities’ were excluded from the analysis. 

The  technical  support  projects  in  2017-2020  often  planned  more  than  one  specific 
activity. A total of 24% of projects used only one activity, 21% of projects used two 
different  activities,  22%  of  projects  –  three  activities,  18%  of  projects  –  four 
activities,  and  15%  of  projects  –  five  or  more  different  activities.  The  number  of 
projects that planned the implementation of one specific activity decreased gradually 
over  the  annual  rounds,  while  the  number  of  projects  having  four,  five  or  more 
activities  increased  substantially  from  2017  to  2019.  If  the  most  common  SRSP-
funded  project  in  2017  often  involved  a  single  activity  (usually  ‘study,  research, 
evaluation’  or 
(e.g. 
‘workshop/conference/seminar’ and ‘study, research, evaluation’), the most common 
projects  of  the  2020  round  combined  three  different  activities  besides  the  common 
activity  of  ‘study,  research  and  evaluation’.  Also,  the  2020  projects  became  more 
complex  in  terms  of  outputs  most  frequently  involving  the  combinations  of  3-4 
outputs (compared to 1-3 outputs in the 2017 projects) (see the table below).  

‘study  visit’)  or  a  combination  of 

two  activities 

Therefore,  the  increasing  number  of  activities  and  their  larger  combinations  per 
project  show  that  the  technical  support  projects  became  more  complex  over 
time. This is because the average size of the budget increased from the 2017 round 
to the 2020 round and, based on the interview data,  the technical support projects 
the 
became  more  comprehensive  and  ambitious 
implementation of the SRSP. Also, the 2019 and 2020 rounds of the programme saw 
an  increase  in  the  volume  of  such  operational  activities  as  ‘collection  of  statistics’, 
‘communication activities’ or ‘compilation/dissemination of material’. In contrast, the 
frequency of such  specific capacity-building activities as ‘study visits’  or  ‘short-term 
expert  mission(s)’  remained  at  the  similar  levels  in  the  2019  and  2020  rounds 
compared  to  the  previous  rounds.97  The  growing  complexity  of  technical  support 

their  design  during 

in 

97 It is possible that the providers of technical support (both under public procurement and grants) more 
frequently  contracted  the  services  of  individual  experts  in  the  2019  and  2020  rounds,  making  the 
execution  of  such  individual  activities  as  ‘short-term  expert  mission(s)’  less  clearly  visible  in  the 

120 

 
 
 
projects  combined  with  an  increased  focus  on  operational  activities  in  their  design 
can  make  the  implementation  of  these  projects  more  difficult  in  the  future,  despite 
their possible relevance to the needs of Member States and beneficiary authorities. 

Table 16. the most COMMON SRSP-funded projects in terms of activities and 
outputs during 2017-2020 

MOST COMMON 
SRSP-FUNDED 
PROJECTS IN 
2017 

MOST COMMON 
SRSP-FUNDED 
PROJECTS IN 
2018 

MOST COMMON 
SRSP-FUNDED 
PROJECTS IN 2019 

MOST COMMON 
SRSP-FUNDED 
PROJECTS IN 2020 

Activities 

Study, research, 
evaluation (7 
unique cases) 
Study visit (7 
unique cases) 
Workshop/confer
ence/seminar; 
Study, research, 
evaluation (6 
unique 
combinations) 

Short-term expert 
mission; Study, 
research, 
evaluation (7 
unique 
combinations) 
Study, research, 
evaluation (5 
unique cases) 

Outputs 

Recommendation
; Analysis, report 
(8 unique 
combinations) 
Recommendation 
(6 unique cases) 
Action plan, 
Roadmap; 
Recommendation
; Analysis, report 
(6 unique 
combinations) 

Action plan, 
Roadmap; 
Recommendation; 
Analysis, report 
(8 unique 
combinations) 
Action plan, 
Roadmap; 
Recommendation; 
Analysis, report 
(7 unique 
combinations) 

Study, research, 
evaluation (10 unique 
cases) 
Workshop/conference/s
eminar; Study, 
research, evaluation (9 
unique combinations) 
Short-term expert 
mission; 
Workshop/conference/s
eminar; Training 
activity; Study, 
research, evaluation (5 
unique combinations) 
Recommendation; 
Analysis, report (14 
unique combinations) 
Action plan, Roadmap; 
Recommendation; 
Analysis, report; 
Workshops (10 unique 
combinations) 
Recommendation; 
Analysis, report; 
Workshops (8 unique 
combinations) 
Action plan, Roadmap; 
Guidelines, Handbook; 
Recommendation; 
Analysis, report; 
Workshops (6 unique 
combinations) 

Study, research, 
evaluation (7 unique 
cases) 
Short-term expert 
mission; 
workshop/conference/s
eminar; study, 
research, evaluation (5 
unique combinations) 
Workshop/conference/s
eminar; training 
activity; study, 
research, evaluation (5 
unique combinations) 

Recommendation; 
Analysis, report; 
Workshops (7 unique 
combinations) 
Action plan, Roadmap; 
Recommendation; 
Analysis, report; 
Workshops (7 unique 
combinations) 
Action plan, Roadmap; 
Guidelines, Handbook; 
Recommendation; 
Analysis, report; 
Workshops (7 unique 
combinations) 

Source: DG REFORM internal monitoring system (N=1,352).  
Note:  unique  cases  and  combinations  refer  to  the  number  of  unique  (the  only  one  of  its  kind)  activities 
and  outputs  found  in  the  design  of  technical  support  projects.  For  instance,  a  total  of  6  SRSP-funded 
projects  only  combined  the  specific  activities  of  ‘workshop/conference/seminar’  and  ‘study,  research, 
evaluation’ in the 2017 round (with other 2017 projects having other types of activities in their design). 

Furthermore,  several  delivery  modes  were  used  during  the  implementation  of  the 
SRSP.  According  to  the  SRSP  Regulation,  the  specific  mode  of  implementation  is 
done either directly by the Commission or indirectly in accordance with the Financial 
Regulation98. Direct management includes public procurement contracts, grants and 
reimbursement of costs incurred by external experts. Fiscalis is also implemented in 
direct  management  by  the  Commission  (Directorate-General  for  Taxation  and 
Customs  Union,  DG  TAXUD).  Entities,  such  as  international  organisations,  which 

design of technical support projects. This issue could be assessed during the evaluation of the TSI in 
the future.   

98 Regulation (EU, Euratom) 2018/1046 of the European Parliament and of the Council of 18 July 2018 on 
the  financial  rules  applicable  to  the  general  budget  of  the  Union,  amending  Regulations  (EU)  No 
1296/2013, (EU) No 1301/2013, (EU) No 1303/2013, (EU) No 1304/2013, (EU) No 1309/2013, (EU) 
No 1316/2013,  (EU)  No 223/2014,  (EU)  No 283/2014,  and Decision  No 541/2014/EU  and repealing 
Regulation (EU, Euratom) No 966/2012. Official Journal of the European Union, L 193/1, 30.7.2018. 
http://data.europa.eu/eli/reg/2018/1046/oj 

121 

 
 
 
 
have  undergone  an  ex-ante  assessment  of  their  procedures  and  systems, 
guaranteeing  a  level  of  protection  of  the  EU  financial  interests  equivalent  to  that 
guaranteed  by  the  Commission  in  direct  management,  can  act  under  indirect 
management  for  the  tasks  they  undertake  to  implement  the  actions.  Technical 
Assistance and Information Exchange instrument (TAIEX), which provides short-term 
expertise  exchange  by  public  sector  experts  from  the  Member  States,  is  not 
contracted but utilised through a Service Level Agreement between DG REFORM and 
Directorate-General for Neighbourhood and Enlargement Negotiations (DG NEAR).99  

In total, there were 1,037 individual delivery modes applied during the programme’s 
implementation  in  2017-2020.  Procurement  (416  cases;  mostly  implemented  by 
private  companies)  and  grants  (332;  often  awarded  to  international  organisations) 
were most often used to provide technical support to EU Member States. A technical 
support project can consist of a mix of different delivery modes to tailor the support 
to  the  needs  of  the  Member  State.  A  total  of  78%  of  projects  used  one  specific 
delivery mode, with 19% of projects applying two different delivery modes, and 3% 
of projects applying three or more different delivery modes. In addition, a total of 9 
multi-country projects were implemented during the 2017-2020 rounds.100 

FIGURE 31: USE OF MAIN DELIVERY MODES DURING THE IMPLEMENTATION OF THE SRSP, 2017-
2020 

40%

32%

17%

45%

40%

35%

30%

25%

20%

15%

10%

5%

0%

5%

3%

2%

1%

Procurement
(N=416)

Grants
(N=332)

TAIEX
(N=171)

Delegation
(N=54)

Fiscalis
(N=30)

Commission
expertise
(N=25)

Multicountry
projects
(N=9)

Source: DG REFORM internal monitoring system (N=1,037).  

A substantial increase in the budget of the programme in 2019 and 2020 produced 
more procurements whose number rose from 80 in 2018 to 136 in 2019 and to 131 
in  2020.  For  comparison,  the  relative  share  of  this  delivery  mode  (out  of  all  the 
delivery modes that year) was 37% in the 2017 round, 34% in 2018, 43% in 2019 
and 43% in 2020. The delivery mode of Fiscalis was applied only under the 2017 and 
2018  rounds  (15  times  each  cycle)  in  the  sectoral  category  of  revenue 
administration and public financial management. 

99  European  Commission  (2020),  Commission  Staff  working  document:  Mid-term  evaluation  of  the 
Programme. 

Structural 
https://ec.europa.eu/info/sites/default/files/report_en_final.pdf 

Support 

Reform 

100  Since  the  SRSP  Regulation  did  not  explicitly  provide  for  the  possibility  of  submitting  multi-country 
requests from EU Member States, this way of delivering technical support was underdeveloped during 
the period 2017-2020.  

122 

 
 
 
Evidence  from  the  interviews  indicates  that  the  selection  of  an  individual  delivery 
mode or a support provider depends on the needs of Member States and the nature 
of activities/outputs planned in each project. In cases when project activities require 
significant operational capacities (many experts and other resources on the ground) 
in specific countries, close contacts with national authorities and good knowledge of 
the  national  language,  the  framework  contract  involving  the  consortia  of  private 
sector  companies  is  often  used  for  the  delivery  of  technical  support.  In  contrast, 
when  an  exchange  of  international  good  practice  and  a  transfer  of  knowledge  from 
other countries is key to achieving the project objectives, international organisations 
are frequently selected as providers of technical support.  

2.1.  Main outputs and results achieved during the implementation of the 

programme  

The monitoring system of DG REFORM contains a list of outputs whose achievement 
is  planned  during  the  execution  of  technical  support  projects.  In  total,  the 
programme  was  expected  to  produce  1,462  outputs  in  2017-2020.  The  number  of 
outputs per annual round grew from 300 (under the 2017 round) to  344 (under the 
2018  round)  and  to  470  (under  the  2019  round),  with  a  decrease  to  354  outputs 
(coinciding with a similar decrease of activities) under the 2020 round. 

(i) 

‘study, 

research 

The  analysis  of  the  SRSP  monitoring  data  indicates  that  the  programme  in  2017-
2020  was  most  often  planned  to  produce  ‘recommendations’  (334  times),  ‘analysis 
and  reports’  (330  times),  ‘workshops’  (234  times),  as  well  as  ‘action  plans  and 
roadmaps’  (207  times)  as  a  result  of  implementing  different  technical  support 
activities.  The  outputs  of  ‘recommendations’  and  ‘analysis  and  reports’  were  most 
frequently  expected  to  be  produced  as  a  result  of  implementing  the  following  two 
activities: 
(ii) 
‘workshop/conference/seminar’. The activity of ‘study, research and evaluation’ was 
also  often  expected  to  lead  to  the  production  of  ‘action  plans  and  roadmaps’.  The 
least  expected  output  in  2017-2020  was  the  ‘production  of  terms  of  reference’  (16 
times).  Similar  types  of  outputs  were  found  to  be  most  frequently  produced  during 
the mid-term evaluation of the 2017-2018 SRSP in terms of design of action plans, 
roadmaps,  guidelines,  recommendations  and  strategies  (88  times),  seminars, 
conferences and workshops (69 times) and  reports (including working visit reports) 
(64  times).101  The  biggest  change  in  the  number  of  planned  outputs  occurred  in 
‘communication  strategy  and  other  events’  whose  volume  tripled  from  6  such 
outputs in each 2017 and 2018 round to 18 and 15 such outputs under the 2019 and 
2020 rounds respectively.  

evaluation’ 

and 

and 

The technical support projects were often expected to achieve more than one output 
in 2017-2020. In total, 20%  of projects were  expected to achieve  only  one output, 
19% of projects – two outputs, 26% of projects – three outputs, 16% of projects – 
four  outputs,  and  19%  of  projects  –  five  or  more  different  outputs.  The  number  of 
projects  that  were  expected  to  produce  one  output  gradually  decreased  over  the 
annual  rounds,  while  the  number  of  projects  expected  to  achieve  five  or  more 
outputs  substantially  increased  in  2019  and  exceeded  other  categories  of  projects 
according  to  their  outputs  in  2020.  Similarly  to  the  number  of  technical  support 
activities  (see  section  1.4  of  this  annex),  this  suggests  an  increase  in  the 
complexity  and  significance  of  projects  in  the  SRSP’s  implementation  over 
time  as  they  started  addressing  more  important  reform  needs  of  the  beneficiary 
Member  States  and  they  were  expected  to  produce  more  value  to  the  formulation 
and  implementation  of  reforms  with  the  availability  of  more  financial  resources  per 

101 Ernst & Young (2018). Mid-term Evaluation of the Structural Reform Support Programme (SRSP) 2017-
the  European  Union,  p.  96,  158. 

2020  (2016  ECFIN  009/A).  Publications  Office  of 
https://doi.org/10.2887/656262 This analysis was based on the results of a survey.  

123 

 
 
project.  According  to  the  results  of  the  interviews,  the  financial  size  of  the  projects 
was  also  matched  by  a  substantial  improvement  in  the  quality  of  requests  for 
technical  support  from  EU  Member  States,  which  was  illustrated  by  the  share  of 
high-score requests among the selected requests. 

FIGURE 32:TOTAL EXPECTED OUTPUTS OF THE SRSP, 2017-2020 

Terms of reference

16

Legislative proposal

Communication strategy (including material) and other
events

39

45

Study visit reports

110

Guidelines, handbook

Action plan, roadmap

Workshops (including material), training, training
material

Analysis, report

Recommendation

153

207

234

330

334

0

50

100 150 200 250 300 350 400

Source:  DG  REFORM  internal  monitoring  system  (N=1,462).  Note:  the  categories  of  ‘other  outputs’  and 
‘none of the outputs’ were excluded from the analysis. 

Furthermore,  the  monitoring  system  of  DG  REFORM  contained  the  four  groups  of 
pre-defined  outcomes  (results).  The  analysis  of  the  SRSP  monitoring  data  indicates 
that  the  programme  in  2017-2020  most  often  supported  the  achievement  of  the 
following  outcomes:  ‘adoption  of  (new)  procedures  and  actions  to  enhance  the 
implementation of reforms’ (447 times) and ‘improved internal working procedures, 
methodologies  and  processes,  organisation’  (447  times).  The  outcome  of 
‘organisational 
resource 
management’  received  somewhat  less  technical  support  from  the  programme  (119 
times).  A  similar  trend  was  observed  during  the  mid-term  evaluation  of  the  2017-
2018  programme,  which  found  ‘adoption  of  new  internal  working  procedures, 
methodologies and processes’ (29 times) to be the most frequently sought outcome, 
with  the  outcome  of  ‘improving  the  efficiency  and  effectiveness  of  human-resource 
management’ (9 times) sought least frequently.102 

change  management, 

improved  human 

change, 

(2016  ECFIN  009/A).  Publications  Office  of 

102 Ernst & Young (2018). Mid-term Evaluation of the Structural Reform Support Programme (SRSP) 2017-
2020 
the  European  Union,  p.  159. 
https://doi.org/10.2887/656262.  Since  this  analysis  was  based  on  the  results  of  a  survey,  the 
formulations  of  the  expected  outcomes  do  not  fully  correspond  to  those  provided  in  the  monitoring 
system of DG REFORM.  

124 

 
 
 
FIGURE 33: BREAKDOWN OF THE PRE-DEFINED OUTCOMES IN THE SRSP, 2017-2020 

447

447

500
450
400
350
300
250
200
150
100
50
0

267

119

Adoption of (new)
procedures and actions
to enhance the
implementation of
reforms

Improved internal
working procedures,
methodologies and
processes, organisation

Adoption of a strategy, a
new law /act or
modification of an
existing one

Organisational change,
change management,
improved human
resource management

Source: DG REFORM internal monitoring system (N=1,338). 

In total, the number of expected outcomes grew from 193 under the 2017 round to 
417 under the 2020  round. The number of expected outcomes increased in all pre-
defined  groups  of  outcomes  over  time,  except  for  the  outcomes  related  to 
resource 
‘organisational 
management’  whose  volume  somewhat  decreased  from  2019  to  2020  (from  41  to 
33). The technical support projects often had more than one pre-defined outcome. A 
total  of  49%  of  projects  pursued  one  pre-defined  outcome,  32%  of  projects  –  two 
pre-defined outcomes, 11% of projects – three outcomes, and only 2% of projects – 
four outcomes. 

change  management, 

improved  human 

change, 

More  technical  support  was  provided  to  achieving  the  outcome  of  ‘organisational 
change,  change  management,  improved  human  resource  management’  in  the 
sectoral field of governance and public administration. The ‘adoption of a strategy or 
a  new  law’  was  more  frequent  in  the  sectoral  categories  of  sustainable  growth  and 
business  environment,  as  well  as  financial  sector  and  access  to  finance,  which  is 
expected,  given  that  these  projects  were  related  to  public  interventions  into  the 
private sector. 

Finally,  there  were  some  differences  across  the  pre-defined  outcomes  in  terms  of 
their links to the specific objectives of the SRSP. For instance, technical support was 
more often provided to the ‘adoption of internal working procedures, methodologies 
and  processes’  when  these  projects  were  contributing  to  the  specific  objective  of 
‘defining  and  implementing  processes  and  methodologies’  (objective  3).  Also,  the 
‘adoption of a strategy, a new law/act or modification of an existing one’ was more 
likely when the projects were pursuing specific objectives 1 and 2 related to ‘design 
of reforms’ or ‘enhancing capacity to implement reforms’. The ‘adoption of improved 
internal  working  procedures,  methodologies  and  processes,  organisation’  was 
frequently  associated  with  ‘the  production  of  guidelines  and  handbooks’  and  the 
‘delivery of training activities’ during the technical support projects.  

125 

 
 
FIGURE 34: BREAKDOWN OF GROUPS OF OUTCOMES ACCORDING TO LEAD UNITS, 2017-2020 

140

120

100

80

60

40

20

0

3

4

1

2

01 - Horizontal
Policies

104

85

89

65

133

90

91

113

104

25

17

33

24

32

56

21

70

56

48

14

B1 - Revenue
Administration &
Public Financial
Management

B2 -
Governance &
Public
Administration

B3 - Sustainable
Growth &
Business
Environment

B4 - Labour
Market,
Education,
Health & Social
Service

B5 - Financial
Sector & Access
to Finance

Adoption of (new) procedures and actions to enhance the implementation of reforms

Improved internal working procedures, methodologies and processes, organisation

Adoption of a strategy, a new law /act or modification of an existing one

Organisational change, change management, improved human resource management

Source:  DG  REFORM  internal monitoring  system  (N=1,280).  Note:  the  categories  of  ‘other  outcomes’  and 
‘none of the outcomes’ were excluded from the analysis.  

126