COMMISSION STAFF WORKING DOCUMENT MID-TERM EVALUATION of the Regulation (EU) No 250/2014 of the European Parliament and of the Council of 26 February 2014 establishing a programme to promote activities in the field of the protection of the financial interests of the European Union (Hercule III programme) and repealing Decision No 804/2004/EC Accompanying the document Commission report to the European Parliament and the Council
Date: 2018-01-10
Institutions: European Anti-Fraud Office, European Commission
Subjects: EU financing, EU programme, cross-border cooperation, economic offence, financial control, fraud, fraud against the EU, general budget (EU), illicit trade

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 10.1.2018  SWD(2018) 3 final  COMMISSION STAFF WORKING DOCUMENT MID-TERM EVALUATION of the Regulation (EU) No 250/2014 of the European Parliament and of the Council of 26 February 2014 establishing a programme to promote activities in the field of the protection of the financial interests of the European Union (Hercule III programme) and repealing Decision No 804/2004/EC Accompanying the document Commission report to the European Parliament and the Council {COM(2018) 3 final}  
 
 
Table of contents 

1. 

2. 

3. 

INTRODUCTION ................................................................................................................................ 3 

BACKGROUND TO THE INTERVENTION ..................................................................................... 4 

IMPLEMENTATION / STATE OF PLAY .......................................................................................... 9 

4.  METHOD ........................................................................................................................................... 10 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS .......................................... 13 

6. 

CONCLUSIONS ................................................................................................................................. 29 

ANNEX 1: PROCEDURAL INFORMATION ............................................................................................ 32 

ANNEX 2: SYNOPSIS REPORT OF STAKEHOLDER CONSULTATION IN THE CONTEXT 

OF THE STUDY ................................................................................................................................ 34 

ANNEX 3: METHODS AND ANALYTICAL MODELS .......................................................................... 42 

2 

 
 
1. 

INTRODUCTION 

Purpose and scope 

This report is a Commission Staff Working document (SWD) that accompanies the mid-
term  evaluation  of  the  Hercule  III  programme.  The  evaluation  of  the  Commission  was 
supported by an external study1. 

The mid-term evaluation is required by Article 13, paragraph 2(a) of the regulation that 
establishes  the  Hercule  III  programme2  (‘the  Regulation')  which  provides  that  “The 
Commission shall carry out a thorough evaluation of the programme and present to the 
European Parliament and to the Council by 31 December 2017, an independent mid-term 
evaluation  report  on  the  achievement  of  the  objectives  of  all  the  actions,  results  and 
impacts,  the  effectiveness  and  efficiency  of  the use  of  resources  and  its added  value  to 
the  Union,  in  view  of  a  decision  on  the  renewal,  modification  or  suspension  of  the 
actions;  the  mid-term  evaluation  report  shall  additionally  address  the  scope  for 
simplification,  internal  and  external  coherence  of  the  programme,  the  continued 
relevance of all objectives of the programme, as well as the contribution of the actions to 
the  Union’s  priorities  of  smart,  sustainable  and  inclusive  growth;  it  shall  also  take  into 
account  evaluation  results  on  the  achievements  of  the  objectives  of  the  Hercule  II 
programme". 

The  scope  of  the  mid-term  evaluation,  as  set  out  in  the  evaluation  roadmap3,  is 
represented  by  all  interventions  supported  under  the  Hercule  III  programme  up  to  June 
2017  as  well  as  the  preparatory  and  implementing  activities  undertaken  by  the 
stakeholders  for  these  interventions.  It  also  covers  those  applications  submitted  by 
potential  beneficiaries  which  remained  unsuccessful.  The  evaluation  assesses  six 
evaluation  criteria  –  relevance  of  all  the  objectives  of  the  programme;  internal  and 
external  coherence  of  the  programme;  effectiveness  in  attaining  it  objectives  and 
expected results; the efficiency of the use of resources; the programme's added value to 
the EU; and sustainability i.e. the effects of Hercule III actions after their completion, as 
well as the contribution of the actions to the Union’s priorities of smart, sustainable and 
inclusive growth. 

1   Mid-term Evaluation of the Hercule III programme, Final Report, CEPS, Economisti Associati, CASE, 

wedoIT, 2017 (hereafter "external study"). 
https://ec.europa.eu/anti-fraud/sites/antifraud/files/herculeiii_midterm_evaluation_en.pdf  

2   Regulation  (EU)  No 250/2014  of  the  European  Parliament  and  of  the  Council  of  26 February  2014 
establishing a programme to promote activities in the field of the protection of the financial interests of 
the  European  Union  (Hercule  III  programme)  and  repealing  Decision  No 804/2004/EC,  OJ  L  84  of 
20 March 2014. 

3   The roadmap for the mid-term evaluation of Hercule III was open to stakeholders' feedback during the 
http://ec.europa.eu/smart-

life 
regulation/roadmaps/docs/2017_olaf_002_midterm_evaluation_en.pdf 

following  website: 

evaluation 

time 

the 

the 

on 

of 

3 

 
 
                                                            
The time period covered by the evaluation is the first half of the seven year period that 
Hercule III programme lasts for, starting on 1 January 2014 (when the Regulation entered 
into force) until June 2017. 

2.  BACKGROUND TO THE INTERVENTION 

2.1 Description of the intervention  

The  Hercule  programme  is  an  instrument  specifically  dedicated  to  supporting  the 
protection  of  the  financial  interests  of  the  European  Union  by  fighting  irregularities, 
fraud  and  corruption.  It  is  administered  by  the  European  Anti-Fraud  Office  (OLAF) 
within the Commission. The first Hercule programme4 was established in 2004 and was 
succeeded by Hercule II (2007-2013)5.  

The current Hercule III programme (2014-2020) was adopted in 2014 on the basis of a 
proposal tabled by the Commission in 2011.  

The  regular  reporting  by  Member  States6  on  irregularities  and  suspected  fraud  cases, 
detected  during  the  implementation  of  the  budget  on  the  basis  of  Article  325  TFEU, 
shows  that  the  financial  impact  throughout  the  years  remains  at  levels  that  require  the 
Union to keep developing activities in order to strengthen the protection of the Union's 
financial interests.  

The general objective of the Hercule III programme is 'to protect the financial interests of 
the Union thus enhancing the competitiveness of the Union's economy and ensuring the 
protection  of  taxpayers'  money'7,  whereas  the  programme's  specific  objective  is  'to 
prevent  and  combat  fraud,  corruption  and  any  other  illegal  activities  affecting  the 
financial  interests  of  the  Union'8.  In  addition,  the  programme  lists  five  operational 
objectives centred around: 

facilitating the exchange of information, experiences and best practices; 

  strengthening cross-border and multi-disciplinary cooperation;  
 
  providing technical and operational support to national authorities;  
 

fighting  specifically  organised  fraud,  especially  cigarette  smuggling  and 
counterfeiting;  

4   Decision 804/2004/EC of the European Parliament an of the Council of 21 April 2004, establishing a 
Community action programme to promote activities in the field of the protection of the Community's 
financial interests (Hercule programme), OJ L 143 of 30 April 2004. 

5   Decision No 878/2007/EC of the European Parliament and of the Council of 23 July 2007 amending 
and  extending  Decision  No  804/2004/EC  establishing  a  Community  action  programme  to  promote 
activities in the field of the protection of the Community's financial interests (Hercule II programme), 
OJ L 193 of 25 July 2007. 

6   The annual reports on the protection of the Union's financial interests Article 325 reports) are available 

at: https://ec.europa.eu/anti-fraud/reports_en.  

7   Article 3 of Regulation (EU) No 250/2014. 
8   Article 4 of Regulation (EU) No 250/2014. 

4 

 
 
                                                            
  promoting comparative law analysis9. 

The Commission manages  a budgetary envelope  of EUR 104.9  million over the period 
2014-2020  for  the  Hercule  III  programme,  with  an  annual  average  of  around  EUR  15 
million.  The  eligible  beneficiaries10  of  the  programme  are  national  or  regional 
administrations  of  Member  States and  research bodies,  as well  as  educational institutes 
and non-profit-making entities that have been established and operate in a Member State 
for at least a year. The programme provides financial support for three types of actions11: 

1. "Technical Assistance"12 (at least 70% of the programme's budget): support aimed at 
providing  specific  knowledge,  equipment  and  information  technology  tools  to  national 
authorities  as  well  as  providing  specific  databases  and  IT  tools  facilitating  data  access 
and analysis.  

2. "Training" (maximum 25% of the programme's budget): support to the organisation of 
targeted  specialised  training,  risk  analysis  workshops,  conferences  and  legal  studies 
focused on the protection of the Union's financial interests.  

3. "Any other action" (not more than 5% of the budget). 

The programme is implemented by means of grants following calls for proposals, public 
procurement  contracts  following  calls  for  tender  and  administrative  arrangements 
concluded  with  the  Joint  Research  Centre  (JRC)  for  the  development  of  specific  tools. 
The programme also provides for the reimbursement of costs incurred by representatives 
from  a  limited  number  of  non-EU  countries  participating  in  training  actions  such  as 
conferences  or  seminars.  The  co-financing  rate  for  grants  does  in  principle  not  exceed 
80% of eligible costs13. 

2.2 Intervention logic 

Annual  Work  Programmes  (AWP)  ensure  that  the  general,  specific  and  operational 
objectives of the Hercule III programme are implemented in a consistent manner. They 
outline the expected results, the methods of implementation and their total amount. With 
regard to grants, the AWPs include the actions financed, the selection and award criteria 
and the maximum co-financing rate. 

AWPs  are  adopted  by  a  Financing  Decision  of  the  Commission.  The  preparation  of  an 
AWP involves a consultation of the main stakeholders within OLAF to ensure that new 
requirements of OLAF's partners are taken into account. A draft version of the AWP is 

9   Article 5 of Regulation (EU) No 250/2014. 
10   Article 6 of Regulation (EU) No 250/2014. 
11   Article 8 of Regulation (EU) No 250/2014. 
12   The  term  ‘technical  assistance’  in  the  context  of  the  Hercule  III  programme  is  construed  differently 
from  the  definition  of Article  186 of  Commission  Delegated  Regulation  2012/1268, as  amended, on 
the Rules of Application of Regulation 966/2012 of 29 October 2012. 
13   It may be increased to up to 90% in exceptional and duly justified cases. 

5 

 
 
                                                            
subsequently sent to the members of the COCOLAF14 and the AFCOS15, for consultation 
and  additional  observations.  On  the  basis  of  these  consultations,  modifications  may  be 
made.  The  next  step  in  the  adoption  of  the  AWP  is  an  internal  consultation  within  the 
Commission in order to ensure that the proposed activities do not overlap with activities 
funded  under  other  Union  programmes,  such  as  Customs  2020  or  the  Internal  Security 
Fund (ISF) activities. 

The  Commission  starts the  implementation  by  preparing  the  call  for  proposals (e.g. for 
technical assistance, conferences, legal training and studies) once the Financing Decision 
is adopted and the appropriate financial resources are made available. Subsequently, the 
submitted  proposals  are  evaluated  according  to  the  selection  and  award  criteria.  The 
successful proposals receive appropriate financial support for the relevant actions. 

OLAF submits an annual Implementation Report to the Parliament and the Council with 
the  main  results  achieved  and  the  relevant  insights  in  terms  of  consistency  and 
complementarity with other EU programmes, as required by Article 13(1). This report is 
annexed as a Commission Staff Working Document to the annual Article 325 report. The 
Committee on Budgetary Control (CONT) of the European Parliament drafts its opinion 
on  the  Article  325  report  by  the  end  of  the  year  and  the  Commission  takes  account  of 
these observations in the elaboration of the next AWP.  

Within OLAF, the programme is managed by a small sector and staff in the budget Unit 
(8 FTE in total). 

14   Advisory  Committee  for  the  Coordination  of  Fraud  Prevention  (COCOLAF)  -  aims  to  build 
cooperation  between  EU  countries  and  the  European  Commission  to  prevent  and  ensure  the 
prosecution of fraud. 

15   Anti-Fraud  Coordination  Service  (AFCOS)  -  a  national  service  in  each  Member  State  designated  to 
facilitate effective cooperation and exchange of information with OLAF according to Article 3(4) of 
Regulation 883/2013. 

6 

 
                                                            
Figure 1: Intervention logic16:

16 Source: external study 

7 

 
 
                                                            
2.3 Baseline and points of comparison  

Hercule  III  was  based  on  an  Impact  Assessment  (IA)17  that  identified  five  key  drivers 
linked  to  nine  problems  that  should  be  addressed  under  the  programme.  The  problems 
and the drivers are interconnected as can be seen in Figure 2.  

Figure 2: Hercule III drivers and problems18 

The  IA  also  found  that  the  positive  impact  of  the  Hercule  II  programme  was  not  fully 
reached  at  the  time,  e.g.  because  some  Member  States  were  not  able  to  co-finance 
Hercule II projects, especially where procurement of technical equipment was concerned. 
Notably on the eastern and southern fringes of the EU, the areas affected were also those 
potential  or  actual  weak  points  where  there  is  the  greatest  common  interest  in 
strengthening  the  control  of  the  EU’s  borders  against  smuggling.  In  response,  an 
important change introduced by the Hercule III programme was to increase the maximum 
rate of funding in co-financed activities from 50% to 80% 19. 

17   Impact  Assessment  accompanying  document  to  the  Proposal  for  a  Regulation  of  the    European 
Parliament and of the Council on the Hercule III programme to promote activities in the field of the 
protection of the European Union’s financial interests (SEC(2011)1610 final and SEC(2011)1611 final 
of 19 December 2011). 

18   Source: SEC(2011)1610 final, Impact Assessment (page 10). 

19   The maximum co-financed rate can be exceptionally raised from 80% to 90% for very specific actions 
satisfying  at  least  two  of  the  following  criteria:  i)  taking  place  at  an  external  EU  border;  ii)  taking 
place  at  the  most  vulnerable  locations;  iii)  reflecting  the  results  of  the  Eurobarometer  survey  of 
citizens’  attitudes  to  counterfeited,  smuggled  cigarettes  and  “cheap  whites”;  and  iv)  reflecting  the 
findings of the 2014 Article 325 report. as regards the number of cases of smuggled cigarettes reported 
and the estimated traditional own resources involved. 

8 

 
 
 
 
                                                            
The  Commission’s  evaluation  of  the  Hercule  II  programme  concluded  that  the 
programme achieved its objectives20. However, the evaluation report also identified some 
areas for improvement: more staff exchanges, access to databases, information exchange 
in  the  procurement  area,  beneficiaries'  reporting  and  creating  synergies  between 
Commission services. Some of these have already been taken on board or  addressed in 
Hercule  III.  For  example,  cross-border  cooperation,  including  staff  exchanges  are 
encouraged in calls for proposals for Hercule III subsidies. Regarding synergies between 
Commission services please see also page 19. 

3. 

IMPLEMENTATION / STATE OF PLAY 

3.1 Administrative implementation 

As detailed in section 2.1, Hercule III actions are divided along the following categories:  

  provision  of  technical  assistance  for  the  competent  authorities  of  the  Member 

States;  

  organisation of targeted specialised trainings, risk analysis training workshops, 

and conferences;  

  as well as an open-ended other actions cluster not falling into the previous two 

categories21.  

The actions are funded under grant agreements or procurement. 

Over  the  period  2014  to  2016,  the  overall  committed  budget  amounted  to  EUR  41 
million,  of  which  75%  was  distributed  via  grant  agreements.  391  applications  were 
submitted  over  that  period  and  125  grants  were  awarded,  while  1700  participants  took 
annually part, on average, in events funded under Hercule III. 

As explained in section 2.2, the programme is implemented on the basis of AWP that are 
adopted by a Financing Decision of the Commission22. The preparation of an AWP for a 
given year (n) starts in the autumn of the preceding year (n-1) by an internal consultation 
within OLAF as well as consulting the main stakeholders. Following these consultations 
the  AWP  is  adopted.  Once  the  Financing  Decision  is  adopted  and  the  appropriate 
financial  resources  are  made  available,  the  Commission  starts  the  implementation 
activities as announced in the AWP. During the spring of the following year (n+1), the 
Commission drafts the annual overview mentioned in Section 2.223 to be annexed to the 
Article  325  report.  The  European  Parliament  adopts  its  resolution  in  spring  of  the  year 

20   COM(2015) 221 of 27 May 2015. 
21   An  example  of  "any  other  action"  is  the  OLAF  commissioned  Eurobarometer  survey.  This  survey 
sought  to  uncover  and  explore  the  attitudes  and  opinions  of  Europeans  in  regards  to  cigarette  black 
market.  It  is  entitled  "Public  perception  of  illicit  tobacco  trade",  pooled  the  answers  of  27,672 
respondents from different social and demographic groups in 28 Member States. The findings of the 
Eurobarometer should help Member States better target awareness-raising campaigns to fight cigarette 
smuggling. 

22   C(2014)3391 final; C(2015) 2234 final; C(2016) 868 final; C(2017) 1120 final. 
23   Article 13(1) of Regulation 250/2014.   

9 

 
 
                                                            
after (n+2). Therefore, a full cycle for the preparation of, implementation and reporting 
on an AWP lasts almost 2.5 years. 

3.2  Monitoring 

The  Regulation  establishing  Hercule  III  provides  that  the  achievement  of  the  specific 
objective  of  the  programme24,  "to  prevent  and  combat  fraud,  corruption  and  any  other 
illegal activities affecting the financial interests of the Union" has to be measured by four 
key  performance  indicators  (KPI)25.  Beneficiaries  of  the  actions  have  to  report  on  the 
results  achieved.  The  modalities  of  this  reporting  differ  according  to  the  type  of  action 
(technical assistance and training)26. In the case of contracts for the purchase of access to 
databases,  the  control  over  the  results  of  the  actions  takes  a  different  form:  the 
Commission  receives  detailed  information  on  the  use  of  the  databases  by  the  different 
users. The findings based on the beneficiaries' reports in both the technical assistance and 
the  training  reports  are  included  in  the  annual  overviews  on  the  implementation  of  the 
programme and are used as input for the estimation of the KPI27.  

4.  METHOD 

4.1. Procedural aspects 

In line with Article 13(2) of Regulation 250/2014 which mandates an independent mid-
term evaluation, OLAF used  an outside contractor to perform  a  study in  support  of the 
Commission  evaluation  (the  ‘external  study’)28.  The  external  study  was  carried  out 
between  February  and  June  2017.  This  study  is  a  key  contribution  to  this  SWD,  in 
combination  with  other  sources  (they  are  referenced  as  appropriate  in  the  relevant 
sections). The main elements used for the external study were desk research and targeted 
consultations of stakeholders through interviews and surveys. Further information on the 
external study methodology can be found in Annex 3. The external study expresses the 
views  of  the  external  contractor  and,  unless  stated  otherwise  in  this  staff  working 

24   Article 4 of Regulation 250/2014. 
25   (a) information on seizures carried out by joint actions and cross-border operations, (b) added value of 
technical equipment funded under the programme, (c) information exchanges among Member States, 
and (d) the number and type of (specialised) training. 

26   For technical assistance grants, the results are reported in a "final technical report" once the request 
for final payment of the grant is made as well as in a "final implementation report" that is submitted 
one year after the closing date of a grant agreement. The reporting condition is set out in the AWP, 
further  specified  in  the  calls  for  proposals  and  required  by  the  grant  agreement.  Beneficiaries  of 
training grants have to submit a "final technical report" with the main results of surveys held among 
the  participants  to  assess  the  relevance  of  the  event  and  user  satisfaction,  as  set  out  in  the  calls  for 
proposals  and  required  by  the  grant  agreement.  As  of  2017,  grant  beneficiaries  have  to  organise  a 
"post-event survey" six months after the event took place to measure the mid-term impact of the event, 
in  particular  in  relation to  the  use  of  skills  acquired  during  a  training  event  and  the  sustainability of 
networking activities. For procurement-based activities, conferences as well as training activities, the 
beneficiaries need to organise user satisfaction surveys following these events. 

27   Article 4 of Regulation 250/2014. 
28 

  The  procurement  process  for  the  external  study  started  four  weeks  after  the  publication  of  the 
evaluation roadmap. The contract was signed on 26 October 2016, implementing framework contract 
No TAXUD/2015/CC/132.  

10 

 
 
                                                            
document, the Commission neither endorses these findings nor can it be held responsible 
for its content.  

An Inter-Service Steering Group (ISSG) of Commission services29 was set up to follow 
the evaluation process. The ISSG reviewed the roadmap, the terms of  reference for the 
external  contract,  the  inception  and  the  end-of–fieldwork  reports.  The  ISSG  was  also 
closely involved in the preparation of this SWD. 

4.2. Data collection 

This SWD answers the questions identified in the evaluation roadmap. These questions 
were  further  detailed  in  the  terms  of  reference  for  the  external  contract  and 
operationalised by the evaluator using judgement criteria and indicators30. The main data 
sources it relied upon to reply to the evaluation questions are: 

Primary data; data that were collected specifically for the purpose of the evaluation via: 

 

interviews  with  Commission  staff;  national  institutions  active  in  fraud  prevention 
and the protection of the EU's financial interests, as well as beneficiaries of actions 
funded under the programme; 

  online surveys with beneficiaries of actions funded during the first two years of the 
programme;  unsuccessful  applicants;  participants  in  events  funded  under  the 
programme, as well as users of services procured under the programme, in particular 
database users. 

Secondary data; data gathered by examining: 

  public  sources,  such  as  the  text  of  the  Regulation  and  supporting  documents  (the 
Commission  proposal  as  adopted  in  2011,  together  with  the  Impact  Assessment); 
Hercule III AWPs; Hercule III Annual Implementation Reports; Article 325 Reports; 
the final evaluation report and interim reports of the Hercule II programme, and legal 
acts in relation to other EU programmes, such as Customs 2020, Fiscalis 2020 or the 
Internal Security Fund; 

  documentary evidence on submitted applications for grants (application forms) and 
awarded contracts and grants (Final Technical Reports, Final Financial Reports  and 
Final Implementation Reports). 

The  evaluator  consulted  574  stakeholders  of  the  programme.  In-depth  interviews  were 
held with 49 civil servants (16 staff members of the Commission and 33 from national 
administrations). In addition, 56 beneficiaries of the programme were interviewed (71% 
of  all  beneficiaries),  67  unsuccessful  applicants  (25%  of  all  rejected  applicants),  321 
participants  in  events  (27%  of  participants  to  whom  a  questionnaire  was  sent)  and  112 
users of databases and services (31% of users to whom a questionnaire was sent).  

29   The ISSG was composed of representatives of the Commission Directorates General for Budget, for 
Taxation and Customs Union, for Migration and Home Affairs and for Justice and Consumers, and by 
the Secretariat General and OLAF.  

30   The evaluator gave an extensive description of its data collection activities in Annex D to its report. 

11 

 
                                                            
Quantitative information on the implementation of the AWPs, such as the budget and the 
number  of  grants/contracts,  broken  down  by  type  of  action  and  Member  State  were 
collected from the overviews on the implementation of the AWPs. 

4.3. Limitations 

The  life  cycle  of  a  programme  like  Hercule  III  is  such  that  it  takes  some  time,  at  the 
beginning  of  the  programme,  to  produce  results.  This  generates  some  limitation  to  the 
input  available  for  a  mid-term  evaluation.  The  data  collection  and  the  interviews  were 
mostly carried out in 2017. By that time, only partial results of actions funded under the 
new programme were available as these actions were funded in the first two years of the 
programme (2014 and 2015)31.  

This limitation was particularly felt for Technical Assistance (TA) actions. 13 out of 25 
TA  actions  surveyed  for  the  evaluation  were  still  ongoing  at  the  moment  of  gathering 
data. In addition, for the 13 completed actions, most beneficiaries reported having only 
recently received the equipment, thus putting them in no position to provide meaningful 
feedback  on  the  outcome  and  impact  of  the  actions.  In  other  words,  at  the  moment  of 
performing the evaluation, the independent evaluator had access only to a limited number 
of  reports  with  information  on  results  and  the  contribution  the  results  made  to  the 
achievement of the programme's general, specific and operational objectives.  

The timing for the external study therefore resulted in some limitations to the tools that 
could  be  used  and  led  the  evaluator  to  rely  to  a  significant  extent  on  the  opinions 
expressed by beneficiaries, applicants, participants, and Commission staff. However, the 
interviewees  arguably  account  for  a  representative  sample  of  the  programme's 
stakeholders, as described above in section 4.2. 

It cannot be excluded that there is a bias in the opinions expressed by the stakeholders, be 
it  positive  for  those  whose  application  was  successful  and  carried  out  some  Hercule-
funded actions, or negative for those stakeholders whose application was rejected. 

Additionally, the evaluation of some training activities under grants, such as conferences 
or training sessions, could only be carried out via the grant beneficiary who organised the 
event  and  who  had  access  to  the  personal  data  of  the  participants  required  for  the 
transmission  of  the  survey  questionnaire(s).  As  not  all  these  grant  beneficiaries  kept 
records of the personal data, it was not possible to contact all the participants to training 
and  conferences  under  grants.  However,  this  had  only  a  minor  impact  on  the  overall 
response rates since it related to events whose aggregated budget amounted to less than 
10% of the financial support made available for training actions. 

31    The  first  AWP  was  adopted  in  May  2014  (three  months  after  the  adoption  of  the  programme  in 
February 2014) and the first calls for proposals were launched by mid-2014. The first grant agreements 
were concluded at the beginning of 2015 and reports with the first results received by mid-2016. For 
technical  assistance  actions,  the  first  final  technical  reports  only  started  becoming  available  by  the 
autumn  of  2016.  Final  implementation  reports  that  technical  assistance  beneficiaries  have  to  submit 
one year after the closing date of the action were not available yet. As regards 2016, the analysis was 
limited  to  applications  received  in  the  context  of  calls  for  proposals,  as  no  action  co-financed  by 
Hercule  III  grants  for  2016  calls  had  been  completed  by  the  time  fieldwork  activities  were  being 
conducted. 

12 

 
                                                            
The external study found that beneficiaries face difficulties providing the data to measure 
the  KPI  a),  b)  and  c)  of  Article  4  of  the  Regulation.  The  collection  of  data  to  measure 
these  indicators  is  reportedly  the  most  burdensome  aspect  of  the  reporting  phase.  The 
reports to feed the KPIs therefore provided only a relatively limited input into the mid-
term evaluation of the programme. The definition of the KPIs in the Regulation allows 
only for results that, in practice, are too general and would require to be further broken 
down to be fully used. This should however be addressed by the successor programme in 
the next MFF period. 

There is no indication, however, that the above limitations would invalidate the result of 
the external study. 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS 

5.1 Relevance 

This section focuses on the relevance of the programme for the protection of the financial 
interests  of  the  Union.  The  questions  are  based  on  the  Better  Regulation  Guidelines 
examples and were adapted for the purposes of the evaluation by the ISSG. 

EQ1:  To  what  extent  have  the  specific  and  operational  objectives  of  the  Hercule III 
programme  proven  to  be  relevant  for  the  general  objective  of  the  protection  of  the 
financial interests of the EU? 

On the basis of the external study32, it appears that the achievements of the specific and 
operational objectives of the programme were relevant to achieving its general objective. 

The latest Article 325 report33 shows that the estimated financial impact of irregularities 
and  fraud  is  hardly  declining  over  the  last  years  (see  Figure  3).    The  Article  325  report 
reports on all (fraudulent) irregularities detected by the Member States. This decline is not 
exclusively  linked  to  the  Hercule  programmes.  The  fluctuations  in  reported  irregularities 
and related financial impact can be also due to the level of reporting and (lack of) measures 
taken  to  address  the  irregularities.  Hercule  III  helps  to  boost  the  operational  and 
administrative  capacity  of  Member  States’  customs  and  other  authorities.  In  that  regard 
substantial  successes  were  reported  in  relation  to  smuggled  and  counterfeit  cigarettes  and 
tobacco:  seizures  were  made  with  the  help  of  equipment  and  training  funded  under  the 
programme34.  

32   External study, page 36. 
33   COM(2017)383 final of 20 July 2017. 
34   SWD(2017) 269 final, Commission Staff Working Document; Annual overview with information on 

the results of the Hercule III programme in 2016. 

13 

 
 
 
 
                                                            
Figure 3: Number of reported irregularities and amount: expenditures 

s
n
o

i
l
l
i

m
€

3000

2500

2000

1500

1000

500

0

2008 2009 2010 2011 2012 2013 2014 2015 2016

Estimated financial impact - Non-fraudulent

Estimated financial impact - Fraudulent

Number of irregularities

20000

15000

10000

5000

0

s
e
i
t
i
r
a
u
g
e
r
r
i

l

f
o

r
e
b
m
u
n

Source: External study, contractor's elaboration on Article 325 reports.  

The  policy  problem  in  connection  with  fraud  and  irregularities  to  the  EU's  financial 
interests that was prevailing at the time of the preparation of the Regulation, as reflected 
in the IA, is still prominent. Therefore, considering that the Regulation deviates from the 
IA  findings  only  on  procedural  matters35,  the  specific  and  operational  objectives  of  the 
programme  could  be  expected  to  be  still  relevant  for  the  general  objective  of  the 
protection of the financial interests of the EU. 

This  is  confirmed  by  the  stakeholders’  perception  as  presented  in  the  bar  chart  below. 
The same is corroborated by the conclusions in final technical reports of actions funded 
under  grant  agreements,  where  all  beneficiaries,  who  submitted  the  final  technical 
reports, reported on how the results contributed to the specific and general objectives (see 
the answers to the next evaluation question).   

35   For instance the possibility for the Commission to adopt delegated acts for the implementation of the 

programme. 

14 

 
 
                                                            
 
 
 
 
 
 
 
Figure 4: To what extent does each operational objective contribute to the PFI and, 
more specifically, to preventing and combating fraud, corruption and other illegal 
activities  against  EU  financial  interests,  including  cigarette  smuggling  and 
counterfeiting? (Average evaluation; number of respondents) 

1. Enhancing transnational and multidisciplinary
cooperation between Member State authorities,
the European Commission and OLAF to prevent
and investigate fraud

2. Facilitating the exchange of information,
experience and best practices related to the
protection of EU financial interests, including staff
exchange

3. Providing technical and operational support to
competent authorities of Member States in their
fight against fraud and other illegal activities

4. Reducing the development of an illegal
economy in key risk areas such as organised
fraud, with special emphasis on actions aimed to
fight cigarette smuggling and counterfeiting

5. Promoting comparative law analysis and
supporting academic analysis of strategic legal
issues with a view to developing a broad
consensus on how to better use legal resources in
the protection of EU financial interests

4,19 

3,97 

4,58 

4,49 

3,98 

4,33 

4,46 

4,22 

4,67 

3,98 

3,88 

4,45 

3,98 

3,45 
3,50 

Beneficiaries

Unsuccessful Applicants

Institutions

1

2

3

4

5

Scale: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest 
extent. Respondents who replied “Do not know/No opinion” or “Not Applicable” are not included. 
Source:  On-line  survey  and  interviews  with  institutions,  beneficiaries  and  unsuccessful  applicants. 
External study. 

This  bar chart  also  shows  that  the stakeholders  perceived  that  the  operational  objective 
"promoting  comparative  law  analysis  and  supporting  academic  analysis  of  strategic 
legal  issues"  was  relevant,  yet  less  than  the  other  operational  objectives.  It  should  be 
observed that this action is very specific to a small group of beneficiaries (research and 
educational  institutes  and  not  for  profit  organisations),  which  explains  why  most  other 
stakeholders have seen this operational objective as less relevant.   

EQ2:  To  what  extent  have  the  activities  of  the  Hercule  III  programme  proven  to  be 
relevant for achieving its operational and specific objectives? 

The activities of Hercule III have been relevant to the achievement of the programme’s 
operational and specific objectives.  

During  the  course  of  the  external  study,  the  stakeholders  largely  agreed  to  this  as  also 
corroborated  by  findings  in  the  final  technical  reports  submitted  by  grant  beneficiaries. 
One  important  explanation  for  this  result  is  that  this  parameter  is  assessed  during  the 
evaluation  of  applications  for  grant.  Applicants  must  demonstrate  to  the  Commission 

15 

 
 
how  the  action  is  expected  to  contribute  to  the  programme's  general,  specific  and 
operational objectives.   

Nonetheless,  two  actions  were  largely  perceived  as  less  relevant  to  the  achievement  of 
the  general  objective.  These  are:    "purchase  of  services  to  store  and  destroy  seized 
cigarettes and other counterfeit goods" and "purchase of services to carry out chemical 
analysis of samples from tobacco and cigarette seizures." The Commission received only 
one application in 2014 and 2015 for grants for the destruction of seizures. Regarding the 
latter objective, the Commission asked, under the programme, the Joint Research Centre 
(JRC)  to  set  up  laboratory  facilities  for  the  chemical  analysis  of  samples  from  seized 
tobacco  and  cigarettes.  The  overall  use  of  this  facility  has  so  far  been  low,  possibly 
because  it  is  not  yet  sufficiently  known  amongst  the  operational  staff  responsible  for 
processing  the  seizures.  To  address  this  issue,  the  Commission  is  organising  in  2017-
2018  training  sessions  in  the  Member  States  for  operational  staff,  financed  under 
Hercule III. 

For  the  same  reasons  identified  under  EQ1  above  some  stakeholders  did  not  find  of 
particular relevance the actions geared towards building and disseminating scientific and 
legal knowledge in the field of the protection of the financial interests of the EU, as this 
action is specific to the actual small group of beneficiaries.  

While  emphasising  the  need  to  streamline  the  number  of  eligible  actions  to  avoid 
dispersion  of  funds  on  too  many  activities,  some  stakeholders36,  suggested  additional 
activities  that  would  contribute  to  the  relevance  of  the  programme.  These  include 
trainings  on  big  data  analysis,  activities  in  connection  with  detecting  and  analysing 
digital evidence, research in the field of digital forensics, multidisciplinary research into 
determinants of fraud affecting the EU financial interests, and risk analysis and profiling 
of passengers to enhance detection of smuggled goods.  

The external study also found that, in view of the parallel increase of irregularities on the 
expenditure side and the decrease on the income side37, more emphasis could be put on 
activities in connection with the expenditure side. 

5.2 Coherence 

This  section  examines  to  what  extent  the  activities  within  the  programme  are  coherent 
(internal  coherence)  and  to  what  extent  the  programme's  activities  are  coherent  with 
activities undertaken by other EU programmes (external coherence) in order to detect any 
overlaps and to identify synergies. 

EQ3: What are the synergies between and within the different types of actions under 
the programme and with other EU supported measures, programmes and actions, such 
as Customs 2020 or Fiscalis 2020? 

The internal coherence of the programme is ensured by (i) the allocation of the budget for 
the period 2014-2020, (ii) the control performed by the Commission on possible overlaps 
and  repetitions  on  the  basis  of  information  provided  in  the  applications,  (iii)  the 

36   The  external  study  did  not  explore  whether  these  suggestions  were  shared  by  a  larger  group  of 

stakeholders. These suggestions therefore have only a limited value.  

37   External study, Annex F, Evolution of key indicators on irregularities and fraud. 

16 

 
                                                            
Regulation’s  requirement  that  applications  for  equipment  be  accompanied  by  other 
activities,  e.g.,  trainings,  and  (iv)  discussions  and  coordination  between Member  States 
on  the  occasion  of  high  level  conferences  financed  by  Hercule  ensuring  some  actions 
which are particularly relevant to the protection of the financial interests of the EU (PFI) 
are performed irrespective of grant applications.  

i. 

There  is  an  internal  coherence  element  in  the  programme  between  the  different 
categories  of  actions  provided  for  in  the  Regulation  (see  Section  2.3).  The 
financial  limits  that  are  set  for  each  category  (at  least  70%  of  the  budget  for 
technical  assistance  actions,  maximum  25%  for  training  actions,  both  anti-fraud 
and legal training actions, and up to 5% for other actions) are a guarantee that this 
coherence will be respected, by balancing the actions where there is more demand 
and  other  actions  that  are  particularly  relevant  to  the  PFI  (e.g.,  conferences  on 
specific  PFI  topics  or  forensics  training).  The  financial  implementation  of  the 
AWPs shows that this indicative allocation has been complied with38. 

ii.  Applicants for grants have to indicate in their applications whether or not similar 
requests  have  been  made  in  the  past  and  during  the  evaluation  of  the  grant 
applications,  allowing  the  Commission  to  closely  monitor  that  actions  do  not 
overlap:  conferences  proposed  in  training  applications  should  not  coincide  or 
address overlapping topics, whereas requests for funding for expensive technical 
equipment are checked to ensure that similar equipment is not already in use by 
the same services or in the same geographical area. The accepted applications for 
grants do indeed comply with this obligation. 

iii. 

The Regulation provides in Article 10(3) that the purchase of equipment shall not 
be  the  sole  component  of  a  grant  agreement.  This  provision  is  implemented  by 
requesting  applicants  to  ensure  that  technical  assistance  actions  contain  training 
for staff to operate the equipment or to include activities enabling the exchange of 
relevant  information  and  best  practices  with  other  law  enforcement  services  or 
peers in neighbouring Member States.  

iv.  High-level  conferences  arranged  by  OLAF  ensure  that  some  key  topics  for  the 
PFI  (e.g.  fight  against  cigarette  smuggling,  communication  activities  on  fraud 
prevention,  fraud  in  structural  funds,  cooperation  in  anti-fraud  activities)  are 
discussed  and  coordinated  at  the  EU  level  by  relevant  national  authorities.  This 
allows a coherent approach to the PFI in general and the programme in particular  
across  Member  States,  irrespective  of  the  specific  actions  for  which  national 
authorities decide to apply for. 

The contractor carried out an analysis of the regulations establishing the programme and 
Customs 2020, Fiscalis 2020, ISF Police and the Justice programme and the most recent 
AWPs. Although each programme pursues very different general objectives, interactions 
can be detected between specific and operational objectives. However as regards external 

38   Over the period 2014-2016, on the basis of Financing Decision, around 70% of the budget was spent 
on grant agreements and the remaining 30% on procured services, in particular digital forensic training 
and  conferences.  At  the  end  of  the  year,  unspent  Hercule  III  commitments  are  spent  on  grants,  thus 
increasing slightly overall percentage spent on grants.  

17 

 
 
 
 
 
                                                            
coherence, the external study found that the overlap between the Hercule programme and 
other  mentioned  programmes  is  very  limited,  essentially  thanks  to  the  Hercule 
programme’s requirements and the Commission internal consultation mechanisms.  

As  explained  above,  the  selection  process  allows  detecting  overlaps,  on  the  basis  that 
applicants have to indicate whether or not they received grants for similar actions from 
other EU programmes.  

Some  equipment  procured  under  Hercule  III  can  be  used  for  purposes  other  than  the 
protection of the EU financial interests. A scanner installed at a border will, for instance, 
not  only  detect  smuggled  cigarettes  but  also  illicit  products  such  as  drugs  or  trafficked 
persons  as  these  might  be  transported  in  vehicles  that  are  checked  by  the  scanner 
procured  under  Hercule  III.  This  would  not  however  weaken  the  coherence  of  the 
programme, and could even be seen as complementing the specific objective of the ISF 
programme to achieve "a uniform and high level of control of the external borders" and 
thus creating synergies.  

The  Commission  internal  coordination  process  allows  preventing  overlaps  at  several 
levels. As testified during the interviews with EU staff, few overlaps exist between EU 
programmes,  such  as  Customs/Fiscalis  2020,  or  the  ISF  (see  the  bar  chart  below, 
Figure 5).  The  Commission  inter-service consultation  process  to  which draft  AWPs  are 
submitted  ensures  complementarity  of  the  programmes.  Additionally,  during  the 
preparation  of  the  AWP  of  Hercule,  Commission  services  responsible  for  the 
management  of  Customs/Fiscalis  2020  and  the  ISF  are  closely  involved  in  order  to 
prevent  any  overlaps  from  occurring  and  to  ensure  that  existing  synergies  can  be 
strengthened  or  that  new  synergies  can  be  identified.  The  AWP  is  also  submitted  to 
different  working  groups  of  Member  States'  representatives  to  collect  information  and 
feed-back to ensure that the proposed AWP addresses existing or new needs and to avoid 
overlaps with other EU and national programmes. Lastly, at the stage of launching calls 
for proposal, representatives of other Commission services39 are closely involved in the 
elaboration of the call documents as well as in the examination of applications in order to 
ensure,  as  far  as  possible,  that  overlaps  do  not  occur  and  that  synergies  with  other  EU 
programmes are strengthened.  

This is supported by the views of the participants to the online survey who mostly replied 
that they, for the most part, were unaware of other programmes and that those who were, 
indeed  detected  synergies,  and  only  limited  overlaps,  between  Hercule  and  other  EU 
programmes (Figure 5). Although more synergies between Commission services could be 
achieved,  this  needs  to  be  carefully  balanced  against  inflating  workload,  creating  extra 
layers and generating delays. If fighting against corruption and VAT fraud would become 
more central in Hercule III, the external study advises to maximise synergies with other 
programmes  managed  by  DG  TAXUD  and  DG  HOME,  which  touch  upon  these  two 
issues that can harm the EU's financial interests. 

39   Directorates-Generals for Taxation and Customs Union (TAXUD), Justice and Consumers (JUST) and 

Migration and Home Affairs (HOME).  

18 

 
                                                            
Figure  5:  Replies  to  the  question:  "To  what  extent  does  the  programme  have 
synergies and/or overlaps with other EU-funded programmes (you are familiar with)?"  

[VALUE] (5)  

[VALUE] (4) 

[VALUE](7) 

[VALUE] (5) 

[VALUE](3) 

[VALUE](4) 

[VALUE](7) 

[VALUE](4) 

[VALUE](3) 

[VALUE](4) 

i

s
e
g
r
e
n
y
S

s
p
a
l
r
e
v
O

Customs
2020/Customs
2013 (DG
TAXUD)

i

s
e
g
r
e
n
y
S

s
p
a
l
r
e
v
O

i

s
e
g
r
e
n
y
S

s
p
a
l
r
e
v
O

i

s
e
g
r
e
n
y
S

s
p
a
l
r
e
v
O

i

s
e
g
r
e
n
y
S

s
p
a
l
r
e
v
O

Fiscalis 2020
/Fiscalis 2013
(DG TAXUD)

Internal Security 
Fund – Police / 
ISEC (DG HOME) 

Justice
Programme (DG
JUST)

Horizon 2020/FP7

Scale: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest 
extent. Respondents who replied “Do not know/No opinion” or “Not Applicable” are not included. 
Source:  On-line  survey  and  interviews  with  institutions,  beneficiaries  and  unsuccessful  applicants. 
External study. 

5.3 Effectiveness 

This section focuses on the extent to which the programme has attained its objectives and 
its expected results.  

EQ4: To what extent have the overall intervention logic/strategy of the programme and 
the  actions  contributed  to  the  achievement  of  the  objectives  of  the  Hercule  III 
programme? 

Over  the  period  covered  by  the  mid-term  evaluation,  the  programme  financed  actions 
which,  because  of  their  nature  or  objectives  clearly  contributed  to  achieving  the 
objectives of the programme (e.g. providing co-financing for a scanner purchase in order 
to  identify  smuggled  tobacco  products).  In  terms  of  measuring  this contribution,  this  is 
indeed corroborated by the external study as far as anti-fraud and legal training actions 
are concerned, as  well as  by the analysis of the final  reports for grants. Because  of the 
data limitations highlighted in Section 4.3, the external study is inconclusive on this point 
as far as technical assistance is concerned.  

Under  training  actions,  65  events  were  organised  for  training,  conferences  or  seminars. 
The events covered a wide range of different topics, such as training for customs staff, 
digital forensic training and training on fraud prevention or general issues in relation to 
the protection of the financial interests of the Union. On the basis of the final reports and 
in  particular  of  the  results  of  the  user  satisfaction  survey40,  the  external  study  (see 

40   As  explained  in  section  4.3,  in  order  to  ensure  data  protection  of  the  participants  of  the  events  and 
users  of  databases,  the  satisfaction  surveys  were  administered  by  the  grant  beneficiaries  to  the 

19 

 
 
 
 
                                                            
 
Figure 6 below) showed that the expected outcomes, such as networking possibilities or 
the acquisition of new skills and competences have been largely achieved. However, only 
a  few  actions  involved  staff  exchanges  between  national  administrations,  although  the 
evaluation  of  the  Hercule  II  programme  recommended  that  more  personal  contacts 
between administrations, e.g. through staff exchanges, would improve the impact of  the 
programme. 

Figure 6: To what extent did the event funded by the programme contribute to the 
achievement  of  the  following  outcomes?  (Average  evaluation;  number  of 
respondents) 

1. Improved investigative performance/abilities of law
enforcement officials

2. Increased awareness of fraud risk indicators and EU
anti-fraud policy

3. Enhanced knowledge of specialised methodologies,
tools and techniques to fight fraud

4. Exchange of information and sharing of best practices
between law enforcement agencies in relation to digital
forensic hardware and software to secure evidence…

5. Improved knowledge in the field of comparative law
with regard to the protection of EU financial interests

6. Improved cooperation between practitioners and
academics

7. Increased awareness of the judiciary and other
branches of the legal profession with regard to the
protection of EU financial interests

3.68 (320) 

4.02 (259) 

3.87 (310) 

3.67 (306) 

3.68 (253) 

3.82 (279) 

3.63 (281) 

1

2

3

4

5

Participants

Scale: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent. 
Note: Number of informed respondents in brackets. Differences in number of respondents across dimensions are due to 
respondents selecting the “Do not know/No opinion” or “Not Applicable” answer. 
Source: Online survey with participants in events. External study. 

Technical assistance grants were awarded to applications covering four different topics: 
investigation tools (cameras, monitoring devices for vehicles or mobile communications, 
adapted vehicles, etc.), detection equipment (scanners, sniffer dogs), Automated Number 
Plate Recognition Systems (ANPRS) and the destruction of seizures. Most of the budget 
was  spent  on  investigation  tools  (54%),  whereas  28%  was  spent  on  detection  tools. 
ANPRS  actions  received  15%  of  the  budget  and  the  destruction  of  seizures  (one  grant 
only) amounted to less than 2%. Taken account of the relatively recent signature of the 
first  grant  agreements  at  the  beginning  of  2015,  only  few  results  were  available  at  the 
start  of  the  evaluation41.  Technical  assistance  actions  also  include  procured  access  to 

participants and by OLAF to the users of databases. The external contractor would receive the results 
of the surveys directly as these were filled in.  

41   The interviews with five of the first beneficiaries of grants awarded under the 2014 budget as well as 
the survey generated some interesting, yet anecdotal, findings from the first months of operation of the 

20 

 
 
                                                                                                                                                                                 
 
databases for use by law enforcement agencies during PFI related operations, laboratory 
facilities  for  the  examination  of  samples  from  tobacco  seizures  as  well  as  the 
development of specific IT-tools  for data-mining and data analysis purposes.  The latter 
provides  substantial  input  to  Member  States'  customs  for  risk  analyses  to  identify 
shipments  at  risk.  The  external  study  found  that  the  users  of  these  services  considered 
them as effective, as shown in the bar chart below. 

The  examination  of  the  available  13  Final  Technical  Reports  on  technical  assistance 
actions did not provide sufficient input for the outcome indicators, at the level of results. 
As indicated earlier, this is due to more than half of sampled actions being still ongoing 
during  the  evaluation  period  while  many  others  were  concluded  very  recently,  thus 
unable  to  provide  more  outcome  indicators.  Yet,  they  provided  anecdotal  examples  of 
achievement. For instance, the deployment of a wide range of investigation and detection 
equipment enabled the seizure of more than 21 tonnes of cigarettes during more than 150 
different  operations.  In  addition,  drugs,  weapons  and  alcohol  were  seized  as  well.  The 
use  of  the  ANPRS  led  to  the  discovery  of  10  tonnes  of  drugs.    While  such  equipment 
contributes  to  successful  operations,  it  must  be  borne  in  mind  that  it  is  used  in 
combination  with  other  tools  and  does  not  allow  attributing  all  the  results  only  to  the 
programme. 

During  the  mid-term  evaluation,  the  Commission  continued  receiving  final  technical 
reports  and  final  implementation  reports  related  to  actions  funded  under  Hercule  II, 
which are comparable to the reports required by Hercule III. These reports also contained 
interesting  information  on  the  results  achieved  with  specific  technical  equipment.  The 
aggregated prevented losses to the national and Union budget in these reports represent 
an amount that is substantially higher than the Hercule II budget. It is likely to be even 
higher  because  the  beneficiaries  are  not  required  to  report  on  a  systematic  basis.  The 
reporting period is mostly limited to a period covering 12-15 months and results beyond 
that period are not reported to the Commission. 

Regarding the evaluation of the perception of the stakeholders, the external study found 
that  the  programmes’  actions  and  their  results  indeed  contribute  to  the  programme's 
objectives.  Overall,  across  all  respondents’  groups,  even  the  action  that  is  seen  as 
contributing  the  least  to  the  programmes’  objectives,  i.e.  the  promotion  of  comparative 
law  analysis  and  the  support  to  academic  analysis  of  strategic  legal  issues,  still  scores 
relatively well (between 3 and 4 on a scale from 1 to 5 - Figure 7). However, the external 
study did also note that more should be done on the expenditure side of the EU budget, 
which is an area currently receiving fewer grants.  

purchased equipment. One of the interviewed beneficiaries purchased an ANPRS used to check more 
than 16 million licence plates that generated more than 3 000 hits (i.e.: licence plates that were flagged 
in databases for different types of warnings, including warnings related to PFI-investigations). Another 
beneficiary  purchased  investigation  tools  that  were  used  in  more  than  500  operations  leading  to  37 
arrests and 22 convictions. 

21 

 
                                                                                                                                                                                 
Figure  7:  Replies  to  the  question:  "To  what  extent  did  the  action  funded  by  the 
following  objectives?"  
the  achievement  of 
programme  contribute 

the 

to 

1. Protect the EU financial interests

2. Prevent and combat fraud, corruption and
other illegal activities against EU financial
interests, including cigarette smuggling and
counterfeiting

3. Enhancing transnational and multidisciplinary
cooperation between Member State authorities,
the European Commission and OLAF to prevent
and investigate fraud

4. Facilitating the exchange of information,
experience and best practices related to the
protection of EU financial interests, including staff
exchange

5. Providing technical and operational support to
competent authorities of Member States in their
fight against fraud and other illegal activities

6. Reducing the development of an illegal
economy in key risk areas such as organised
fraud, with special emphasis on actions aimed to
fight cigarette smuggling and counterfeiting

7. Promoting comparative law analysis and
supporting academic analysis of strategic legal
issues with a view to developing a broad
consensus on how to better use legal resources…

4.39 (49) 

3.92 (12) 

4.50 (46) 

3.91 (11) 

4.07 (45) 

4.00 (12) 

4.22 (49) 

3.67 (12) 

4.10 (39) 

3.83 (12) 

3.91 (34) 

3.27 (11) 

3.93 (30) 

3.11 (9) 

1

2

3

4

5

Beneficiaries

Institutions

Scale: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest 
extent. Respondents who replied “Do not know/No opinion” or “Not Applicable” are not included. 
Source:  On-line  survey  and  interviews  with  institutions,  beneficiaries  and  unsuccessful  applicants. 
External study. 

EQ5:  To  what  extent  have  these  objectives  been  achieved  through  the  Hercule  III 
programme’s interventions and to what extent have other factors played a role? 

Fraud,  corruption  and  irregularities  to  the  EU  budget  unfold  in  a  multi-factor 
environment  where  the  EU  and  national  regulatory  frameworks,  political  will,  judicial 
capacity,  and  criminal  behaviours  inter-play,  among  others.  The  achievement  of  the 
objectives  of  the  programme  is  therefore  heavily  influenced  by  these  factors.  As  an 
illustration,  the  positive  impact  of  the  procurement  of  scanners  to  identify  smuggled 
tobacco products will be usually stronger in a country where the judiciary fight against 
the criminal groups organising such smuggling is a priority and vice versa. However, it is 
impossible  to  measure  the  interplay  between  these  other  factors  and  Hercule's 
interventions in the achievement of the objectives of Hercule III.  

The programme's stakeholders have identified a number of specific factors as influencing 
the achievement of the programme's objectives. The most important factor hindering the 
achievement  of  the  objectives,  with  no  significant  differences  in  ranking  among  all 
stakeholder categories, is the ability of fraudsters to “adapt quickly to new circumstances 

22 

 
 
at  the  EU  level,  operate  cross-border  and  exploit  the  weakest  points  in  the  external 
borders”. The other identified factors are: 

-  “the  large  number  and  varied  nature  of  competent  national  and  regional 

authorities” leading to an uneven PFI; 

-  “differences in incentives and capacities between MS”;  
-  “divergent application of criminal law and penal sanctions” across EU MS;  
-  “pressure on public finance” requiring more cost-efficient solutions. 

A few stakeholders, not statistically representative, have identified some other obstacles, 
including:  weak  political  willingness  to  protect  the  EU  financial  interests;  rising 
scepticism  vis-à-vis  EU  institutions;  poor  knowledge  about  fraud  affecting  the  EU 
financial  interests;  limited  financial  envelope  of  the  programme  and  limited  financial 
resources available at MS level to protect the EU financial interests; and lack of skilled 
personnel and modern technical tools to combat and prevent fraud. Consideration might 
be  given  to  the  impact  of  these  factors  during  the  final  evaluation  of  the  Hercule  III 
programme.   

5.4 Efficiency 

This  section  examines  whether  the  results  of  the  actions  have  been  achieved  at  a 
reasonable cost.  

EQ6: To what extent have the desired effects been achieved at reasonable costs? 

Cost-effectiveness is inherent to the implementation of technical assistance actions: they 
mostly  consist  of  the  acquisition  of  equipment  that  has  to  be  purchased  by  public 
procurement  procedures  where  the  "best  value  for  money"  principle  is  used  to  select 
equipment.  Nevertheless,  due  to  the  data  limitation  (see  section  4.3),  further 
quantification was not possible. The external study found that the costs per participant for 
trainings are broadly aligned with those recorded by other EU-funded programmes. 

The  average  beneficiary  of  the  programme  spent  approximately  18  person-days  to 
complete  the  application  phase,  4.4  person-days  for  the  grant-awarding  phase  and  13 
person-days  for  the  reporting  phase.  The  external  study  found  that  this  is  in  line  with 
other  EU  funded  programmes.  This  translates  into  the  average  applicant  spending 
approximately EUR 4 500 for transaction costs, i.e.: all costs incurred for the acquisition, 
monitoring and reporting (for more detail per type of action, see Figure 8 below).  

The external study also established the average cost per participant to anti-fraud training 
at EUR 467 and that to legal training at EUR 694. This cost is broadly in line with other 
EU-funded programmes. No such analysis could be done for technical assistance because 
of limited availability of data. 

The  external  study  demonstrated  that  regulatory  costs  linked  to  the  application  phase 
yield “value for money”. The beneficiaries of anti-fraud training activities were expected 
to get about EUR 12 for each euro spent in preparing a proposal (the expected return on 
investment).  For  legal  training  proposals  this  amount  was  around  EUR  7  for  each  euro 
spent.  While  beneficiaries  of  technical  assistance  grants  were  expected  to  get  almost 
EUR 67, as the average grant requested was much greater (Figure 8).  

23 

 
 
 
Figure 8: Costs/benefits analysis 

Administrations / Research and educational institutes and non-profit-
making entities 

Regulatory costs42  

Benefits / Return on investment43 for 
each euro invested in the application 
process if grant is awarded 

Anti-fraud training 
activities 

EUR 2 195 - Application phase 
EUR 467    - Grant award and 
contracting phase 
EUR 1 830 - Reporting phase 

EUR 12.4 

Legal training 
activities 

EUR 4 492 - Total 

EUR 3 235  - Application phase 
EUR 886    - Grant award and 
contracting phase 
EUR 2 900 - Reporting phase  

EUR 7.3 

EUR 7 021- Total 

Technical assistance  EUR 1 839 - Application phase 

EUR 66.8 

EUR 486    - Grant award and 
contracting phase 
EUR 1 312 - Reporting phase  

EUR 3 637 - Total 

Source: External study 

EQ7:  Could  the  same  effects  have  been  achieved  with  lower  costs  if  procedures  had 
been  simpler,  involving  less  administrative  burden  and/or  efficient  implementation 
mechanisms had been applied? 

On the basis of the external study’s findings, the application process does not appear to 
be  too  cumbersome  for  the  majority  of  stakeholders  (approximately  70%).  30%  of 
applicants  expressed  reservations  on  the  application  process  and  33%  of  beneficiaries 
found the reporting process to be cumbersome. Among those, some made suggestions for 
improvement  which,  according  to  the  external  study,  would  generate  some  substantial 
savings  (see  footnotes  56,  57  and  58).    If  their  suggestions  were  accepted,  the  external 
study  estimates  that  savings  of  about  EUR  2  000  per  action  would  materialise  (see 
Figure 9 below). Some regulatory obligations are, however, inherent to EU procurement 

42   Regulatory  costs  incurred  when  dealing  with  administrative  activities  are  linked  to  three  specific 
phases:  i)  drafting  grant  proposals;  ii)  completing  the  grant  award  and  contracting  phase;  and  iii) 
complying with reporting obligations. 

43   The expected value of a proposal is calculated as the grant requested multiplied by the likelihood of 
getting the grant (calculated by dividing the number of successful applications by the total number of 
applications). This value is then divided by the regulatory costs incurred in the application phase, thus 
measuring the expected “return on investment” at the moment of applying for a grant.  

24 

 
 
 
 
                                                            
and cannot be simplified given the current legal framework. Therefore, only some of the 
suggestions  of  the  applicants  could  be  implemented.  Almost  all  respondents  who 
considered the grant awarding and contracting phase cumbersome suggested what could 
be done for making the application phases less cumbersome. However, each  suggestion 
was provided by a limited number of consulted stakeholders. 

Figure 9: Potential savings 

Administrations / Research and educational institutes and 
non-profit-making entities 

Potential Savings 

Application44 

EUR 1 100  

Grant award and contracting45 

EUR 350 

Reporting46 

Total 

Source: External study 

EUR 700 

EUR 2 150 

In  2017, the Commission  services  have  started using an electronic management system 
for  the  submission,  processing  and  management  of  grant  applications  under  the 
programme (‘e-grant system’).  The objective of this system is to streamline the process 
and  alleviate  the  burden  of  the  applicants.  The  improvements  brought  about  by  the  e-
grant system will address a large number of these suggestions, for instance by digitising 
the  whole  process  and  introducing  the  use  of  digital  signatures.  The  e-grant  system  is 
expected to noticeably alleviate the administrative burden and related time and resources 
spent on the applications in particular.  

44   Avoiding  redundancies  in  the  information  requested;  improving  the  readability  of  the  application 
forms; providing more guidance on the application process, e.g. via webinars or tutorials; reducing the 
amount  and  detail  of  information  requested,  e.g.  by  postponing  some  information  obligations  to  the 
grant  award  phase;  translating  calls  for  proposals  and  guidance  documents  into  all  EU  languages; 
digitising the entire application, including the use of digital signatures. 

45   Digitising  the  process;  providing  assistance  in  all  EU  languages  and  reporting  in  all  EU  languages; 
shortening the time-gap between the submission of the application and the signature of the contract; 
allowing for subcontracting an organisation rather than only physical persons. 

46   Digitising  the  reporting  process,  e.g.  by  creating  a  centralised  platform  to  upload  relevant  data; 
consideration of the specific needs of research and educational institutes (which, for instance, may not 
be  able  to  prepare  timesheets);  allowing  the  inclusion  of  person-days  spent  on  preparing  reporting 
documents  in  the  budget  of  the  action;  and  devoting  a  specific  time  window  to  reporting  activities, 
after the completion of the action. 

25 

 
 
 
 
 
                                                            
5.5 EU added value 

This  section  covers  the  programme's  added  value  at  the  level  of  the  European  Union, 
compared with no intervention at EU level in this field.  

EQ8:  Has  the  programme  allowed  delivering  results  that  could  not,  or  to  a  lesser 
extent, be achieved by interventions undertaken at national or regional level? 

The external study found that the stakeholders almost unanimously recognised the added 
value  of  the  programme  (see  in  particular  Figure  10),  in  all  its  components,  and  its 
capacity to generate unique results and benefits that could not be generated at national or 
regional levels. 

The  primary  explanation  is  the  lack  of  budget  to  fund  similar  actions  at  national  or 
regional  levels.  Absent  the  programme,  the  actions  provided  for  by  Hercule  would 
probably not exist. Additionally, from the point of view of the Commission and Member 
States’  services,  Hercule  III  funded  interventions  generate  better  results  and  more 
benefits than comparable national/regional interventions would47.  

Figure 10: Would the results and benefits of actions funded by the programme also 
be generated in the absence of the programme? (% of respondents)  

62 

37 

38 

39 

13 

0 

2 

0 

Definitely would not Probably would not

Probably would

Definitely would

beneficiaries

institutions

9 

0 

Do not know/No
opinion

Source: Online survey and interviews with institutions and beneficiaries. External study. 

Importantly also, the programme contributes to a better protection of EU external borders 
and  to  strengthening  long-lasting  cross-border  cooperation,  cross-border  exchange  of 
information  and  cross-border  exchange  of  best  practices  (see  Figure  11  below).  This 
confirms  the  added-value  of  a  programme  at  the  level  of  the  EU.  Some  beneficiaries, 
although  not  a  statistically  significant  number  of  them,  suggested  that  there  is  more 
potential in centralised (common) purchases of technical equipment as this would reduce 
costs for national authorities and would ensure cross-border use of that equipment. 

47 External study, page 90.  

26 

 
 
 
 
                                                            
Figure 11: To what extent did the actions funded by the programme lead to cross-
border cooperation, exchange of information and/or best practices and/or common 
use of equipment? (Average evaluation; number of respondents) 

1. Cross-border cooperation

2. Cross-border exchange of information

3. Cross-border exchange of best practices

4.11 (38) 

4.00 (13) 

4.22 (41) 

3.85 (13) 

4.17 (42) 

4.00 (13) 

4. Common use of equipment across EU
countries

3.00 (25) 

3.44 (9) 

1

beneficiaries

2

institutions

3

4

5

Scale: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest 
extent. 
Source: Online survey and interviews with institutions and beneficiaries. External study. 

The majority of the funded actions are expected to generate benefits in terms of exchange 
of  information  and  good  practices  between  Member  States  as  well  as  cross-border 
cooperation,  although  to  a  lesser  extent  in  the  case  of  technical  assistance  actions  (see 
Figure 12). 

Figure 12: Percentage of actions enabling cross-border cooperation 

Exchange of information and best 
practices 
Cooperation with other MS 

Anti-fraud 
training 

Legal 
training 

Technical 
assistance 

100% 

76% 

88% 

94% 

45% 

24% 

Total 

71% 

54% 

Note: For technical assistance, results are based on information sourced from application forms. For legal 
training,  37.5%  of  the  results  are  based  on  information  sourced  from  application  forms.  For  anti-fraud 
training, 4% of the results are based on information sourced from application forms. 
Source: External evaluator’s elaboration on application forms and available reporting documents. 

Overall,  these  results  are  coherent  with  the  IA  which  had  identified  that  there  were 
inadequate  national  solutions  to  a  number  of  problems  within  the  scope  of  the 
programme  (e.g.  insufficient  information  sharing  between  authorities  or  inadequate 
sharing of best practices).  

EQ9:  Does  the  intervention  at  the  EU  level  provide  added  value  in  terms  of  the 
efficient use of financial resources as compared to a possible intervention at national 
level? 

The  external  study  found  that  the  financial  resources  are  more  efficiently  used  through 
the programme than national interventions would, for instance by generating cost savings 
or securing better commercial terms (e.g. lower searching costs to find the most adequate 
equipment and lower purchasing costs thanks to volume discount). Furthermore, in their 
feedback  67%  of  beneficiaries  and  85%  of  relevant  stakeholders  from  the  Commission 
and national services mentioned that Hercule III enables more efficient use of resources, 

27 

 
 
 
as well as improved planning, monitoring and quality standards due to transparency and 
reporting requirements.   

5.6 Sustainability 

This  section  examines  the  likelihood  that  the  programme's  actions  will  continue 
generating positive effects after their completion.  

EQ10:  To  what  extent  are  the  (positive)  effects  of  the  intervention  likely  to  last  after 
the intervention has ended? 

Overall, it appears that the effects of the programme last after the intervention has ended.  

The external study found that the beneficiaries of the programme are of the view that the 
Hercule  III  funded  actions  will,  or  are  likely  to,  continue  producing  benefits  after  their 
completion.  Equipment  purchased  under  technical  assistance  are  used  for  several  years 
after  their  acquisition,  while  training  actions  contribute  to  acquiring  skills  and  building 
networks  likely  to  be  used  over  a  longer  period  of  time.  The  beneficiaries  of  training 
actions  are  asked  whether  they  will  keep  on  performing  the  action  funded  by  the 
programme after the EU’s financial support has come to an end for the  purposes of the 
Final Technical Report. The external study analysed 30 out of 41 of available reporting 
documents for training actions funded by grants in 2014 and 2016. The analysis showed 
that more than 50% of the reviewed training actions will not continue in the absence of 
EU funding. However, this seems to address the sustainability of the action itself, rather 
than that of its effects.  

The  experience  with  Hercule  II  also  provides  some  interesting  input  into  the 
sustainability  question  that  is  also  relevant  to  the  Hercule  III  programme.  The 
sustainability  of  the  results  under  grants  awarded  under  the  Hercule  II  programme  was 
promoted by covering maintenance as well as training costs to enable staff to operate the 
equipment acquired under technical assistance grants. Beneficiaries were asked to submit 
a final implementation report one year after the closing date of the project to demonstrate 
its effectiveness and the sustainability of the outcomes. As regards training, seminars and 
conferences,  the  participants  were  generally  confident  that  the  acquired  skills  and 
competences  can  be  put  into  practice  over  time:  sustainability  was  achieved  in  this 
respect too. This was particularly true for participants in digital forensic training.  

5.7 Contribution to smart, sustainable and inclusive growth 

The external study found that the contribution of Hercule III to the five headline targets 
identified by the Europe 2020 strategy for smart, sustainable and inclusive growth is not 
direct.  However,  protection  of  EU  financial  interests  allows  for:  i)  securing  a  larger 
amount  of  financial  resources  to  pursue  EU  initiatives  that  aim  to  achieve  the  headline 
targets  (revenues);  and  ii)  ensuring  that  such  resources  are  spent  to  foster  smart, 
to  other  ends 
sustainable  and 
(expenditures).  In  addition,  the  increased  protection  of  EU  external  borders  stemming 
from  Hercule  III  actions  contributes  to  creating  a  level  playing  field  between  EU 
businesses  and  their  competitors  based  in  third  countries,  which  is  considered  another 
key aspect to ensuring the timely achievement of Europe 2020’s targets. 

inclusive  growth  rather 

illegally  diverted 

than 

28 

 
 
6.  CONCLUSIONS 

On the basis of the answers to the evaluation questions, together with the Commission's 
findings in relation to the results of the Hercule III programme, the following conclusions 
can be drawn with regard to the evaluation criteria.  

Relevance 

Against  the  background  that  the  key  drivers  and  problems  identified  in  the  impact 
assessment (see Figure 2) for the Commission's proposal for the Hercule III programme 
are  still  prevalent  today,  in  particular  the  overarching  fraud  phenomenon,  the  external 
study  demonstrated  that  the  specific  and  operational  objectives  continue  to  be  relevant 
for the protection of the EU’s financial interests.  

A few specific actions listed under Article 8 of the Regulation (e.g. “purchase of services 
to  carry  out  chemical  analysis  of  samples  from  tobacco  and  cigarette  seizures”  and 
“purchase of services to store and destroy seized cigarettes and other counterfeit goods”) 
were  found  to  be  less  relevant  to  fulfilling  their  mission  in  relation  to  the  PFI.  This 
finding can largely be explained by the fact that these actions target a narrow group of 
possible  participants  and,  consequently,  the  bulk  of  the  participants  do  not  see  their 
relevance. To address some of these issues, the Commission is organising in 2017-2018 
training sessions in the Member States for operational staff, financed under Hercule III, 
to  increase  awareness  of  the  laboratory  facilities  for  the  chemical  analysis  of  samples 
from seized tobacco and cigarettes. 

Some  beneficiaries  suggested  a  few  ideas  of  possible  additional  operational  objectives 
which would improve the relevance of the programme, in particular in the areas of cross-
border cooperation between Member States, cooperation with non EU partners, as well as 
new  technological  developments.  To  what  extent  these  ideas  are  shared  by  the  other 
stakeholders  was  not  tested  within  the  external  study.  Their  value  is  therefore  limited. 
However,  they  are  coherent  with  the  Commission’s  experience  in  implementing  the 
programme  and  the  direction  towards  which  the  Commission  has  started  steering  the 
programme over the last few years.  

Coherence 

The  evaluation  found  that  overall  both  the  internal  and  external  coherence  of  the 
programme  were  ensured.  The  programme’s  internal  coherence  between  its  different 
components  is  guaranteed  by  the  financial  limits  which  the  Regulation  sets  for  each  of 
them,  by  balancing  the  actions  where  there  is  more  demand  and  other  actions  that  are 
particularly  relevant  to  the  PFI  (e.g.,  conferences  on  specific  PFI  topics  or  forensics 
training).  Internal  coherence  is  also  guaranteed  by  the  possibility  to  detect  overlaps 
between applications at the selection stage, as well as the requirement that the purchase 
of equipment be complemented by technical assistance activities (eg, trainings to use the 
equipment). External coherence of the programme is ensured by the Commission internal 
consultation mechanisms that allow preventing duplications and overlaps with other EU 
programmes. Such consultation takes place essentially not only at the stage of elaboration 
of the AWP, but also during the selection of applications for grants. These conclusions 
are supported by the overall perception of coherence by stakeholders, as evidenced by the 
external study. Although more synergies with  other  Commission programmes  might be 
possible,  this  needs  to  be  carefully  balanced  against  generating  delays  and  inflating 
workload.  

29 

 
Effectiveness 

The  actions  financed  under  Hercule  III  contributed  to  the  achievement  of  the 
programme’s  general,  specific  and  operational  objectives.  The  programme  financed 
actions which, because of their features, clearly contributed to achieving the objectives of 
the  programme,  for  instance  when  funding  the  procurement  of  investigation  and 
detection  tools  (eg,  devices  and  animals  to  carry  out  inspections,  monitoring  devices, 
scanners) or supporting trainings in, for example, the digital forensic area. 

A large number of training events were funded and these events covered a wide range of 
topics relevant to the PFI. The external study concludes, on the basis of the final reports 
of  the  actions  and  beneficiaries’  surveys,  that  those  actions  generated  results  that  were 
largely  in  line  with  the  expected  outputs.  Nonetheless,  only  few  actions  involved  staff 
exchanges  between  national  administrations  and  few  actions  involved  international 
participation.  The  external  study  did  also  note  that  more  should  be  done  on  the 
expenditure  side  of  the  EU  budget,  which  is  an  area  that  has  received  relatively  fewer 
grants. 

With regards to technical assistance, the timing for the evaluation, i.e. early in the life of 
the programme, made it challenging to further consolidate the above findings about the 
effectiveness of the programme.  

Among the  many external factors interfering  with the achievement of the programme’s 
objectives,  the  ability  of  fraudsters  to  “adapt  quickly  to  new  circumstances  at  the  EU 
level, operate cross-border and exploit the weakest points in the external borders” was 
perceived as the most important. Such external factors tend to harm the PFI and require 
continued efforts by national authorities to combat and prevent fraud.  

Efficiency 

The  external  study  shows  that  the  costs  generated  by  the  programmes’  training  actions 
are  in  line  with  other  EU  programmes,  be  it  in  terms  of  the  quantity  of  person-days 
necessary  at  the  different  stages  (preparation  of  an  application,  signing  of  the  grant 
agreements,  and  reporting),  of 
the  acquisition  and 
implementation of actions under grant agreements, or the average costs per participant in 
training actions. 

transaction  costs  for 

the 

Cost-effectiveness  is  inherent  to  the  implementation  of  technical  assistance  actions  as 
they  mostly  consist  of  the  acquisition  of  equipment  that  has  to  be  purchased  by  public 
procurement  procedures  where  the  "best  value  for  money"  principle  is  used  to  select 
equipment.  However,  the  results  obtained  following  the  deployment  of  the  equipment 
during PFI related operations is difficult to assess for lack of data. 

On the basis of the external study’s findings, the application and reporting stages do not 
appear  to  be  too  cumbersome  for  a  large  majority  of  stakeholders.  Some  of  the  critical 
applicants  made  a  number  of  suggestions  that  are  currently  in  the  process  of  being 
addressed.  However,  only  part  of  these  suggestions  could  be  addressed,  as  some 
regulatory obligations are inherent to EU financial and procurement rules and cannot be 
simplified  given  the  current  legal  framework.  The  Commission  has  started  using  a 
system for the electronic submission of applications for grants and for the management of 
grant  agreements.  The  improvements  brought  about  by  the  electronic  management 
system  will  address  a  large  number  of  the  applicants’  suggestions,  for  instance  by 
digitising the whole process and introducing the use of digital signatures. The electronic 

30 

 
system is expected to noticeably alleviate the administrative burden and related time and 
resources spent on the applications in particular.  

EU added value 

The  external  study’s  results  demonstrated  that  the  benefits  from  the  programme  would 
not materialise without the support of Hercule III or equivalent EU funding. The actions 
allowed  for  a  better  protection  of  the  Union's  external  borders.  In  addition,  the 
programme  is  an  enabler  for  cross-border  cooperation,  in  particular  cross-border 
exchanges of information and best practices. The training actions facilitated networking 
opportunities  between  the  participants  to  the  events  thus  contributing  to  strengthening 
cross-border  cooperation.  The  stakeholders  almost  unanimously  recognised  the  added 
value of the programme and its essential contribution to generating benefits. 

The  external  study  also  revealed  that  the  programme's  transparency  and  reporting 
requirements  contribute  to  the  improvement  of  the  beneficiaries’  planning,  monitoring 
and  quality  standards  and  to  the  quality  of  the  beneficiaries'  financial  management  in 
general.  

Sustainability 

There is a consensus among beneficiaries that the actions funded under the programme 
will  continue  producing  benefits,  even  after  their  completion.  This  was  particularly 
relevant  for  technical  assistance  actions  where  the  deployment  of  equipment  will 
continue generating results beyond the closing date of a grant agreement, but also applied 
to  training  actions,  where  the  beneficiaries  were  generally  positive  about  the  lasting 
effect  of  networking  and  exchanges  of  information  and  best  practices.  Nonetheless,  a 
substantial  number  of  beneficiaries  indicated  that  they  would  discontinue  the  activities 
funded under the programme, if it were to be terminated. This finding suggests that the 
discontinuation of the programme would negatively impact the overall protection of the 
financial interests of the Union. 

Contribution to smart, sustainable and inclusive growth 

The evaluation found that the programme contributes indirectly to EU’s strategic goal for 
smart,  sustainable  and  inclusive  growth,  in  particular  by  protecting  the  financial 
resources that are used to achieve the priority’s goals (revenues) and ensuring that such 
resources are spent to foster smart, sustainable and inclusive growth rather than illegally 
diverted  to  other  ends  (expenditures).  The  increased  protection  of  EU  external  borders 
supported  by  the  programme  also  contributes  to  creating  a  level  playing  field  between 
EU businesses and their competitors based in third countries. 

31 

 
 
Annex 1: Procedural information 

This  annex  provides  procedural  information.  It  explains  how  the  evaluation  was 
managed both in terms of organisation and time. It provides information about external 
expertise used.  

A1.1  Lead DG:  

European Commission, European Anti-Fraud Office (OLAF). 

A1.2  Organisation: 

The  evaluation  roadmap48  was  published  in  August  2016  and  sets  out  the  purpose  and 
scope  of  the  exercise.  The  roadmap  presented  the  questions  to  be  addressed  under  the 
five categories of effectiveness, efficiency, relevance, coherence and EU added value.  

An Inter-Service Steering Group (ISSG) was set up to support the evaluation. The tasks 
of the ISSG were to check the key elements of the external study, to support and monitor 
the  evidence  gathering  and  stakeholder  consultation  process,  to  review  all  external 
deliverables  for  the  evaluation  as  well  as  the  SWD  and  to  assist  with  the  quality 
assessment of the consultant's evaluation report. 

The  ISSG  was  composed  of  the  Secretariat-General  of  the  Commission,  the  Legal 
Service and DGs TAXUD, JUST, HOME and BUDG.  

The  Commission  contracted  out  an  external  study  in  support  to  the  evaluation  (see 
below), which provided the basis for this SWD. 

A1.3  Timetable: 

Date 
June 2016 
5 July 2016 
09 September 2016 

25 October 2016 

Description 
Establishment of the Steering Group  
1st Steering Group Meeting: Evaluation roadmap and ToR 
Specifications for study contract to support the evaluation sent to 
the framework contractor 
2nd Steering Group Meeting: Kick off meeting with the external 
contractor 
Award  of  a  study  contract  and  signature  of  the  contract  with 
CEPS on behalf of the consortium 
3rd Steering Group Meeting: Inception report meeting 

12 December 2016 
February - June 2017  Targeted stakeholder consultation –surveys and interviews 
30 June 2017 
4 September 2017 
24 October 2017 
January 2018  
Source: OLAF  

4th Steering Group Meeting: End-of-fieldwork meeting 
5th Steering Group Meeting: External Draft final report meeting 
6th Steering Group Meeting: 1st draft Staff Working Document  
Adoption of the report accompanied by the SWD 

48   http://ec.europa.eu/smart-regulation/roadmaps/docs/2017_olaf_002_midterm_evaluation_en.pdf 

32 

 
 
 
                                                            
A1.4  Exceptions to the Better Regulation Guidelines: 

Given the scope of the Regulation and the well-defined group of stakeholders, targeted 
consultations were considered as an appropriate and proportional means of consultation 
rather than an open public consultation. Considering that the public at large could not be 
considered as directly impacted by the provisions of the Regulation, responsible for their 
application,  or  possessing  specific  evidence  needed  for  the  evaluation,  an  open  public 
consultation was not carried out. 

A1.5  Consultation of the Regulatory Scrutiny Board: 

The evaluation has not been selected for scrutiny by the Regulatory Scrutiny Board. 

A1.6  External Expertise: 

An  external  independent  study  is  the  basis  for  the  conclusions  presented  in  this 
document.  A  contract  was  signed  on  25  October  2016  with  Economisti  Associati-led 
consortium (external contractor)49.  

The external contractor carried out targeted consultations for the evaluation as explained 
in Annex 2.

49   Using  the  framework  contract  n°  2015/CC/132  concluded  by  DG  TAXUD  for  the  "provision  of 

evaluation and Impact Assessment related services". 

33 

 
 
 
                                                            
Annex 2: Synopsis report of stakeholder consultation in the 
context of the Study 

This  annex provides  a  synopsis  of the  stakeholder  consultation that  was  carried  out  for 
this evaluation by an external contractor as part of the external support study. It presents 
the main steps and findings of the consultation of interested parties and stakeholders. It 
shows  that  the  stakeholder  consultation  process  complied  with  the  Commission's 
stakeholder  consultation  principles  and  minimum  standards  taking  into  account 
proportionality as required by the Better Regulation Guidelines50. 

A1.1 

Feedback on evaluation roadmap 

The evaluation roadmap was opened for feedback during the course of the evaluation. No 
feedback was received in this period.  

A1.2  Open public consultation 

Regulation 250/2014 established the Hercule III programme to promote activities in the 
field of the protection of the financial interests of the European Union. The scope of this 
mid-term evaluation covers the interventions supported under the Hercule III programme 
to  date  as  well  as  the  preparatory  and  implementing  activities  undertaken  by  the 
stakeholders for these interventions. 

The groups of stakeholders it targets and impacts are well-defined, i.e. for the most part 
administrators and law enforcement officers at national and regional level. The public at 
large  cannot  be  considered  as  directly  impacted  by  the  provisions  of  the  Regulation  or 
responsible  for  their  application  or  possessing  specific  evidence  that  would  usefully 
contribute  to  the  evaluation  of  the  application  of  the  Regulation.  Therefore,  an  open 
public consultation was not carried out and stakeholders were consulted in targeted 
consultations as explained below. 

A1.3  Targeted consultations – overview of stakeholders consulted  

This section presents an overview of different stakeholder groups consulted via in-depth 
interviews and a survey.  

A total of 574 stakeholders were consulted through interviews and survey, comprising 
16  officials  of  Commission  or  national  services,  56  beneficiaries  (i.e.  71%  of  all 
beneficiaries), 67 unsuccessful applicants (i.e. 25% of all unsuccessful applicants), 321 
participants  in  events  (i.e.  27%  of  all  participants  invited  to  reply)  and  112  users  of 
services  (i.e.  31%  of  all  users  invited  to  reply).  Whereas  49  stakeholders  (i.e.  all 
officials  from  institutions  and  33  beneficiaries)  were  interviewed,  the  remaining 
stakeholders participated in the online surveys. 

50  

Better Regulation Guidelines, p. 70. 

34 

 
 
 
                                                            
European Commission and national officials 

Furthermore,  exploratory  interviews  were  conducted  with  several  OLAF  officials  and 
with  officials  from  DG  BUDG,  DG  JUST,  DG  TAXUD.  These  interviews  were 
additional  to  the  16  interviews  with  the  Commission  and  national  services  conducted 
during the Fieldwork phase, which were divided as follows: five interviews with OLAF 
officials (Hercule Sector, Internal Auditor, units B1, C3, and D4); two interviews with 
officials from other DGs (DG JUST and DG TAXUD); nine interviews with officials 
from  national  institutions  (eight  AFCOS  which  are  also  members  of  the  Advisory 
Committee  for  the  Coordination  of  Fraud  Prevention  (COCOLAF)51;  and  one  central 
customs  authority)  in  Austria,  Bulgaria,  France,  Germany,  Italy,  Lithuania,  Poland, 
Romania and Slovenia.  

Beneficiaries 

In total, 56 beneficiaries from 18 Member States were consulted52, i.e. 71% of those 
that were awarded a grant either in 2014 or 2015 (Figure 1).  

Figure 1. Consulted beneficiaries by Member States and category of action (number 
of respondents) 

3 

4 

2 

1 

3 

6 

2 

3 

4 

1 

4 

3 

2 

1 

3 

2 

1 

2 

2 

2 

1 

1 

1 

1 

1 

IT

FR RO LT

PL

SK HU BG DE

EL

BE NL SE CY

FI HR LV MT

Anti-fraud training

Legal training and studies

Technical Assistance

Unsuccessful applicants 

All  entities  that  applied  without  success  to  the  Hercule  III  calls  for  proposals  in  2014, 
2015  and  2016  were  invited  to  participate  in  the  online  consultation.  67  respondents 

51  COCOLAF’s  mission  is  “to  advise  the  Commission  on  any  matter  relating  to  the  prevention  and 
prosecution  of  fraud  and  all  other  illegal  activities  adversely  affecting  the  financial  interests  of  the 
Community,  and  on  any  matter  relating  to  cooperation  between  the  competent  authorities  of  the 
Member States or between Member States and the Commission to protect the financial interests of the 
Community, in order to organise more effectively close and regular cooperation between the competent 
see: 
fraud”. 
authorities 
http://ec.europa.eu/transparency/regexpert/index.cfm?do=groupDetail.groupDetail&groupID=656 

counter 

further 

details 

For 

to 

52  Interviews  were  conducted  with  beneficiaries  from  Belgium,  Bulgaria,  France,  Italy,  Lithuania,  the 

Netherlands, Poland and Romania. 

35 

 
 
 
                                                            
from 21 Member State completed the survey, i.e. 25% of all unsuccessful applicants 
during  the  first  three  years  of  the  programme  (Figure  2)53.  Almost  50%  of  respondents 
were  located  in  Poland  and  Romania;  this  is  in  line  with  data  registered  in  the  total 
population, as 121 out of 267 (45%) unsuccessful applications during the first three years 
of the programme were submitted by Polish and Romanian entities.  

Figure 2. Consulted unsuccessful applicants by Member States and category of 
action (number of respondents) 

13  14 

3 

2 

2 

2 

3 

1 

4 

3 

2 
1 

2 

1 
1 

2 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

PL RO BG ES HU EE LV CZ LT SI BE CY DE DK EL HR IE IT MT SE UK

Anti-fraud training

Legal training and studies

Technical Assistance

Events participants 

312 events participants from 25 Member States and nine from third countries completed 
the online survey (Figure 3).  

53   Six  invitations  to  participate  in  the  survey  were  bounced,  as  the  email  address  indicated  in  the 

application form was either incorrect or obsolete. 

36 

 
  
 
 
                                                            
Figure 3. Consulted participants by country of origin (number of respondents) 

47 

43 

40 

29 

24  23 

16 

13 

10  10  9  8  8 

6  5  5  4  4  4  3  3  2  2  1  1  1 

Unit: Number of respondents. 
Note: 321 total respondents. 
Source: Authors’ elaboration on online survey with participants in events. 

About one-third of the respondents took part in events arranged in a different MS from 
their country of residence (Figure 4). The three largest groups of participants by type of 
organisation  included  customs  officials,  police  officials  and  academic  audiences 
(Figure 5).  Interestingly,  many  respondents  had  participated  in  more  than  one  type  of 
event in the context of the same action funded by Hercule III (Figure 6). The reason is 
that  several  actions  combined  different  types  of  event  to  achieve  the  expected  results. 
Respondents  were  aged  mainly  between  25  and  54,  with  women  representing  less  than 
40% of the sample (Table 1).  

Figure 4. Participants in events held in the same Member State and in a different 
Member State 

33% 

67% 

National Participation

Cross Border Participation

Note: 321 respondents. 

37 

 
 
 
 
 
Figure 5. Consulted participants by type of organisation 

Law enforcement: Customs

Academic audience

Law enforcement: Police

State revenue service

Ministry of Finance

Local/regional authorities

NGO representatives

Legal practitioners: Lawyers,
solicitors, barristers
Ministry of Interior/Home affairs

Other

27% 

3% 

4% 

5% 

20% 

12% 

9% 

6% 

7% 

7% 

Note: 321 respondents. 

Figure 6. Consulted participants by type of event attended 

147 

94 

91 

23 

21 

20 

29 

Conference

Training
course

Seminar

Round-table Working

Meeting

Other

group

Note: 321 respondents; each respondent may have participated in more than one event in the 
context of the same action funded by Hercule III.  

Table 1. Consulted participants by age category and gender 

Age category 

Below 25 

25-34 

35-44 

45-54 

55-64 

65 

Total 

Female 

1.3% 

9.1% 

14.4% 

9.1% 

4.4% 

0.0% 

38.1% 

Source: External study 

Prefer not to disclose 

0.0% 

0.6% 

0.3% 

0.3% 

0.0% 

0.0% 

1.3% 

Total 

2.5% 

19.4% 

37.5% 

27.8% 

11.6% 

1.3% 

100.0% 

Gender 

Male 

1.3% 

9.7% 

22.8% 

18.4% 

7.2% 

1.3% 

60.6% 

38 

 
 
 
 
 
For  confidentiality  reasons,  relevant  contact  details  for  users  of  services  procured  by 
Hercule  III  were  not  provided  to  the  external  contractor;  hence,  OLAF  sent  the 
invitations  to  complete  the  online  survey  to  users.  Against  this  background,  based  on 
information provided by OLAF, 352 users were invited to complete the survey, which 
was  eventually  completed  by  112  respondents  (32%  response  rate;  Table  2).  Users 
were  mainly  based in Germany, the UK, Belgium and Italy (Figure 7). Almost  70%  of 
respondents are users of statistics and IT tools; 30% are users of databases (e.g. NTELX, 
GTI,  GRS,  D&B  reporting  system,  SEASEARCHER,  GTA);  only  6%  are  users  of 
services to carry out chemical analysis of samples from tobacco and/or cigarette seizures 
(Figure 8). 

Table 2. Share of total population represented by consulted beneficiaries, 
unsuccessful applicants, participants in events and users of services  

Type of stakeholder 
Total beneficiaries 
Beneficiaries TA 
(2014-15) 
Beneficiaries AFT 
(2014-15) 
Beneficiaries LTS 
(2014-15) 
Total applicants 
Applicants TA 
(2014-16) 
Applicants AFT 
(2014-16) 
Applicants LTS 
(2014-16) 
Participants 
(2014-15) 
Users 
(2014-16) 

Total sample 
56 

Total population 
79 

Sample / population 
71% 

25 

21 

10 

67 

56 

5 

6 

321 

112 

38 

25 

16 

267 

174 

61 

32 

1,194* 
(3,400)** 

352* 

66% 

84% 

63% 

25% 

32% 

8% 

19% 

27%* 
(9%)** 

32%* 

Note: *Participants/users invited to complete the online survey. **Participants in events 
funded during the first two years of the programme according to the Annual Implementation 
Reports. 

As  regards  the  type  of  organisation,  the  lion’s  share  of  respondents  belongs  to  the 
customs  category  (Figure  9).  No  information  is  available  with  regard  to  the  total 
population of users; hence, no conclusion can be drawn about the share of the population 
represented by the sample. 

39 

 
 
 
Figure 7. Consulted users by country and type of service (number of respondents) 

5 

1 

9 

9 

4 

5 

1 

2 

8  4 

2 

3 

5 

4 

3 

2 

1 
1 
1  1  1  1 

4  4 

3 

1 

2 

2 

1 

3 

2  2  2 

1  1  1 

2  2 

1 
1 

1  1  1 

1 

DE UK BE IT SI PL RO SK HU IE NL SE BG CZ DK FR CY ES PT FI EL MT EE HR LT AT

User of statistics and IT tools

User of chemical analysis of tobacco and cigarettes

User of databases

Note: 112 total respondents. 

Figure 8. Consulted users by type of service  

26% 

6% 

68% 

User of databases

User of chemical analysis of tobacco and cigarettes

User of statistics and IT tools

Note: 112 respondents. 

40 

 
 
 
 
Figure 9. Consulted users by type of organisation 

7% 

4% 

1% 

1% 

Law enforcement: Customs Law enforcement: Police

Local/regional authorities

National Authorities

Other

88% 

Note: 112 respondents.

41 

 
 
Annex 3: Methods and analytical models 

This annex provides a description of the methodological approach to the evaluation taken 
by  the  external  contractor  that  supported  this  SWD.  It  summarises  the  main 
methodological elements and cross-references to the more detailed methodological annex 
of the external report supporting this SWD. 

A1.4  Logic of intervention  

The  intervention  logic  developed  in  the  context  of  the  evaluation  includes  a  detailed 
description  of  the  needs  and  problems  that  the  programme  is  supposed  to  address,  the 
objectives that the programme is expected to achieve, the activities that the programme 
intends  to  implement,  the  expected  results  of  the  programme  and  the  logical  links 
between these various components. A comparison between the expected results (i.e. how 
the  programme  was  intended  to  work)  and  the  actual  results  stemming  from  the 
intervention  under  analysis  (i.e.  how  the  programme  worked  in  reality)  enables  a  mid-
term evaluation based on evidence. Against this background and in line with the Better 
Regulation  Guidelines,  the  following  components  of  the  intervention  logic  were 
identified and described: 

  The rationale for the intervention 

  Needs and problems 

  Objectives (general, specific, operational) 

  The intervention 

 

Inputs/activities 

  The expected results of the intervention 

  Outputs (expected) 

  Outcomes (expected) 

 

Impacts (expected) 

A1.5  Evaluation Framework  

The  evaluation  framework  serves  the  purpose  of  guiding  the  mid-term  evaluation  and 
includes the following items: 

  evaluation criteria; 

  evaluation questions; 

42 

 
 
 
 
 
 
 
 

 

judgment criteria; 

indicators; 

  data sources; 

  data collection methods. 

The six evaluation criteria are generally used to evaluate EU policies and are enshrined in 
the 2015 “Better Regulation Guidelines”54 and the “Better Regulation Toolbox”55. These 
criteria  are  made  more  specific  through  evaluation  questions.  Judgment  criteria, 
expressed in the form of statements are used to answer the various evaluation questions. 
They  can  be  then  confirmed  and/or  rejected  by  the  research.  A  comprehensive  set  of 
indicators  serve  the  purpose  of  applying  judgment  criteria  and  have  been  chosen 
according  to  the  RACER  framework56.  Data  sources  are  the  sources  of  evidence  on 
which the external study relies to apply the judgment criteria and answer the evaluation 
questions.  Data  collection  method  is  the  approach  adopted  to  gather  the  required 
evidence. 

A1.6  Data collection  

Primary  data,  i.e.  data  that  were  collected  specifically  for  the  purpose  of  the 
Assignment via: 

 

interviews  with:  i)  Commission  staff;  ii)  national  institutions  active  in  fraud 
prevention and PFI; and iii) beneficiaries of actions funded by the programme; and 

  online surveys with: i) beneficiaries of actions funded during the first two years 
of  the  programme;  ii)  unsuccessful  applicants;  iii)  participants  in  events  co-
financed by Hercule III; and iv) users of services procured by Hercule III.  

Secondary data, i.e. data gathered by consulting: 

  public available sources including, inter alia, the Regulation and accompanying 
material  (e.g.  the  IA),  AWPs,  Annual  Implementation  Reports,  Article  325 
reports, final evaluation reports and interim reports of the Hercule II programme 
and legal texts establishing other programmes (e.g. Customs 2020, Fiscalis 2020, 
Internal Security Fund); and 

54  European  Commission,  Better  Regulation  Guidelines,  19  May  2015, 
http://ec.europa.eu/smart-regulation/guidelines/docs/swd_br_guidelines_en.pdf. 

available 

at: 

55  European  Commission,  Better  Regulation  “Toolbox”,  19  May  2015, 

available 

at: 

http://ec.europa.eu/smart-regulation/guidelines/docs/br_toolbox_en.pdf. 

56   The  RACER  framework  (see  European  Commission,  Better  Regulation  “Toolbox”,  19  May  2015) 
requires  adopting  indicators  that  are:  i)  relevant,  i.e.  closely  linked  to  the  EQ  as  operationalised 
through  the  judgment  criteria;  ii)  accepted,  i.e.  retrieved  from  relevant  literature  or  best  evaluation 
practices; iii) credible, i.e. easy to interpret and unambiguous, especially in view of drawing evidence 
for policy-making; iv) easy to monitor, i.e. measurable at a low cost; and v) robust, which is equivalent 
to saying they cannot be manipulated by the regulators or regulated subjects. 

43 

 
 
 
 
                                                            
  official  documentary  evidence  on  both  submitted  proposals  (i.e. 
application  forms)  and  awarded  contracts  and  grants  (i.e.  feedback  from 
grant beneficiaries in the form of Final Technical Reports, Final Financial 
Reports and Final Implementation Reports (limited to technical assistance 
actions)). 

A1.7  Evaluation challenges and limitations 

This section describes the main challenges encountered and the associated limitations of 
the  evaluation  results  and  mitigation  measures  put  in  place.  In  doing  so  it  provides  an 
overall assessment of the robustness of the methodology applied and the reliability of the 
available data. 

  The population coverage ensured by the consulted national institutions (Member 
States  receiving  more  than  60%  of  grants  awarded  in  2014  and  2015,  plus  two 
MS that did not receive any grant) and beneficiaries (71% of the total population) 
allows to draw robust conclusions for all evaluation questions.  

the  Hercule  III  programme,  which  requires 

  However,  the  shares  of  total  population  represented  by  unsuccessful  applicants 
(25%), participants in events (27% of invited participants, 9% of total population) 
and users of services (32% of invited users) are lower. Therefore, more caution is 
required when interpreting conclusions based on these surveys. In order to ensure 
the  highest  quality  of  the  findings:  i)  all  evaluation  questions  are  addressed  by 
combining  feedback  from  more  than  one  stakeholder  category;  ii)  whenever 
possible,  data  and  information  collected  from  stakeholders  are  compared  with 
evidence  retrieved  from  application  forms  and  reporting  documents  in  order  to 
ascertain consistency across data sources consulted to perform the evaluation. 
  The  timing  of  the  mid-term  evaluation  is  compliant  with  Article  13  of  the 
Regulation  establishing 
the 
Commission  to  present  an  Independent  Mid-Term  Evaluation  Report  to  the 
European  Parliament  and  the  Council  by  31  December  2017.  Against  this 
background, the evaluation is confined to actions funded during the first two 
years  of  the  programme  (2014  and  2015).  As  regards  2016,  the  analysis  is 
limited  to  applications  received  in  the  context  of  calls  for  proposals;  in  fact,  no 
action  co-financed  by  Hercule  III  grants  for  2016  calls  was  completed  at  the 
moment  of  conducting  fieldwork  activities.  Some  actions  covered  by  the  2014 
and  2015  budget  were  still  ongoing  during  fieldwork;  hence,  their  impact  can 
only  be  recorded  to  a  limited  extent.  Due  to  the  unexpected  high  number  of 
applications, some delays in awarding the grants and finalising the funded actions 
were  registered.  In  addition,  based  on  information  retrieved  from  application 
forms,  half  of  the  technical  assistance  actions  funded  in  2014  and  2015  were 
expected  to  last  more  than  12  months.  These  data  limitations  impinge  on  the 
assessment of both outcomes and impacts:  

  With  regard  to  outcomes,  13  out  of  25  technical  assistance  actions 
surveyed  for  this  Assignment  were  still  ongoing  at  the  moment  of 
gathering  data  and  information  to  assess  the  effectiveness  of  the 
programme. In addition, for completed actions, most beneficiaries argued 
that they only recently received the devices, tools and systems co-financed 

44 

 
 
 
 
by  Hercule  III.  As  a  consequence,  whereas  most  of  the  beneficiaries  of 
technical  assistance  actions  provided  output  indicators  (e.g.  type  and 
number of items purchased with Hercule III funds), evidence on outcome 
indicators  (e.g.  successful  operations,  number  of  arrests,  convictions 
seizures, etc.) is scant. Therefore, limited conclusions can be drawn about 
the  alignment  between  actual  and  expected  outcomes  of  technical 
assistance actions.  

  Impacts  occur  in  the  long-term  and  cannot  be  captured  by  a  mid-
term evaluation. In principle, in the long-run, the Hercule III programme 
is  expected to  improve the PFI, thus ensuring  the competitiveness of the 
European economy and the protection of taxpayers’ money. It is too early 
to  measure  such  impacts,  especially  considering  that  the  first  actions 
funded by the programme were only implemented in 2015.  

  Available  primary  and  secondary  data  and  information  mainly  allow 
evaluating actions  funded via  grant  agreements, which represent  75%  of 
the overall commitments in 2014 and 2015. The evaluation of databases, IT 
tools  and  analyses  of  cigarettes  and  tobacco  samples  funded  under 
procurement is performed on the basis of feedback provided by respondents 
to the online survey with users; hence, all TA actions57 have been assessed. It 
was  not  possible  to  collect  evidence  from  the  contractors  directly  due  to 
confidentiality,  data  protection  and  contractual  reasons  in  case  of  the 
evaluation of conferences, digital forensics training, studies and other actions 
procured  by  Hercule  III.  Therefore,  OLAF  contacted  the  contractor  to 
distribute the links to the online surveys and to request additional information 
where necessary (e.g. satisfaction surveys). Such actions represent less than 
10%  of  the  overall  commitments  in  the  first  two  years  of  the  programme. 
Therefore, the Assignment covers the bulk of the programme. 

  The  external  evaluator  did  not  receive  the  contact  details  for  event 
participants  and  users  of  services  procured  by  Hercule  III  due  to 
confidentiality and data protection reasons.  

With  regard  to  the  survey  with  participants,  OLAF  requested  beneficiaries  of  anti-
fraud and legal training grants to invite participants in the events they arranged to 
complete the online survey. This approach may lead to biased results as beneficiaries 
might have distorted incentives, e.g. to share introductory emails only with a limited set 
of  “satisfied”  participants  or  not  to  support  the  survey  at  all,  especially  in  case  they 
believe there is a risk of negative feedback. It was not possible to control for this type of 
bias. Whenever possible, a consistency check58 was performed between data collected via 
the  online  survey  with  participants  and  data  retrieved  from  Final  Technical  Reports  of 

57   TA actions funded via grant agreements and procurement contracts represent 80% of the commitments 

in 2014 and 2015. 

58   A  consistency  check  aims  to  detect  whether  the  value  of  two  or  more  data  items  are  not  in 
contradiction  (UN  Statistical  Commission,  UNECE,  2000.  Glossary  of  Terms  on  Statistical  Data 
Editing). 

45 

 
 
 
                                                            
anti-fraud training actions. These include, inter alia, the participants’ assessment of each 
event  with  replies  provided  by  about  3,000  participants  (i.e.  almost  90%  of  all 
participants  Table  1)59.  In  addition,  to  facilitate  future  evaluations,  the  external  study 
suggests  including  additional  reporting  requirements  for  beneficiaries  of  actions 
involving  the  organisation  of  events:  beneficiaries  should  provide  OLAF  with  email 
addresses of all participants in events, accompanied by a data protection waiver allowing 
OLAF and OLAF’s contractors to contact the participants for evaluation purposes.  

Similarly, OLAF was in charge of sending the invitations to complete the online survey 
to  users of the  services funded by  Hercule III. While this approach  did not  lead  to any 
potential bias, information about the total population of users is unavailable.  

Overall, the contractor declared that the planned data collection and analytical exercises 
could  be  implemented  as  planned.  Most  of  the  challenges  were  overcome  and/or 
mitigated against. The robustness of the evidence base gathered was judged satisfactory 
by the evaluators. 

59   Whereas  the  survey  with  participants  allows  gathering  the  stakeholders’  perceptions  a  few  months 
after the event was held, participants’ assessments included in Financial Technical Reports are based 
on data gathered during the event. Therefore, the latter does not allow capturing whether participants, 
for example, used the skills acquired during the event in their work activities or interacted with people 
they met at the event after the event ended. 

46