Commission staff working document - Developing the tolerable risk of error concept for the research, energy and transport policy area - Accompanying document to the Communication from the Commission to the European Parliament, the Council and the Court of Auditors - Commission Communication: More or less controls? Striking the right balance between the administrative costs of control and the risk of error {COM(2010) 261 final} {SEC(2010) 640}
Date: 2010-06-02
Institutions: European Commission
Subjects: EU expenditure, budgetary control, cost analysis, general budget (EU), implementation of the budget

EN 

EN 

   EN 

 
 
EUROPEAN COMMISSION 

Brussels, 26.5.2010 
SEC(2010) 641 

COMMISSION STAFF WORKING DOCUMENT 

Developing the tolerable risk of error concept for the research, energy and transport 
policy area 

Accompanying document to the 

COMMUNICATION FROM THE COMMISSION TO THE EUROPEAN 
PARLIAMENT, THE COUNCIL AND THE COURT OF AUDITORS 

Commission Communication: More or less controls? Striking the right balance between 
the administrative costs of control and the risk of error 

{COM(2010) 261 final} 
{SEC(2010) 640} 

EN 

   EN 

 
 
 
  
 
 
 
 
 
 
1. Introduction 

The  purpose  of  this  document  is  to  present  the  technical  aspects  underlying  the 
Commission's  proposal  for  the  tolerable  risk  of  error  (TRE)  for  the  research,  energy  and 
transport policy area. It sets out statistical models for analysing the relationship between costs 
of controls and error rates and explains the results derived there from. 

Part  2  of  the  paper  briefly  outlines  the  specific  characteristics  of  the  research,  energy  and 
transport control environment that needed to be taken into account in the modelling and to 
inform the decision on TRE.  

In Part 3 follows the presentation of the dataset used in the model and its limitations.  

Part 4 explains the methodology established in order to collect costs of control data as well 
as the results of this exercise. This part gives an overall description of the current situation as 
regards costs dedicated to control activities in the research, energy and transport policy 
area  whereby  control  is  understood  in  a  broad  sense  comprising  activities at  all  stages  of  a 
project  cycle  starting  from  the  selection  of  beneficiaries  until  the  implementation  of  audit 
results through the recovery of any undue payments. Such activities are classified as “control 
activities” where they result in decisions which influence the eligibility of an activity and/or 
of certain of its costs1. This definition encompasses the evaluation of proposals (selecting the 
projects  to  be  financed,  ensuring  they  meet  eligibility  criteria  and  the  achievement  of  the 
objectives  as  adopted  by  Council  and  Parliament);  the  detailed  negotiation  of  the  grant 
agreement  (the  translation  of  each  of  the  retained  scientific  proposals  into  a  legally  binding 
instrument  including  both  scientific  and  financial  aspects);  the  management  of  the  project 
(verifying  the  outputs  which  demonstrate  the  state  of  advancement  of  the  project  and 
influence the eligibility of costs), the control of all aspects of cost claims and the control on 
the spot of projects and the associated costs of the recovery of any amount due as a result of 
the errors detected.   

Part 5 provides an analysis of the errors found in the research, energy and transport policy 
group  by  the  Court  of  Auditors  in  recent  years.  It  presents  the  classification  of  errors 
according to their causes and draws conclusions there from. 

Finally  in  Part  6  two  concrete  TRE  models  developed  by  the  Commission  are  introduced 
together  with  their  underlying  assumptions.  The  models  are  used  to  illustrate  a  number  of 
scenarios  for  the  relationship  between  costs  of  controls  and  errors  in  order  to  derive  TRE 
range there from. 

1 The Commission's internal control systems are based on the internationally-accepted COSO model which 
defines internal control as "a process designed to provide reasonable assurance regarding the achievement of 
objectives in effectiveness and efficiency of operations, reliability of financial reporting, and compliance with 
applicable laws and regulations". (http://www.coso.org) 

EN 

2 

   EN 

 
 
 
                                                 
 
2. Specific characteristics of the research, energy and transport control environment  

Grant  expenditure  in  the  research,  energy  and  transport  area  is  subject  in  the  main  part  to 
centralised management by the Commission, either directly by the Commission's Directorates 
General,  or  indirectly  through  executive  agencies  or  joint  undertakings.  This  implies  the 
concentration of supervision and control activities and resources within the Commission and 
it can be therefore concluded that the majority of control costs in this area are borne by the 
Commission.  This  was  certainly  the  case  in  2009  when  the  data  were  collected  for  this 
exercise  as  the  two  new  research  executive  agencies  (ERCEA  and  REA)  and  several  joint 
undertakings2 were at the capacity-building stage. 

The Commission relies on project coordinators to distribute funds to project participants and 
to consolidate their financial and scientific reporting on the implementation of the project to 
the  Commission3.  Equally,  the  Commission  relies  on  the  accounting  and  internal  control 
systems of the beneficiaries to ensure the accuracy and regularity of payments, carrying out 
itself desk-based checks and sample controls on the beneficiaries' premises. In consequence, 
the Commission, coordinators and beneficiaries may bear non-negligible costs linked to these 
controls4.  These  have  not  been  considered  for  the  assessment  of  the  cost  of  controls.  If  the 
Commission  was  required  to  authorise  the  payments  to  each  beneficiary,  the  number  of 
payments would be roughly multiplied by 7 (7.29 is the average number of beneficiaries per 
FP6 project) which would render the scheme unmanageable. 

Some three quarters of the expenditure in this area is via the successive research framework 
programmes  while  the  rest  is  spent  through  the  Trans-European  Networks,  Nuclear 
Decommissioning and other grant activities. The Commission's main instruments to provide 
research grants to beneficiaries are multiannual funding schemes – Framework Programmes 
that  are  based  on  Union  pre-financing  which  is  then  “cleared”  by  actual  costs  declared  by 
beneficiaries, which can be a year or more after the pre-financing: for this reason eligibility of 
costs can be finally checked only after the execution of the activity (through checks based on 
self declarations of contractors and – for the more detailed eligibility controls- on the spot at a 
beneficiary’s premises). 

Rules  for  participation  in  Framework  programmes  define  the  conditions  to  be  met  by  cost 
claims in order to be considered eligible costs and additional guidance assists beneficiaries to 
meet  the  requirements.  However,  the  system  of  reimbursement  of  costs  means  that  a  very 
detailed level of proof is requested from beneficiaries (for example, personnel costs based on 
detailed  record-keeping  of  the  hours  spent  by  each  staff  member  on  a  project  and  on  the 
detailed recording of all salary and other costs actually paid to the individual concerned). As 
recognised  by  the  Court  of  Auditors,  such  requirements  mean  that  beneficiaries’  cost 
statements are inherently error prone. 

In practice, given the complexity of the control of eligible costs, the fact that the payments are 
made  on  the  basis  of  self-declaration  where  no  supporting  documents  are  requested  in 
advance means that detailed control of cost claims is most often carried out on the spot (on a 
sample basis) where the auditor has access to all of the necessary supporting information. To 
this  extent  Commission  services  have  established  a  resource  intensive  on-the-spot  control 

2 IMI, Artemis, Clean Sky, ENIAC, FCH. 
3 Hereinafter we will refer to both coordinators and project participants as "beneficiaries". 
4 Part of such costs may be reimbursed in practice as overheads. 

EN 

3 

   EN 

 
                                                 
strategy  having  a  multiannual  control  objective  to bring the  residual level of  error from on-
the-spot controls below 2% at the end of FP6 (end of 2010). In spite of major control efforts, 
as  indicated  in  2009  Annual  Activity  Reports  of  the  relevant  DGs,  and  based  on  current 
information, it is not certain that this objective can be reached: the current multi-annual error 
rate for FP6 reported by the concerned DGs in their 2009 Annual Activity Reports is higher 
than 2% even taking account of the fact that nearly 1,900 FP6 audits have been carried out by 
the Commission. 

Other than Framework Programmes expenditure concerns mainly Trans-European-Networks 
and  Nuclear  Decommissioning.  These  activities  as  well  as  their  control  systems  differ  from 
those  for  the  Framework  Programmes.  For  example,  for  nuclear  decommissioning,  the 
expenditure is either channelled through implementing bodies such as a certified agency with 
full state guarantee or administered through joint management (with the EBRD). Noticeably 
fewer errors are found by the Commission in these areas. 

3. The Dataset used in the model and its limitations  

The  Commission’s  proposal  for  tolerable  risk  in  this  area  is  based  on  two  dimensions:  the 
costs of control and the capacity of these to detect (and correct) errors. Costs of control used 
in this analysis are the result of an extensive data collection exercise within the Commission 
Directorates General and Executive Agencies dealing with the relevant programmes. Data on 
errors  is  essentially  based  on  error  rates  detected  in  2008  by  the  Court  of  Auditors  in  the 
policy groups covered by Chapter 7 of its Annual Report (Research, energy and transport). 

The  error  rates  found  by  the  Court  of  Auditors  have  been  broadly  comparable  with  the 
multiannual, residual levels of error for FP6 found by Commission services in recent years5. 
While the latter could have been used as a basis for this analysis, they are an integral part of 
the  control  system  and  could  themselves  have  been  perceived  as  biased.  It  was  therefore 
decided  to  use  errors  coming  from  the  Court  as  they  are  fully  independent  from  the 
Commission's internal control system. 

The research policy group (as defined by the Court in Chapter 7) comprises payments from 
budget  titles:  06  (Energy  and  transport),  08  (Research),  09  (Information  and  Society),  10 
(Direct research). For the purposes of the modelling presented in this document, a number of 
assumptions needed to be made in order to align both sets of data enabling comparability (see 
below annex 2). 

The  remaining  payments  in  2008  (extracted  from  the  Commission's  accounting  system  per 
project  coordinator/beneficiary6)  were  made  to  over  5,500  entities  and  represented  around 
€7,2  billion   of  budgetary  expenditure.  By  far  the  biggest  proportion  of  this  expenditure 
belongs  to  the  6th  and  7th  Research  Framework  Programmes  (76  %)  as  set  out  in  Figure  1 
below: 

5  Three  Directorates  General  managing  the  expenditure  under  FP6  reported  in  2009  the  residual  error  rates  in 
FP6 of: 2,7%; 2,21%; 4,04% as reported in their Annual Activity Reports 2009. 
6  The  accounting  system  provided  data  on  payments  made  for  coordinators  in  framework  programmes  and 
beneficiaries for other areas (for example non-research energy and transport).  

EN 

4 

   EN 

 
                                                 
9%

1%

3%

12%

32%

FP 5

FP 6

FP 7

TEN

43%

Nuclear
Decommissioning

Other

Figure 1: Breakdown of payments in 2008 as used for this Commission proposal and their share in the total of 
€7,2 bn. ( Source, ABAC) 

It  needs  to  be  underlined  that,  for  projects  with  more  than  one  participant,  the  models  use 
payments  to  coordinators  (for  simplification  purposes)  but  that  the  Commission  is  expected 
by the Court to ensure accuracy of payments to each beneficiary (by the coordinators) and that 
the  Commission  remains  responsible  for  inaccuracies  and  control  deficiencies  by  the 
beneficiaries. 

For the budget lines included in this population, the errors detected by the Court in 2008 were 
identified7.  There  were  23  quantifiable  errors  which  were  taken  into  consideration  and  the 
corresponding average error rate (most likely error) amounted to 2,998%. Whereas most of the 
payments related to FP7 (43%), none of the errors identified by the ECA in 2008 concerned 
FP7 as this was principally pre-financing which has a very low risk of error. The errors were 
concentrated on FP6 (20 of the 23 errors). 

4. Methodology and the results of the collection of costs of control exercise 

The  methodology  for  the  collection  of  costs  was  based  on  the  distinct  phases  of  the 
management of a project: selection, negotiation, project management, on-the-spot audits and 
implementing  of  audit  results.  At  each  of  these  stages,  controls  are  embedded  in  the 
administrative procedures to prevent, detect or correct potential errors and weaknesses. They 
constitute  the  essential  steps  in  ensuring  that  grant  agreements  are  signed  with  eligible 
beneficiaries  for  eligible  activities,  that  the  outputs  of  projects  are  satisfactory  and  that  cost 
claims submitted are in conformity with the rules. The table below sets out a description of 
the tasks treated as control activities associated to different phases of the project cycle and the 
reasoning for their inclusion in relation to the general control objectives at the relevant project 
stage: 

7 The ECA audited for Chapter 7 150 transactions selected in MUS sample in 2008. 
8  One  of  the  errors  related  to  the  Chapter  7  (of  MEDIA  programme)  was  not  taken  into  account  in  the  basic 
model  due  to  the  fact  that  the  relevant  funding  programme  was  moved  to  another  Directorate  General  of  the 
Commission (which will participate in a subsequent TRE exercise), however its impact was included in a part 
dealing with the sensitivity analysis. 

EN 

5 

   EN 

 
 
 
                                                 
Phase of a project 

General control objective  

Main tasks 

Selection of proposals 

To  ensure  that  the  proposals  selected 
for  financing  are  characterised  by 
scientific  excellence,  will  fulfil  the 
operational  objectives  set  out  in  the 
specific  work  programmes  and  will 
provide  the  best  value  for  public 
money 

Negotiation of 
contracts 

Proper  translation  of  each  retained 
legally 
scientific  proposal 
binding instrument – grant agreement 

into  a 

Project management 

On-the-spot audits 

applicable 

To ensure, prior to final payment, that 
and 
all 
(of 
regulatory 
operational and financial nature) have 
been respected 

requirements 

contractual 

of 

Control  objectives  of  on-the-spot 
audits  are  twofold:  to  provide  an 
the 
indication 
adequate 
effectiveness  of  ex-ante  controls  and 
to  ensure 
the 
expenditure and therefore the legality 
underlying 
and 
transactions on a multiannual basis. It 
is  a  key  element  of  the  control 
strategy for research 

the  accuracy  of 

regularity 

of 

Implementation of 
audit results 

To  ensure 
that  audit  results  are 
properly  implemented,  amounts  due 
are 
identified 
recovered 
weaknesses addressed 

and 

preparation and definition of 

planning and coordination of 

preparation of the final list of 

o 
work programmes  
o 
the calls 
the evaluation of proposals 
o 
received – screening of submitted 
proposals for their eligibility 
o 
proposals 
o 
carried out and operational 
objectives of projects 
o 
(determination of EU 
contribution) and duration 
o 
draft contracts  
o 

clarification of the work to be 

preparation of the projects' 

validation of the legal status  
financial viability control 

estimation of costs 

financial management tasks: 

ex-ante operational and 

check/ approve of financial 

o 
(when necessary) 
o 
and scientific reports 
o 
preparation and execution of 
payments 
o 
financial initiation and 
verification 
o  work of on-the-spot audit 
units 
o  work of operational units in 
terms of preparation of the 
documentation for audited 
projects 
o 
procedures with beneficiaries 
o  work performed by external 
audit companies 
o 
on the eligibility of specific 
expenditure 
o 
revenue, recovery orders, 
proposals for extension of 
findings to non-audited areas 
o 
with beneficiaries 
o 
recoveries 
o  waive, cancellation 
adjustments of recovery orders 

preparation of forecasts of 

adoption of enforced 

consultation with legal units 

exchange of correspondence 

organisation of contradictory 

Table 1: General control objectives and tasks at different project stages. 

EN 

6 

   EN 

 
In order to obtain reliable data on the costs of control by Commission staff, the Directorates 
General  and  Executive  Agencies  in  the  research  policy  group9  applied  the  same 
methodology10  to  estimate  staff  resources  carrying  out  the  defined  control  activities.  These 
staff resources were calculated as “full-time equivalents” and the costs derived from applying 
the standard average personnel cost used for human resources estimates in the Budget. These 
costs are €122.000 for an official and €64.000 for an external agent. 

The costs of control also include external costs: these comprise the cost of audit certificates in 
the  Framework  Programmes  that  were  taken  into  account11  whenever  relevant  and  feasible. 
The  cost  of  audits  of  projects  by  external  providers  on  behalf  of  the  Commission  is  also 
included. Costs borne by beneficiaries were not included in the collection exercise as these are 
independent  from  the  Commission's  and  performing  an  additional  cost  collection  exercise 
including all beneficiaries (or even a sample) would have been time and resource intensive. 
However it needs to be borne in mind that the higher administrative burden associated with 
the level of controls carried out on beneficiaries (and the higher the level of associated costs), 
the more likely they are to discourage participation in EU programmes. 

It is important to note that in the models that were developed (as presented below) in the short 
to medium term solely marginal costs influence the estimated level of TRE. 

As  per  the  above  methodology  established  by  the  Commission,  the  cost  of  control  in  the 
research, energy and transport policy group totalled around €267m annually . 

Whereas  it  was  decided  to  base  the  statistical  model  on  the  payments  relating  to  the  policy 
group  area  from  the  Chapter  7  of  the  ECA's  annual  report,  the  analysis  of  costs  of  controls 
presented below is broader as it concerns all "traditionally research" Directorates General12 of 
the  Commission  and  the  Executive  Agencies.  The  models  presented  however  exclude  those 
costs related to activities not included in the Court's Chapter 7. 

The figure 2 below presents the structure of  annual  costs of control in research, energy and 
transport spent within the Commission. The bulk of the total costs (43%) is borne at the stage 
of  project  management,  second  largest  amount  is  spent  on  selection  of  proposals  (18%) 
followed by negotiation of contracts (16%). 

9  Commission  services  that  took  part  in  the  exercise:  RTD,  INFSO,  TREN,  ENTR,  REA,  ERCEA,  TENTEA, 
EACI 
10 One of the Executive Agencies decided to use different but comparable methodology, which was justified by 
the fact that they deal with grants falling into different Chapters of the ECAs Annual Report and it would have 
been difficult to make a distinction 
11  For  FP6  audit  certificates  were  obligatory for  each  beneficiary,  for  FP7  only  for  beneficiaries  receiving  per 
reporting period more than 375.000 euro. Costs spent by beneficiaries on external audit certificates are treated as 
eligible and are reimbursed by the Commission. 
12 Although data concerning DG ENTR did not feed into the model (as they are not part of the current Chapter 7) 
they do manage the expenditure under the framework programmes. 

EN 

7 

   EN 

 
                                                 
Structure of costs of control per project stage

6% 6%

18%

11%

16%

43%

Selection of proposals

Negotiation of contracts

Project management

Ex-post audits

Ex-post results

Audit certificates

Figure  2:  Structure  of  costs  of  controls  per  project  stage  for  the  research,  transport  and  energy  policy  area, 
including Commission Services: RTD, INFSO, TREN, ENTR, EACI, EACEA, ERCEA, REA, TEN-TEA. 

The wide definition of controls used for this exercise was in line with the COSO13 definition 
of internal control, which inspired the internal control framework in the Commission. These 
controls are not limited to checks and verifications but include all the measures management 
and  staff  take  to  ensure  the  effectiveness  and  efficiency  of  the  operations  as  well  as  the 
legality and regularity of the underlying financial transactions. 

5.  Analysis  of  the  errors  found  in  research,  transport  and  energy  policy  group  by  the 
Court of Auditors 

Each  year  the  Court  of  Auditors  selects  and  audits  a  sample  of  transactions  for  each  policy 
group underlying the Chapters of its annual report. Based on the result of its audits (as well as 
on  the  assurance  derived  from  the  functioning  of  the  respective  supervisory  and  control 
systems),  it  gives  a  specific  assessment  on  the  legality  and  regularity  of  underlying 
transactions. Errors that the Court finds are classified as either non-quantifiable (for example 
an inappropriate tendering procedure or non-respect of the deadlines set by the regulation) or 
quantifiable (as for example declaration of ineligible costs by a beneficiary). 

For the purpose of this current exercise, all errors that the Court detected in 2007 and 2008 for 
research,  energy  and  transport  corresponding  with  the  dataset  on  payments  as  presented  in 
part 3 were analysed. For the purpose of establishing the TRE model only quantifiable errors 
have been taken into consideration as only they have a direct measurable impact on the value 
of the underlying transactions. 

Figure 3 below provides an overview of the main types of errors. The errors reported by the 
Court  were  divided  into  two  broad  categories:    those  with  an  impact  on  the  error  rate  and 
those  without.  Reported  errors  with  an  impact  on  the  error  rate  are  further  divided  between 
errors  originating  at  the  beneficiaries  and  those  for  which  a  root  cause  lay  with  the 
Commission. 

13 The internationally recognised COSO framework - http://www.coso.org/ 

EN 

8 

   EN 

 
 
                                                 
Figure 3: shows that errors were classified into eight groups. 

The primary and crucial conclusion from the classification of errors into the above categories 
was  that  there  was  a  certain  level  of  error  which  cannot  be  reduced  –  due  to  differences  of 
interpretation between the Commission and the Court or to issues for which the Commission 
has decided that controls would not be possible or cost-effective. This concerns in particular 
errors  of  category  D  and  E  mentioned  above.  Analysis  of  DAS  2007  and  2008  shows  that 
such errors may account for an error rate in excess of 1% - that is to say more than half-way 
towards the 2% threshold. 

For errors caused by legal discrepancy with the Court (category D) the Commission considers 
that no control failure occurred (thus it could not have been avoided in advance). For errors 
for  which  there  were  no  adequate  controls  put  in  place,  remedial  measures  for  preventing 
similar  errors  in  the  future  would  require  introduction  of  new  controls.  These  new  controls 
can  prove  costly  to  implement  (even  to  the  extent  of  being  cost-prohibitive)  or  their 
introduction may not diminish errors in the short term (for example due to previously defined 
regulatory  interpretations).  This  analysis  therefore  confirms  the  intuitive  impression  that  a 
"zero-error" situation is virtually unachievable and a certain level of inherent error needs to be 
accepted even before taking account of control costs. 

An  additional  argument  that  a  'zero  error'  situation  is  not  achievable  comes  from  a  cost 
benefit analysis of the on-the-spot audit effort. There were over 5,300 payments in this policy 
area to beneficiaries of research and other grant programmes in 2008. For research framework 
programmes, payments are made to coordinators which make payments to project participants 
(over 7 per project on average). This means that payments were made in respect of cost claims 
related to some 35,000 individual participations in 2008. Controlling each of these on the spot 
would cost some €2 billion. 

EN 

9 

   EN 

 
 
For  DG  Research,  even  with  such  extensive  audit  coverage,  the  residual  error  rate  would 
remain  close  to  1%.  The  expected  actual  recovery  (as  opposed  to  the  error  detected)  from 
each audited beneficiary is around 1,63% of the EU contribution. This implies that it is not 
cost-effective to audit beneficiaries with a cumulated EU contribution below € 3,7m because 
under  this  threshold,  the  expected  value  of  error  is  less  than  the  marginal  cost  (€60.000)  of 
detection  and  correction  through  an  on  the  spot  control.  The  number  of  beneficiaries  below 
the cost-effective audit threshold is over 14.480. If we apply the Court's error rate of 3%, this 
would mean that an error rate of 0.96% of the total EU contribution in FP6, cannot be cost-
effectively detected and corrected. 

The above analysis implies that almost 2% of error may be unavoidable (at least at reasonable 
cost)  as  some  1%  relates  to  differences  of  legal  interpretation  with  the  Court  and  0.9%  to 
beneficiaries which it is not cost-effective to control on the  spot (even though some control 
should be carried out on some of them for deterrent effect). 

Another  significant  finding  of  this  analysis  shows  that  most  of  the  errors  with  a  financial 
impact  (A  and  B  category)  could  have  been  avoided  only  by  controlling  the  project  on  the 
spot. Improving desk-based ex-ante checks proved to be the solution for only a very limited 
number of cases (in about 6% of errors). This argument is backed-up by the fact that the Court 
considers the desk-based ex ante controls before payment in FP6 to be effective14. 

Figure 4 below shows the structure of errors per typology defined above 

Share of different types of errors (A-H)

40%

35%

30%

25%

20%

15%

10%

5%

0%

A

B

C

D

E

F

G

H

Figure  4:  Different  types  of  DAS  errors  per  typology  established  by  the  Commission.  In  order  to  even  out 
potential abnormality of a given year, an average of two years (2007 and 2008) was taken. 

6. Modelling the relationship between the costs of control and errors – description of the 
models and results they provide 

The  DAS  is  based  on  the  internationally-accepted  Monetary  Unit  Sampling  (MUS) 
methodology,  the  most  commonly-used  statistical  sampling  method  in  auditing  (and  one 
which is used by the Court of the DAS). In this approach the auditor: 

14 Court of Auditors Annual Report 2008, Chapter 7, Table 2.1 Assessment of selected supervisory and control 
systems, pg 154. Court audit of ex-ante desk checks concerned one of research DGs (DG INFSO). 

EN 

10 

   EN 

 
 
                                                 
•  decides what level of confidence is required from the sample (effectively, the 

probability of the results of the sample reflecting reliably the situation across the 
whole population) and the materiality level (which sets a degree of tolerance in the 
accuracy of the underlying transactions). The Court, in line with common practice, 
uses a 95% confidence level and a 2% materiality rate; 

•  fixes the level of assurance he will draw from the quality of the systems in place (if 

he considers the systems as effective, he may draw some of the 95% overall 
confidence level from these systems: if not, the whole 95% is provided by the 
examination of a sample of transactions); 

•  calculates the sample size required based on the confidence level, materiality level 

and the assurance drawn from systems and selects the sample of transactions 
accordingly; and 

•  audits the selected transactions, records the results and uses statistical techniques to 

estimate the level of error in the population as a whole. 

The  statistical  basis  underlying  MUS  is  the  "normal  distribution";  which  means  that  (for  a 
confidence  level  of  95%)  there  is  a  probability  of  95%  that  the  actual  error  rate  across  the 
whole population of the transactions will be within two "standard deviations" from the apex 
of the distribution curve. This apex is called the "Most Likely Error" (MLE) and represents 
the best estimation of the error rate. However, as some uncertainty is involved in assessing the 
results of any sampling, this uncertainty is expressed by two extreme points (the Upper Error 
Limit (UEL) and the Lower Error Limit (LEL)): respectively, with a 95% confidence  level, 
there is a 2.5% probability that the actual error in the population is higher than the UEL and a 
2.5% probability that the actual rate is lower than the LEL (see Figure 5 below). 

Figure 5: MUS curve – in 2,5% cases the real error in the population may exceed the upper error limit and in 
2,5% cases the error may be actually smaller than the lower error limit. 

Two models were created for the purposes of this concrete proposal. The first was based on 
achieving  a  given  UEL.  This  was  used  notably  to  identify  the  cost  of  control  to  obtain  a 
virtual guarantee that an error rate of less than 2% could be achieved. The second model was 
based on the cost of achieving a given MLE, which is the benchmark used by the Court: of 
course in this scenario there would be a 50% probability of the error rate being greater than 
the MLE. 

EN 

11 

   EN 

 
 
The population used for the model concentrated on the latest (2008) DAS, for which 23 errors 
with financial impact were detected by the Court (categories A-E) in this area. The value of 
these  errors  as  a  proportion  of  the  total  payment  audited  ranged  from  0,72%  to  95,23%. 
Extrapolated  to  calculate  an  error  rate  for  the  whole  population  (that  is  including  both 
transactions affected by error and those which were error-free), this gave an overall average of 
2,99%. 

The  two  models  described  below  examine  how  the  average  error  in  the  population  of 
payments in 2008 might have changed if more or less had been spent on controls. They are of 
course  only  indicators  of  the  potential  relationship  between  control  and  undetected  error. 
They involved the use of several assumptions: 

1) all identified unduly paid amounts are immediately recovered (whereas in reality there is a 
time  lag  between  detection  and  recovery  and  some  amounts  may  be  waived  or  cannot  be 
recovered)  

2)  the  assumed  error  rate  is  an  average  based  on  the  DAS  error  rate  -  the  same  for  all 
beneficiaries: if an audit is performed the amount recovered (in the basic model) is 2,99% of 
the audited amount (whereas in fact some audits may result for example in 0% or 100% error 
rate) 

3)  the  assumed  error  rate  for  all  beneficiaries  is  defined  as  fully  systematic  (that  is  that  the 
same error is repeated across all programmes of the same beneficiary) whereas in reality it has 
a systematic and non-systematic component in itself. Commission audits generally involve 
the verification of a sample of a beneficiary's cost claims, the identification of errors and the 
recovery  of  these.  Where  an  error  is  likely  to  have  affected  non-audited  cost  claims,  the 
Commission may invite a beneficiary to reimburse an appropriate sum related to these costs 
claims or it may ask the beneficiary to restate these cost claims to remove the impact of such 
systematic errors. The Commission's audits on FP6 have shown that the non-systematic errors 
account  for  a  significant  proportion  (1.63%)  of  the  actual  error  rate  detected.  The 
Commission's  model  however  assumes  that  audits  will  clean  each  beneficiary  of  error  (that 
has  fully  systematic  nature)  and  therefore  overstates  the  "cleaning"  effect  of  audits  (and 
correspondingly underestimates the likely cost of removing errors from the population). 

Several simulations were performed based on variations in the criteria used: for both models 
therefore more than one scenario is presented. 

The statistical models introduced below are based solely on cost-benefit considerations and 
do  not  include  key  political  aspects  which  also  need  to  be  taken  into  account  in  fixing  the 
TRE  level.  The  dataset  (as  described  in  detail  above)  consisted  of  payments  to  over  5,500 
beneficiaries for a total amount of €7,2 bn . In lin e with the analysis of errors presented above, 
it  is  assumed  that  errors  can  only  be  detected  "on  the  spot".  Costs  spent  on  controls  are 
therefore to a large extent fixed in the short-term (€200m)  and the marginal cost is calculated 
by adding or removing on-the-spot audits. From the empirical data15 an average cost per audit 
amounting  to  €60.000  (including  the  cost  of  the  re sulting  recoveries)  per  project  participant 
was calculated and used in the models presented below. 

15 Average cost of an audit per planning audits in FP6 audit strategy.  

EN 

12 

   EN 

 
                                                 
In the models the benefits of controls are defined as recoveries from audits performed on-the 
spot,  however  it  needs  underlining  that  not  all  error  amounts  established  in  the  course  of 
audits can be actually recovered by the Commission and some may be waived or be simply 
unrecoverable due to the financial situation of the beneficiary. Model 1 (as described below) 
is  focused  primarily  on  achieving  a  2%  error  rate  and  does  not  seek  the  maximisation  of 
benefits of controls at all costs. Model 2 takes account of the benefits of control by means of 
calculating  "a  yield"  for  each  coordinator  defined  as  the  difference  between  amounts 
recovered from the audited coordinator and costs spent on controlling this coordinator (on the 
spot controls are carried out when the yield is positive: the expected recovery exceeds the cost 
of the individual control). 

6.1. MODEL 1: achieving a given UEL 

To give a virtual guarantee of achieving a "green light" (under 2% error level) in this Chapter 
of the Court's report, the Commission would need to carry out sufficient controls on the spot 
to make it almost certain that the Upper Error Limit was 2%. This model estimates the level 
of  marginal  costs  (on  top  of  the  existing  ex  ante  controls)  that  would  need  to  be  spent  to 
guarantee  a  level  of  error  in  the  population  of  2%.  For  comparative  purposes  the  costs  of 
achieving a given MLE were also examined. 

 The model took the 2008 population of payments and calculated the cost of control based on 
audits in the year of all of the largest beneficiaries together with sample-based checks on the 
remainder.  Several  scenarios  were  used,  assuming  audits  of  different  numbers  of  the  top 
beneficiaries  with  a  sample  of  the  remainder.  Figure  6  below  illustrates  the  results  of  the 
different scenarios: 

costs for different scenario's

top 500

top 400

top 350

top 300

top 200

top 100

2% MLE

2% UEL

0

20

40

60

80

100

120

140

160

180

Millions

Figure 6: Model 1 presents scenarios to achieve levels of MLE and UEL of 2% by controlling on the spot the top 
100, 200, 300, 350, 400 and 500 beneficiaries and a sample of the remainder. 

EN 

13 

   EN 

 
 
Figure 6 illustrates an incremental rise in costs of control related to additional audits that need 
to be executed on the spot. It should be noted that the fixed costs of €200 million (those of the 
other stages in the control chain) are unaffected by this analysis. 

The  way  of  virtually  guaranteeing  a  "green  light"  would  be  to  carry  out  enough  control  to 
deliver a UEL of 2% (as explained above - a situation when we are more or less sure that the 
error  found  in  the  population  will  not  exceed  2%,  the  MLE  at  this  stage  is  almost  0%). 
Regardless of the chosen audit strategy (whether we  audit  top 100 or top 500 beneficiaries) 
costs  that  would  need  to  be  spent  on  controls  to  achieve  2%  (an  equivalent  of  DAS  green 
light)  would  be  considerable  –  ranging  from  €1 55m  to  €166  m.  The  cost  of  being  sure  of 
achieving the 2% benchmark would therefore be prohibitive (at over €150 million). Even the 
cost of achieving an MLE of 2% is considerable (at over €90 million), compared with what is 
being spent currently on on the spot controls (less than €50 million) to arri ve at an error rate 
of almost 3%. 

The  above  analysis  is  based  on  one  year  only  –  2008.  In  order  to  have  a  wider  field  for 
examination  and  results  there  from,  the  data  from  2007  was  introduced  to  the  model. 
Several parameters needed to be adapted, such as the number of sampled transaction for the 
policy group (180 instead of 150 in 2008), number of payments per beneficiary, average error 
detected  (higher  in  2007  than  in  2008).  Succeeding  calculations  revealed  that  in  order  to 
obtain a guarantee of having in the population an error not higher than 2% (ULE) even more 
resources would need to be spent on additional controls - around €190m. This result exceeds 
the outcome for 2008 substantially, in large part due probably to the higher error rate in the 
DAS 2007 for this Chapter. This in turn was possibly partly due to the fact that the numbers 
of on the spot audits by the Commission increased significantly between 2007 and 2008 and it 
could  therefore  be  expected  that  the  Commission  "cleaned"  more  projects  of  error  in  2008 
than  in  the  previous  year.  Additionally,  the  proportion  of  final  payments  was  increasing  as 
FP6 came into full swing (typically final payments are the most error prone ones as it is at this 
stage that the actual costs of a project are finally verified). Further analysis also suggests that 
the decrease in the  error  rate  in 2008  might have been due to phasing out of FP5, which in 
contrast to FP6 did not include an audit certification policy. 

6.2. MODEL 2: targeting the MLE 

The second model is based on the most likely error, using the same basic data as for Model 1. 
The  basic  principle,  following  statistical  theory,  is  that  each  Euro  in  the  population  of 
payments in 2008 was affected by the same percentage of error as the population audited by 
the Court (2.99%). This percentage was converted into a "yield" per beneficiary: defined as 
the difference between the assumed recovery from an audit (calculated as 2.99% of the total 
payment to that beneficiary) and the cost of auditing that beneficiary. The cost of auditing a 
single beneficiary was calculated on the basis of the audits carried out in the research policy 
group  relevant  to  the  established  dataset  in  2008  (733)  and  the  costs  related  to  on-the-spot 
controls (including recovery of errors detected): €60.000. For the purposes  of the exercise it 
was  assumed  that  the  remaining  control  costs  were  "fixed"  as  on-the-spot  control  was 
identified as the principal tool for detecting and correcting errors. 

A curve can then be derived with multiple points which can be used to calculate the cost of 
"adding"  individual  on-the-spot  controls,  increasing  the  cost  and  reducing  the  error  in  that 
beneficiary. The ranking of the beneficiaries for this curve can be done in two ways: in the 
best-case  scenario  we  assume  that  we  will  visit the beneficiaries with the highest yield first 

EN 

14 

   EN 

 
(assuming the localisation of errors is known), in a second case we choose the beneficiaries to 
be audited randomly. 

Scenario A: MODEL 2A "best-case scenario" 

In  the  best  case  scenario  the  beneficiaries  were  ranked  ranging  from  the  ones  bringing  the 
highest yield  to  the lowest. So  the  errors that are subsequently removed are removed  in  the 
most cost-effective way. At the start point we have an error rate of 2,99%. In order to reach an 
error rate of 2,9% we need to get rid of 0,09% error in the population of €7,2 bn spread over 
5538  beneficiaries,  which  means  that  we  would  need  to  recover  additionally  (0,09%  *  €7,2 
bn)  €6,5m  euro.  In  such  a  way  we  progress  movi ng  down  from  the  initial  level  of  error  of 
2,99%, through auditing additional beneficiaries and reducing the error rate. 

Scenario B: MODEL 2B "random selection" 

In  this  scenario  the  coordinators  were  not  ranked  by  yield,  their  selection  was  random.  The 
random simulation was run 1000 times, which gave a straight line with 10 individual points, 
at  each  of  which  the  corresponding  cost  of  controls  can  be  derived.  More  points  could  be 
generated on this line but this would not change the overall results. 

Scenario 2AB: scenarios 2A and 2B combined – TRE range 

When the product of models 2A and 2B are combined, one could confidently presuppose that 
the TRE point would be located somewhere between the best-case scenario and a more real-
life  (random)  situation  described  above.  The  corresponding  results  are  set  out  in  Figure  7 
below. This shows that the best-case scenario, which assumes we know where the errors are 
located, would be highly cost-effective in reducing the error: the reduction to 2.5% error rate 
would  cost  less  than  €1  million.  However,  th e  random  scenario,  which  can  effectively  be 
considered  as  a  "worst-case"  scenario  (as  the  Commission  does  have  information  which 
enables  it  to  target  risks),  would  imply  a  much  higher  cost  to  reach  the  same  2.5%  target 
(some  €55  million).  The  range  between  these  two  figures  is  very  wide:  however,  it  is  not 
unreasonable  to  assume  that  the  cost  of  reducing  the  error  rate  (MLE)  to  2.5%  could  lie 
somewhere  towards  the  median  of  the  range  €1  million  to  €55  million:  somewhere  around 
€28 million. This would imply increasing the current cost of on-the-spot controls by almost 
80%.  The  chart  below  visually  presents  two,  combined  scenarios  for  the  Model  2  with  the 
TRE range lying between the two lines. 

EN 

15 

   EN 

 
Scenario 2AB - TRE range

s
e
c
r
u
o
s
e
r

l

o
r
t
n
o
C

1.000.000.000

100.000.000
10.000.000

1.000.000

100.000
10.000

1.000

100
10

1

2,90%

2,80%

2,70%

2,60%

2,50%

2,40%

2,30%

2,20%

2,10%

2,00%

Error Rate

Random

Best Case

Figure 7: TRE range between the best case and random scenarios 

The curves relate to the best-case scenario and a given random scenario for the Model 2. They 
present  the  reduction  in  the  error  rate,  which  is  achieved  by  dedicating  additional  resources 
for control. A logarithmic scale (also know as ratio scale) has been used when plotting the "y" 
values in order to reduce the large range of values16. 

Scenario 2C: "reducing the costs of control" 

To  provide  fuller  information  for  the  inter-Institutional  discussions  on  the  concrete  level  of 
TRE in this sector, the Commission services also examined what might happen if the number 
of on-the-spot controls were to be reduced, thus accepting a higher error rate. This is also a 
quasi best-case scenario. In this case the beneficiaries were ranked, starting from the one who 
brings the smallest yield (beneficiaries with the lowest amount of error to be recovered by the 
Commission). 

This analysis gives indicative error rates for achieving a given (higher) MLE than that of the 
Court. The costs associated can be assumed to be the minimum to achieve the given level and 
are  (as  explained  earlier)  incremental.  The  table  below  presents  different  (higher  than  the 
current  2,99%  MLE)  levels  of  error  that  could  be  implied  by  spending  less  on  on  the  spot 
controls (by diminishing number performed). 

16  A  logarithmic  scale  is  a  scale  of  measurement  that uses  the  logarithm  of  a value  instead of  the value  itself. 
Logarithmic scales have been explicitly designed to reduce large range of values to a more manageable range. 
The logarithmic scale used here is one in "factors of ten" which makes use of so-called decades on the vertical 
axis.  A decade is a factor of 10 difference between two numbers (as order of magnitude difference) rather than 
unit steps. 

EN 

16 

   EN 

 
 
  
 
                                                 
Error rate 

Reduction in control costs 

3,5% 

4% 

4,5% 

€8 million 

€15,5 million 

€22 million 

     Table 2: predicted higher error rates corresponding to reduced costs of controls 

6.3. The implications of the analysis for a TRE level 

The analysis above shows that guaranteeing an error rate of 2% would be prohibitive in cost 
terms. Likewise, any reduction in the current error rate could be costly to achieve and would 
be  unlikely  to  be  cost  effective.  Of  course,  the  error  rate  will  evolve  year-on-year  as  the 
balance of framework programmes and other activities change: and statistical sampling also 
has  its  own  uncertainties  which  can  affect  the  error  rate  in  a  given  year.  The  Commission 
services  also  believe  that  Framework  Programme  7  (for  which  most  payments  to  date  have 
been  low-risk  pre-financing)  could  be  more  error-prone  than  FP6  when  it  comes  to  treating 
cost  claims  due  to  the  reduction  in  the  numbers  of  audit  certificates  and  the  low  number  of 
beneficiaries  eligible  for  the  acceptance  of  their  personnel  costing  methodologies  (which 
would  permit  the  claiming  of  average  personnel  costs  rather  than  actuals,  thus  reducing  the 
risk  of  error).  As  a  result,  it  would  be  prudent  to  fix  a  range  for  TRE.  This  could  be 
reconsidered in the light of any major changes in the control environment. 

It is clear from the above that the TRE level should be fixed in the current "yellow" zone of 
the Court: analysis by DG Research has indicated that reducing the multi-annual error rate for 
its payments alone would cost some €126 million for FP7 (2,100 audits multiplied by €60,000 
unit cost). The analysis above indicates that reducing the current MLE significantly would be 
likely  to  be  costly.  Similarly,  the  2008  error  rate  of  2.99%  may  evolve  as  FP7  comes  on-
stream and the higher inherent risk may influence the error rate upwards. 

In this light, the Commission proposes a TRE level in the yellow range (2%-5%). A DAS 
error  rate  around  the  middle  of  this  range  would  be  acceptable  and  justified.  Beyond 
this level, additional  action  would be taken to reduce the  error rate through increased 
controls addressing the major causes of error and taking into account the multi-annual 
nature of the expenditure. This rate should cover all Research Framework Programmes 
irrespective of the Chapter of the Court's annual report in which they are included as 
well  as  all  activities  included  in  this  analysis  even  if  they  are  reallocated  to  other 
Chapters of the Court's annual report in future. 

Fixing a tolerable rate of error higher than 2% could allow a fundamental refinement of the 
current  control  strategy.  In  particular  the  magnitude  of  controls  (audit  coverage)  and  focus 
could be addressed with a view to lower the control burden both for the major beneficiaries 
(where  this  is  cost-effective)  and  the  Commission  (and  thus  reduce  the  disincentive  to 
beneficiaries  to  participate  in  Union  programmes).  Whilst  maintaining  an  adequate  level  of 
representative  controls  (to  support  the  regular  need  for  assurance  on  sound  financial 
management), such a revised control strategy would allow to focus less on audit coverage and 
more on targeted risk based audits and fraud prevention controls. On such basis, a sufficiently 
solid accountability framework could be ensured whilst maintaining a sufficient deterrent to 

EN 

17 

   EN 

 
irregular use of EU research funds. As a result, scarce financial and human resources could be 
reoriented to other priority research policy areas. 

EN 

18 

   EN 

 
Annex 1: Glossary 

Audit certificates: independent certification of the costs incurred and claimed under the FP6 
project  for a  given period.  Their  purpose was to give the Commission reasonable assurance 
that eligible costs are claimed in accordance with the relevant provisions of the contract. The 
auditor  must  be  competent  and  independent  from  the  contractor.  The  Certificate  on  the 
Financial Statements (CFS) in FP7 replaced the FP6 Audit Certificates. FP7 introduced also 
additional notions of audit certificates. 

Ex-ante verifications: all checks put in place by the authorising officer responsible in order 
to verify operational and financial aspects of an operation prior to its execution. The purpose 
of this verification is to ascertain compliance with the applicable provisions and the principle 
of sound financial management. 

Framework  programmes:  since  1984,  research  and  innovation  activities  of  the  EU  are 
grouped  in  one  large  programme  called the  Framework  Programme  (FP).  Research 
framework  programmes  are  the  EU's  main  financial  and  legal  instruments  to  implement  the 
European  Research  Area.  The  current  FP7  (2007-2013) is  designed  as  a  key  contribution  to 
the EU's strategy for growth and jobs. 

FP6/7  audit  strategy:  audit  strategy  established  jointly  by  the  Commission's  research 
services in order to assess the legality and regularity of financial transactions under FPs and 
provide  the  basis  for  corrective  and  recovery  mechanisms  and,  in  particular,  to  eliminate 
errors occurring in the cost statements provided by the beneficiaries. It aims at reaching the 
residual multiannual level of error of 2% by the end of the Programme. 

Full time equivalent (FTE): ratio used to measure a person's involvement in an activity. FTE 
=  1  means  that  the  person  is  equivalent  to  a  full-time  employee;  whereas  an  FTE  =  0.5 
indicates  that  the  employee  works  only  half-time  on  a  certain  project  (e.g.  on  performing  a 
control activity). 

Most Likely Error (MLE): an error that is equivalent to the best estimation using statistical 
techniques  of  an  error  that  can  be  found  in  a  population  having  a  normal  distribution.  It  is 
calculated based on the errors found in an audited sample. 

Monetary Unit Sampling (MUS): a method of statistical sampling used to assess the amount 
of monetary misstatement in a population. It is the most commonly used sampling method in 
auditing. 

Non-systematic  portion  of  an  error  rate:  the  part  of  an  error  rate  detected  during  control 
that cannot be applied to other payments to the same beneficiary because its nature is random. 

Non-quantifiable  errors:  errors  that  do  not  have  direct  measurable  financial  impact  on  the 
value of a transaction. These may be errors which concern non-observance of a condition for 
payment or other compliance issue for example non-respect of deadlines set by a regulation 
but which does not affect the amount paid. 

EN 

19 

   EN 

 
 
Normal distribution: a symmetrical distribution in statistics, represented on a graph by a bell 
curve, in which the class with the highest frequency is at the centre and the classes with the 
lowest frequency are at each end. 

On-the-spot control: an optional control (distinct from both the ex-ante verification and the 
initiation  of  an operation)  that  contributes  to an  increase in the  assurance of  the authorising 
officer  by  delegation  concerning  the  use  of  EU  funds  (for  which  the  Commission  is 
accountable).  On  the  spot  control  checks  the  proper  implementation  of  operations  already 
financed (a posteriori - intermediate or final payment) as regards their regularity, conformity 
and  sound  financial  management.  These  checks  are  done  on  documents  and/or  on-the-spot 
and can be selected based on a statistical sample, judgementally by means of a risk analysis or 
randomly. 

Quantifiable  errors:  errors  that  affect  the  eligibility  conditions  and  that  have  direct 
measurable  impact  on  the  value  of  the  underlying  transactions  (error  rate),  for  example 
reimbursement  errors  resulting  from  ineligible  items  included  in  the  cost  statements  by  a 
beneficiary. 

Systematic  portion  of  an  error  rate:  a  part  of  an  error  rate  that  can  be  applied  to  other 
payments to the same beneficiary as its underlying causes are of the same nature and they are 
highly likely to be repeated in other cost statements.  

EN 

20 

   EN 

 
Annex 2: Definition of the dataset on payments: 

Budget lines description 

Reasons for exclusion 

Payments < 20.000 euro 

Small grants and payments to natural persons (as experts) were 
eliminated.  Whereas  they  could  be  as  error  prone  as  bigger 
payments  the  financial  impact  of  errors  detected  therein  is 
overall less material. 

Contributions 
countries 
from EEA countries) 

(payments 

from 

third 
received 

These  amounts  are  used  to  finance  expenditure  related  to 
Framework  Programmes  in  the  future, so  their  inclusion  would 
result in double counting 

Budget title 10 – direct research  Direct  research  is  managed  by  the  Joint  Research  Centre.  This 
spending  consists  mainly  of  a  large  number  of  small  value 
transactions in a form of direct actions, which are quite distinct 
from  other  research  spending  and  require  different  control 
strategies. 

Administrative 
(budget chapters XX 01) 

expenditure 

The  proportion  of  administrative  expenditure  was  indirectly 
included in the calculation of cost of controls by using average 
costs  for  the  estimates  on  “Human  resources".  Administrative 
expenditure will be analysed separately in the upcoming second 
wave  of  Commission's  Communication  on  concrete  TRE 
proposals.  

Framework  programme  4  and 
preceding FPs 

Payments related to Framework programmes prior to FP5 were 
not  taken  into  consideration  as  these  programmes  are  now 
phasing out. 

Media programme 

Management  of  this  programme  was  transferred  to  a  different 
Directorate  General,  which  falls  into  another  chapter  of  the 
ECA's  annual  report.  (To  assure  the  comparability,  costs  of 
controlling this programme were not included on the costs side). 

EN 

21 

   EN