EN    EN EN 
 
EN    EN  EUROPEAN COMMISSION 
Brussels, 26.5.2010 
SEC(2010) 641 
  
COMMISSION STAFF WORKING DOCUMENT 
Developing the tolerable risk of error co ncept for the research, energy and transport 
policy area 
  
Accompanying document to the 
  
COMMUNICATION FROM THE CO MMISSION TO THE EUROPEAN 
PARLIAMENT, THE COUNCIL AND THE COURT OF AUDITORS 
Commission Communication: More or less cont rols? Striking the right balance between 
the administrative costs of control and the risk of error 
  
{COM(2010) 261 final} 
{SEC(2010) 640} 
 
EN 2   EN 1. Introduction 
The purpose of this document is to present the technical aspects  underlying the 
Commission's proposal for the to lerable risk of error (TRE) for the research, energy and 
transport policy area. It sets out statistical models for analysing the relationship between costs of controls and error ra tes and explains the results derived there from. 
Part 2 of the paper briefly outlines the specific characteristics of the research, energy and 
transport control environment  that needed to be taken into  account in the modelling and to 
inform the decision on TRE.  
In Part 3 follows the presentation of the dataset used in the model  and its limitations.  
Part 4 explains the methodology established in order to collect costs  of control data as well 
as the results of this exercise. This part gives an overall description of the current situation as 
regards costs dedicated to control activities in the research, energy and transport policy 
area  whereby control is understood in a broad sens e comprising activities at all stages of a 
project cycle starting from the selection of beneficiaries until the implementation of audit results through the recovery of any undue paymen ts. Such activities are classified as “control 
activities” where they result in decisions which influence the eligibility of an activity and/or 
of certain of its costs
1. This definition encompasses the evaluation of propos als (selecting the 
projects to be financed, ensu ring they meet eligibility crite ria and the achievement of the 
objectives as adopted by Council and Parliament); the detailed negotiation of the grant agreement (the translation of each  of the retained sc ientific proposals into a legally binding 
instrument including both scie ntific and financial aspects); the management of the project 
(verifying the outputs which demonstrate the state of adva ncement of the project and 
influence the eligibility of costs), the control of  all aspects of cost claims and the control on 
the spot of projects and the associated costs of the recovery of any amou nt due as a result of 
the errors detected.   
Part 5 provides an analysis of the errors  found in the research, energy and transport policy 
group by the Court of Auditors in recent years.  It presents the classification of errors 
according to their causes and dr aws conclusions there from. 
Finally in Part 6 two concrete TRE models  developed by the Co mmission are introduced 
together with their underlying assumptions. The models are used to illustrate a number of 
scenarios for the relationship between costs of  controls and errors in order to derive TRE 
range  there from. 
  
                                                
 
1 The Commission's internal control systems are base d on the internationally- accepted COSO model which 
defines internal control as "a process designed to pr ovide reasonable assurance regarding the achievement of 
objectives in effectiveness and efficiency of operations , reliability of financial reporting, and compliance with 
applicable laws and regulations". ( http://www.coso.org ) 
 
 
EN 3   EN 2. Specific characteristics of the research, energy and transport control environment  
Grant expenditure in the research, energy and tr ansport area is subject in the main part to 
centralised management by the Commission, either  directly by the Comm ission's Directorates 
General, or indirectly through executive agen cies or joint undertakin gs. This implies the 
concentration of supervision a nd control activities and resources within the Commission and 
it can be therefore concluded that the majority of control costs in this area are borne by the 
Commission. This was certainly the case in 2 009 when the data were collected for this 
exercise as the two new research executive agencies (ERCEA and REA) and several joint 
undertakings2 were at the capacity-building stage. 
The Commission relies on project c oordinators to distribute funds to project participants and 
to consolidate their financial and scientific re porting on the implementation of the project to 
the Commission3. Equally, the Commission relies on th e accounting and internal control 
systems of the beneficiaries to ensure the accu racy and regularity of payments, carrying out 
itself desk-based checks and sample controls on the beneficiaries' premises. In consequence, 
the Commission, coordinators and beneficiaries may bear non-negligible costs linked to these 
controls4. These have not been considered for the assessment of the cost of controls. If the 
Commission was required to authorise the payments to each beneficiary, the number of 
payments would be roughly multiplied by 7 (7. 29 is the average number of beneficiaries per 
FP6 project) which would rende r the scheme unmanageable. 
Some three quarters of the expenditure in this  area is via the successi ve research framework 
programmes while the rest is spent throug h the Trans-European Networks, Nuclear 
Decommissioning and other grant activities. Th e Commission's main instruments to provide 
research grants to beneficiaries are mu ltiannual funding schemes – Framework Programmes 
that are based on Union pre-financing which is  then “cleared” by act ual costs declared by 
beneficiaries, which can be a year  or more after the pre-financing:  for this reason eligibility of 
costs can be finally checked only after the ex ecution of the activity (t hrough checks based on 
self declarations of contractors and – for the more detailed eligibility controls- on the spot at a 
beneficiary’s premises). 
Rules for participation in Framework programme s define the conditions to be met by cost 
claims in order to be considered eligible cost s and additional guidance a ssists beneficiaries to 
meet the requirements. However, the system of  reimbursement of costs means that a very 
detailed level of proof is requested from bene ficiaries (for example, personnel costs based on 
detailed record-keeping of the hours spent by each staff member on a project and on the detailed recording of all salary  and other costs actually paid to  the individual concerned). As 
recognised by the Court of Auditors, such re quirements mean that beneficiaries’ cost 
statements are inherently error prone. 
In practice, given the complexity of the control of eligible costs,  the fact that the payments are 
made on the basis of self-d eclaration where no supporting doc uments are requested in 
advance means that detailed contro l of cost claims is most ofte n carried out on the spot (on a 
sample basis) where the auditor has access to all of the necessary supporting information. To 
this extent Commission services  have established a resource intensive on-the-spot control 
                                                
 
2 IMI, Artemis, Clean Sky, ENIAC, FCH. 
3 Hereinafter we will refer to bot h coordinators and project participants as "beneficiaries". 
4 Part of such costs may be reimbursed in practice as overheads. 
 
EN 4   EN strategy having a multiannual control objective to bring the residual level of error from on-
the-spot controls below 2% at the end of FP6 (end of 2010). In spite of major control efforts, 
as indicated in 2009 Annual Activity Reports of the relevant DGs, and based on current 
information, it is not certain that this objectiv e can be reached: the current multi-annual error 
rate for FP6 reported by the concerned DGs in their 2009 Annual Activity Reports is higher 
than 2% even taking account of the fact that n early 1,900 FP6 audits have been carried out by 
the Commission. 
Other than Framework Programmes expenditure concerns mainly Trans-European-Networks 
and Nuclear Decommissioning. These activities as  well as their control systems differ from 
those for the Framework Programmes. For example, for nuclear decommissioning, the 
expenditure is either channelled through implemen ting bodies such as a ce rtified agency with 
full state guarantee or administ ered through joint management (with the EBRD).  Noticeably 
fewer errors are found by the Commission in these areas. 
3. The Dataset used in the model and its limitations  
The Commission’s proposal for tolerable risk in  this area is based on two dimensions: the 
costs of control and the capacity of these to de tect (and correct) errors. Costs of control used 
in this analysis are the result of an extensive data collection exercise within the Commission 
Directorates General and Executive Agencies de aling with the relevant programmes. Data on 
errors is essentially based on error rates det ected in 2008 by the Cour t of Auditors in the 
policy groups covered by Chapter 7 of its A nnual Report (Research, en ergy and transport). 
The error rates found by the Court of Auditors  have been broadly comparable with the 
multiannual, residual levels of error for FP6 fo und by Commission servi ces in recent years5. 
While the latter could have been us ed as a basis for this analysis, they are an integral part of 
the control system and could themselves have been perceived as biased. It was therefore 
decided to use errors coming from the Cour t as they are fully independent from the 
Commission's internal control system. 
The research policy group (as defined by the C ourt in Chapter 7) comprises payments from 
budget titles: 06 (Energy and tran sport), 08 (Research), 09 (In formation and Society), 10 
(Direct research). For the purposes of the mode lling presented in this  document, a number of 
assumptions  needed to be made in order to align bot h sets of data enabling comparability (see 
below annex 2). 
The remaining payments in 2008 (extracted  from the Commission's accounting system per 
project coordinator/beneficiary6) were made to over 5,500 enti ties and represented around 
€7,2 billion  of budgetary expenditure. By far the biggest proportion of this expenditure 
belongs to the 6th and 7th Research Framework Programmes ( 76 % ) as set out in Figure 1 
below: 
                                                 
5 Three Directorates General managing  the expenditure under FP6 reported in 2009 the residual error rates in 
FP6 of: 2,7%; 2,21%; 4,04% as reported in their Annual Activity Reports 2009. 
6 The accounting system provided data on payments ma de for coordinators in framework programmes and 
beneficiaries for other areas (for example non-research energy and transport).  
 
EN 5   EN 1%
32%
43%12%3%9%FP 5
FP 6
FP 7
TEN
Nuclear
Decommissioning
Other
 
Figure 1 : Breakdown of payments in 2008 as used for this  Commission proposal and their share in the total of 
€7,2 bn. ( Source, ABAC ) 
It needs to be underlined that, for projects w ith more than one participant, the models use 
payments to coordinators (for simplification purposes) but that the Commission is expected 
by the Court to ensure accuracy of payments to each beneficiary (by the coordinators) and that 
the Commission remains responsible for ina ccuracies and control deficiencies by the 
beneficiaries. 
For the budget lines included in this population, th e errors detect ed by the Court in 2008 were 
identified7. There were 23 quantifiable errors which were taken into consideration and the 
corresponding average error rate (most likely error) amounted to 2,998%. Whereas most of the 
payments related to FP7 (43%), none of the errors identified by th e ECA in 2008 concerned 
FP7 as this was principally pre-financing which ha s a very low risk of error. The errors were 
concentrated on FP6 (20 of the 23 errors). 
4. Methodology and the results of the colle ction of costs of control exercise 
The methodology for the collection of costs was based on the distinct phases of the 
management of a project: selection, negotiati on, project management, on- the-spot audits and 
implementing of audit results. At each of these stages, controls are embedded in the 
administrative procedures to prevent, detect or  correct potential errors and weaknesses. They 
constitute the essential steps in ensuring that  grant agreements are signed with eligible 
beneficiaries for eligible activities, that the outputs of projects are satisfactory and that cost claims submitted are in conformity with the rule s. The table below sets out a description of 
the tasks treated as control activities associated to different phases of the project cycle and the 
reasoning for their inclusion in re lation to the general control object ives at the relevant project 
stage: 
 
                                                
 
7 The ECA audited for Chapter 7 150 transactions selected in MUS sample in 2008. 
8 One of the errors related to the Chapter 7 (of MEDIA programme) was not taken into account in the basic 
model due to the fact that the relevant funding programme was moved to another Directorate General of the 
Commission (which will participate in a subsequent TRE ex ercise), however its impact was included in a part 
dealing with the sensitivity analysis. 
 
EN 6   EN Phase of a project General control objective  Main tasks 
Selection of proposals To ensure that the proposals selected 
for financing are characterised by 
scientific excellence, will fulfil the 
operational objectives set out in the specific work programmes and will provide the best value for public money o preparation and definition of 
work programmes  
o planning and coordination of 
the calls 
o the evaluation of proposals 
received – screening of submitted proposals for their eligibility 
o preparation of the final list of 
proposals  
Negotiation of 
contracts Proper translation of each retained 
scientific proposal into a legally binding instrument – grant agreement o clarification of the work to be 
carried out and operational objectives of projects 
o estimation of costs 
(determination of EU contribution) and duration 
o preparation of the projects' 
draft contracts  
o validation of the legal status  
o financial viability control 
(when necessary) 
Project management To ensure, prior to final payment, that 
all applicable contractual and 
regulatory requirements (of 
operational and financ ial nature) have 
been respected o check/ approve of financial 
and scientific reports 
o financial management tasks: 
preparation and execution of payments 
o ex-ante operational and 
financial initiation and 
verification 
On-the-spot  audits Control objectives of on-the-spot 
audits are twofold: to provide an 
adequate indication of the 
effectiveness of ex-ante controls and to ensure the accuracy of the expenditure and therefore the legality and regularity of underlying 
transactions on a multiannual basis. It 
is a key element of the control strategy for research o work of on-the-spot audit 
units 
o work of operational units in 
terms of preparation of the documentation for audited projects 
o organisation of contradictory 
procedures with beneficiaries 
o work performed by external 
audit companies 
o consultation with legal units 
on the eligibility of specific expenditure 
Implementation of audit results To ensure that audit results are 
properly implemented, amounts due 
are recovered and identified 
weaknesses addressed o preparation of forecasts of 
revenue, recovery orders, 
proposals for extension of 
findings to non-audited areas 
o exchange of correspondence 
with beneficiaries 
o adoption of enforced 
recoveries 
o waive, cancellation 
adjustments of recovery orders 
Table 1 : General control objectives and ta sks at different project stages. 
 
EN 7   EN In order to obtain reliable data on the costs of control by Comm ission staff, the Directorates 
General and Executive Agencies in the research policy group9 applied the same 
methodology10 to estimate staff resources carrying out  the defined control activities. These 
staff resources were calculated as “full-time e quivalents” and the costs derived from applying 
the standard average personnel cost used for human resources estimates in the Budget. These costs are €122.000 for an official and €64.000 for an external agent. 
The costs of control also include external costs: these comprise the cost of audit certificates in 
the Framework Programmes that were taken into account
11 whenever relevant and feasible. 
The cost of audits of projects by external providers on behalf of the Commission is also 
included. Costs borne by beneficiar ies were not included in the collection exercise as these are 
independent from the Commission's and perfor ming an additional cost collection exercise 
including all beneficiaries (or even a sample) would have been time and resource intensive. 
However it needs to be borne in  mind that the higher administra tive burden associated with 
the level of controls carried out  on beneficiaries (and th e higher the level of associated costs), 
the more likely they are to discoura ge participation in EU programmes. 
It is important to note that in the models that were developed (as presented below) in the short 
to medium term solely marginal costs influence the estimated level of TRE. 
As per the above methodology established by the Commission, the cost of control in the 
research, energy and transport policy group totalled around €267m annually . 
Whereas it was decided to base the statistical  model on the payments relating to the policy 
group area from the Chapter 7 of the ECA's annual report, the analysis of costs of controls 
presented below is broader as it concerns all "traditionally research" Directorates General12 of 
the Commission and the Executive Agencies. Th e models presented however exclude those 
costs related to activities not included in the Court's Chapter 7. 
The figure 2 below presents the structure of annual costs of c ontrol in research, energy and 
transport spent within the Comm ission. The bulk of the total cost s (43%) is borne at the stage 
of project management, second largest amount is spent on selection of proposals (18%) 
followed by negotiation of contracts (16%). 
                                                 
9 Commission services that took part  in the exercise: RTD, INFSO, TREN, ENTR, REA, ERCEA, TENTEA, 
EACI 
10 One of the Executive Agencies decide d to use different but co mparable methodology, which was justified by 
the fact that they deal with grants falling into different  Chapters of the ECAs Annual Report and it would have 
been difficult to make a distinction 
11 For FP6 audit certificates were obl igatory for each beneficiary, for FP 7 only for beneficiaries receiving per 
reporting period more than 375.000 euro. Costs spent by be neficiaries on external aud it certificates are treated as 
eligible and are reimbursed by the Commission.  
12 Although data concerning DG ENTR did not feed into th e model (as they are not part of the current Chapter 7) 
they do manage the expenditure under the framework programmes. 
 
EN 8   EN Structure of costs of control per project stage
18%
16%
43%11%6%6% Selection of proposals
Negotiation of contracts
Project management
Ex-post audits
Ex-post results
Audit certificates
 
Figure 2:  Structure of costs of controls per project stage for the research, transport and energy policy area, 
including Commission Services: RTD, INFSO, TREN, ENTR, EACI, EACEA, ERCEA, REA, TEN-TEA.  
The wide definition of controls used for this exercise was in line with the COSO13 definition 
of internal control, which inspired the inte rnal control framework in the Commission. These 
controls are not limited to checks and verificatio ns but include all the measures management 
and staff take to ensure the effectiveness an d efficiency of the operations as well as the 
legality and regularity of the underlying financial transactions. 
5. Analysis of the errors found in research, transport and energy policy group by the 
Court of Auditors 
Each year the Court of Auditors selects and au dits a sample of transactions for each policy 
group underlying the Chapters of its annual report. Based on the result of its audits (as well as 
on the assurance derived from the functioning  of the respective supervisory and control 
systems), it gives a specific assessment on th e legality and regularity of underlying 
transactions. Errors that the Court finds are cl assified as either non-quantifiable (for example 
an inappropriate tendering proced ure or non-respect of the deadlin es set by the regulation) or 
quantifiable (as for example declaration of ineligible costs by a beneficiary). 
For the purpose of this current exercise, all e rrors that the Court de tected in 2007 and 2008 for 
research, energy and transport corresponding with the dataset on payments as presented in 
part 3 were analysed. For the purpose of esta blishing the TRE model onl y quantifiable errors 
have been taken into considera tion as only they have a direct measurable impact on the value 
of the underlying transactions. 
Figure 3 below provides an overview of the main types of errors. The e rrors reported by the 
Court were divided into two broad categories:  those with an impact on the error rate and 
those without. Reported errors with an impact  on the error rate are further divided between 
errors originating at the beneficiaries and those for which a root cause lay with the 
Commission. 
                                                 
13 The internationally recognised COSO framework - http://www.coso.org/  
 
EN 9   EN 
 
Figure 3:  shows that errors were cl assified into eight groups. 
The primary and crucial conclusion from the clas sification of errors into the above categories 
was that there was a certain le vel of error which cannot be reduced – due to differences of 
interpretation between the Commission and the C ourt or to issues for which the Commission 
has decided that controls would not be possible or cost-effective.  This concerns in particular 
errors of category D and E mentioned above . Analysis of DAS 2007 and 2008 shows that 
such errors may account fo r an error rate  in excess of 1% - that is to say more than half-way 
towards the 2% threshold. 
For errors caused by legal discrepancy with the Court (category D) the Commission considers 
that no control failure occurred (thus it could not have been a voided in advance). For errors 
for which there were no adequate controls put  in place, remedial measures for preventing 
similar errors in the future would require in troduction of new controls. These new controls 
can prove costly to implement (even to the extent of bei ng cost-prohibitive) or their 
introduction may not diminish errors in the shor t term (for example due to previously defined 
regulatory interpretations). This analysis ther efore confirms the intuitive impression that a 
"zero-error" situation is virtually unachievable an d a certain level of inhe rent error needs to be 
accepted even before taking account of control costs. 
An additional argument that a 'zero error' situation is not achievable  comes from a cost 
benefit analysis of the on-the-spot audit effo rt. There were over 5,300 payments in this policy 
area to beneficiaries of research and other grant programmes in 2008. For research framework 
programmes, payments are made to coordinators which make payments to project participants 
(over 7 per project on average). This means that pa yments were made in respect of cost claims 
related to some 35,000 individual participations in 2008. Control ling each of these on the spot 
would cost some €2 billion. 
 
EN 10   EN For DG Research, even with such extensive a udit coverage, the resi dual error rate would 
remain close to 1%. The expected actual recove ry (as opposed to the error detected) from 
each audited beneficiary is around 1,63% of the EU  contribution. This implies that it is not 
cost-effective to audit benefici aries with a cumulated EU contribution below € 3,7m because 
under this threshold, the expect ed value of error is  less than the margin al cost (€60.000) of 
detection and correction through an on the spot control. The number of beneficiaries below 
the cost-effective audit threshold is over 14.480. If we apply the C ourt's error rate of 3%, this 
would mean that an erro r rate of 0.96% of th e total EU contribution in FP6, cannot be cost-
effectively detected and corrected. 
The above analysis implies that almost 2% of error may be unavoidable (at least at reasonable 
cost) as some 1% relates to differences of legal interpretation with the Court and 0.9% to 
beneficiaries which it is not co st-effective to control on the spot (even though some control 
should be carried out on some of  them for deterrent effect). 
Another significant finding of th is analysis shows that most of the errors with a financial 
impact (A and B category) could have been av oided only by controll ing the project on the 
spot. Improving desk-based ex-ante checks proved  to be the solution for only a very limited 
number of cases (in about 6% of errors). This argument is backed-up by the fact that the Court 
considers the desk-based ex ante controls  before payment in FP6 to be effective14. 
Figure 4 below shows the structure of  errors per typology defined above 
Share of different types of errors (A-H)
0%5%10%15%20%25%30%35%40%
ABCDEFGH
 
Figure 4 : Different types of DAS errors per typology established by the Commission. In order to even out 
potential abnormality of a given year, an average of two years (2007 and 2008) was taken. 
6. Modelling the relationship between the costs of control and errors – description of the 
models and results they provide  
The DAS is based on the internationa lly-accepted Monetary Unit Sampling (MUS) 
methodology, the most commonly-used statisti cal sampling method in auditing (and one 
which is used by the Court of the DAS ). In this approach the auditor: 
                                                 
14 Court of Auditors Annual Report 2008, Chapter 7, Table 2.1 Assessment of selected supervisory and control 
systems, pg 154. Court audit of ex-ante desk checks concerned one of research DGs (DG INFSO). 
 
EN 11   EN • decides what level of confidence is re quired from the sample (effectively, the 
probability of the results of the sample re flecting reliably the situation across the 
whole population) and the materi ality level (which sets a degree of tolerance in the 
accuracy of the underlying transactions). Th e Court, in line with common practice, 
uses a 95% confidence level and a 2% materiality rate; 
• fixes the level of assurance he will draw from the quality of the systems in place (if 
he considers the systems as effective, he may draw some of the 95% overall 
confidence level from these systems: if  not, the whole 95% is provided by the 
examination of a sample of transactions); 
• calculates the sample size required based on the confidence level, materiality level 
and the assurance drawn from systems and selects the sample of transactions accordingly; and 
• audits the selected tr ansactions, records the results a nd uses statistical techniques to 
estimate the level of error in  the population as a whole. 
The statistical basis underlying MUS is the "normal distribution"; which means that (for a 
confidence level of 95%) there is a probability of 95% that the actual error rate across the 
whole population of the tr ansactions will be with in two "standard deviations" from the apex 
of the distribution curve. This apex is called the "Most Likely Error" (MLE) and represents the best estimation of the error rate. However, as some uncertain ty is involved in assessing the 
results of any sampling, this unc ertainty is expressed by two ex treme points (the Upper Error 
Limit (UEL) and the Lower Error Limit (LEL)):  respectively, with a 95% confidence level, 
there is a 2.5% probability that the actual e rror in the population is higher than the UEL and a 
2.5% probability that the act ual rate is lower than th e LEL (see Figure 5 below). 
 
Figure 5 : MUS curve – in 2,5% cases the real error in the popu lation may exceed the upper error limit and in 
2,5% cases the error may be actually smaller than the lower error limit. 
Two models were created for the purposes of this concrete proposal. The first was based on 
achieving a given UEL. This was used notably to identify the cost of control to obtain a 
virtual guarantee that an erro r rate of less than 2% could be achieved. The second model was 
based on the cost of achieving a given MLE, wh ich is the benchmark used by the Court: of 
course in this scenario there would be a 50% pr obability of the error ra te being greater than 
the MLE. 
 
EN 12   EN The population used for the model concentrated on  the latest (2008) DAS, for which 23 errors 
with financial impact were detected by the Court (categories A-E) in this area. The value of these errors as a proportion of the tota l payment audited ranged from 0,72% to 95,23%. 
Extrapolated to calculate an error rate for the whole popul ation (that is including both 
transactions affected by error a nd those which were erro r-free), this gave an overall average of 
2,99%. 
The two models described below examine ho w the average error in the population of 
payments in 2008 might have changed if more or  less had been spent on controls. They are of 
course only indicators  of the potential relationship betw een control and undetected error. 
They involved the use of several assumptions: 
1) all identified unduly paid amounts are immediat ely recovered (whereas in reality there is a 
time lag between detection and recovery and some amounts may be waived or cannot be 
recovered)  
2) the assumed error rate is an average based on the DAS error rate - the same for all 
beneficiaries: if an audit is performed the am ount recovered (in the ba sic model) is 2,99% of 
the audited amount (whereas in fa ct some audits may result for example in 0% or 100% error 
rate) 
3) the assumed error rate for all beneficiaries is  defined as fully systematic (that is that the 
same error is repeated across all programmes of the same beneficiary) whereas in reality it has 
a systematic and non-systematic component  in itself. Commission audits generally involve 
the verification of a sample of a beneficiary's co st claims, the identification of errors and the 
recovery of these. Where an error is likely to have affected non-audited cost claims, the 
Commission may invite a beneficiary to reimburse  an appropriate sum related to these costs 
claims or it may ask the beneficiar y to restate these cost claims to remove the impact of such 
systematic errors. The Commission's audits on FP6 have shown that the non-systematic errors 
account for a significant propor tion (1.63%) of the actual error rate detected. The 
Commission's model however assumes that audits  will clean each beneficiary of error (that 
has fully systematic nature) and therefore overs tates the "cleaning" effect of audits (and 
correspondingly underestimates the likely cost  of removing errors from the population). 
Several simulations were performed based on vari ations in the criteria used: for both models 
therefore more than one scenario is presented. 
The statistical models introduced below are based solely on cost-benefit considerations  and 
do not include key political aspect s which also need to be ta ken into account in fixing the 
TRE level. The dataset (as described in deta il above) consisted of payments to over 5,500 
beneficiaries for a total amount of €7,2 bn . In lin e with the analysis of errors presented above, 
it is assumed that errors can only be detected  "on the spot". Costs spent on controls are 
therefore to a large exte nt fixed in the short-term (€200m) and the marginal cost is calculated 
by adding or removing on-the-spot audits. From the empirical data
15 an average cost per audit 
amounting to €60.000 (including the cost of the re sulting recoveries) per project participant 
was calculated and used in the models presented below. 
                                                 
15 Average cost of an audit per planning audits in FP6 audit strategy.  
 
EN 13   EN In the models the benefits of controls are defi ned as recoveries from audits performed on-the 
spot, however it needs underlini ng that not all error amounts es tablished in the course of 
audits can be actually recovered by the Commis sion and some may be waived or be simply 
unrecoverable due to the financial situation of the beneficiary. Model 1 (as described below) 
is focused primarily on achieving a 2% error rate and does not seek the maximisation of 
benefits of controls at all costs. Model 2 take s account of the benefits  of control by means of 
calculating "a yield" for each coordinator defined as the difference between amounts recovered from the audited coordinator and cost s spent on controlling th is coordinator (on the 
spot controls are carried out when the yield is positive: the expected recovery exceeds the cost 
of the individual control). 
6.1. MODEL 1
: achieving a given UEL 
To give a virtual guarant ee of achieving a "green light" (under 2% error level) in this Chapter 
of the Court's report, the Commission would need  to carry out sufficient controls on the spot 
to make it almost certain that the Upper Error Limit was 2%. This model estimates the level of marginal costs (on top of the existing ex an te controls) that would need to be spent to 
guarantee a level of error in th e population of 2%. For comparative purposes the costs of 
achieving a given MLE were also examined. 
 The model took the 2008 population of payments and calculated the cost of control based on 
audits in the year of all of the largest benefi ciaries together with sample-based checks on the 
remainder. Several scenarios were used, assu ming audits of different numbers of the top 
beneficiaries with a sample of the remainder. Figure 6 below illustrates the results of the different scenarios: 
0 20 40 60 80 100 120 140 160 180
Millionstop 100top 200top 300top 350top 400top 500costs for different scenario's
2% MLE
2% UEL
 
Figure 6:  Model 1 presents scenarios to achieve levels of M LE and UEL of 2% by controlling on the spot the top 
100, 200, 300, 350, 400 and 500 beneficiaries and a sample of the remainder. 
 
EN 14   EN Figure 6 illustrates an incremental rise in costs of  control related to additional audits that need 
to be executed on the spot. It should be noted that the fixed costs of €200 million (those of the 
other stages in the control chain)  are unaffected by this analysis. 
The way of virtually guaranteeing a "green lig ht" would be to carry  out enough control to 
deliver a UEL of 2% (as explaine d above - a situation when we ar e more or less sure that the 
error found in the population will not exceed 2% , the MLE at this stage is almost 0%). 
Regardless of the chosen audi t strategy (whether we audit t op 100 or top 500 beneficiaries) 
costs that would need to be spent on controls to achieve 2%  (an equivalent of DAS green 
light) would be considerable – ranging from €1 55m to €166 m. The cost of being sure of 
achieving the 2% benchmark would therefore be prohibitive (at over €150 million). Even the cost of achieving an MLE of 2% is considerable (at over €90 million), compared with what is being spent currently on on the spot controls (le ss than €50 million) to arri ve at an error rate 
of almost 3%. 
The above analysis is based on one year onl y – 2008. In order to have a wider field for 
examination and results there from, the data from 2007 was introduced to the model.  
Several parameters needed to be adapted, such  as the number of sampled transaction for the 
policy group (180 instead of 150 in 2008), number of payments pe r beneficiary, average error 
detected (higher in 2007 than in 2008). Succeeding calculations  revealed that in order to 
obtain a guarantee of having in the population an  error not higher than 2% (ULE) even more 
resources would need to be spent on additional controls - around €190m. This result exceeds the outcome for 2008 substantially, in large part due probably to the higher error rate in the 
DAS 2007 for this Chapter. This in turn was possi bly partly due to the fact that the numbers 
of on the spot audits  by the Commission increased sign ificantly between 2007 and 2008 and it 
could therefore be expected that the Commission "cleaned" mo re projects of error in 2008 
than in the previous year. A dditionally, the proportion of fi nal payments was increasing as 
FP6 came into full swing (typically  final payments are the most er ror prone ones as it is at this 
stage that the actual cost s of a project are finally  verified). Further anal ysis also suggests that 
the decrease in the error rate in 2008 might ha ve been due to phasing out of FP5, which in 
contrast to FP6 did not include  an audit certification policy. 
6.2. MODEL 2
: targeting the MLE 
The second model is based on the most likely error,  using the same basic data as for Model 1. 
The basic principle, following statistical theo ry, is that each Euro in the population of 
payments in 2008 was affected by the same percen tage of error as the population audited by 
the Court (2.99%). This percenta ge was converted into a "yield " per beneficiary: defined as 
the difference between the assumed recovery from  an audit (calculated as 2.99% of the total 
payment to that beneficiary) and the cost of auditing that beneficiary. The cost of auditing a 
single beneficiary was calculated on the basis of the audits carried out in the research policy 
group relevant to the established dataset in 20 08 (733) and the costs re lated to on-the-spot 
controls (including recovery of errors detect ed): €60.000. For the purposes  of the exercise it 
was assumed that the remaining control costs were "fixed" as on-the-spot control was identified as the principal tool for detecting and correcting errors. 
A curve can then be derived with multiple points which can be used to calculate the cost of 
"adding" individual on-the-spot co ntrols, increasing the cost a nd reducing the error in that 
beneficiary. The ranking of the beneficiaries for this curve can be done in two ways: in the 
best-case scenario we assume that we will visi t the beneficiaries with the highest yield first 
 
EN 15   EN (assuming the localisation of errors  is known), in a second case we  choose the beneficiaries to 
be audited randomly. 
Scenario A: MODEL 2A "best-case scenario"  
In the best case scenario the beneficiaries were ranked ranging from the ones bringing the 
highest yield to the lowest. So the errors that are subsequently removed are removed in the most cost-effective way. At the start point we ha ve an error rate of 2,99% . In order to reach an 
error rate of 2,9% we need to get rid of 0,09%  error in the populati on of €7,2 bn spread over 
5538 beneficiaries, which means that we woul d need to recover a dditionally (0,09% * €7,2 
bn) €6,5m euro. In such a way we progress movi ng down from the initial level of error of 
2,99%, through auditing additional benefi ciaries and reducing the error rate. 
Scenario B: MODEL 2B "random selection"
 
In this scenario the coordina tors were not ranked by yield, their selection was random. The 
random simulation was run 1000 times, which gave a straight line with 10 individual points, 
at each of which the corresponding cost of c ontrols can be derived. More points could be 
generated on this line but this would not change the overall results. 
Scenario 2AB: scenarios 2A and 2B combined – TRE range  
When the product of models 2A and 2B are co mbined, one could confidently presuppose that 
the TRE point would be located somewhere betw een the best-case scenario and a more real-
life (random) situation described above. The corresponding results ar e set out in Figure 7 
below. This shows that the best-case scenari o, which assumes we know where the errors are 
located, would be highly cost-effective in reduci ng the error: the reductio n to 2.5% error rate 
would cost less than €1 million. However, th e random scenario, which can effectively be 
considered as a "worst-case" scenario (a s the Commission does have information which 
enables it to target risks), would imply a much  higher cost to reach the same 2.5% target 
(some €55 million). The range between these two figures is very wide: however, it is not unreasonable to assume that the cost of reduc ing the error rate (MLE) to 2.5% could lie 
somewhere towards the median of the range €1 million to €55 million: somewhere around €28 million. This would imply increasing the current cost of on-the-spot controls by almost 80%. The chart below visually presents two, co mbined scenarios for the Model 2 with the 
TRE range lying between the two lines. 
 
EN 16   EN Scenario 2AB - TRE range
1101001.00010.000100.0001.000.00010.000.000100.000.0001.000.000.000
2,90%
2,80%
2,70%
2,60% 2,50%
2,40%2,30% 2,20%
2,10%2,00%Error RateControl resources
Random Best Case
  
Figure 7 : TRE range between the best case and random scenarios 
The curves relate to the best-case scenario a nd a given random scenario  for the Model 2. They 
present the reduction in the error rate, which is achieved by dedicating additional resources 
for control. A logarithmic scale (also know as ra tio scale) has been used when plotting the "y" 
values in order to reduce the large range of values16. 
Scenario 2C: "reducing the costs of control"  
To provide fuller information for the inter-Inst itutional discussions on the concrete level of 
TRE in this sector, the Commission services al so examined what might happen if the number 
of on-the-spot controls were to be reduced, t hus accepting a higher error rate. This is also a 
quasi best-case scenario. In this case the bene ficiaries were ranked, starting from the one who 
brings the smallest yield (benef iciaries with the lowest amount of error to be recovered by the 
Commission). 
This analysis gives indicative error rates for ach ieving a given (higher) MLE than that of the 
Court. The costs associated can be assumed to be the minimum to achieve the given level and are (as explained earlier) incremental. The tabl e below presents different (higher than the 
current 2,99% MLE) levels of error that could be implied by spending less on on the spot controls (by diminishi ng number performed). 
 
                                                
 
16 A logarithmic scale is a scale of measurement that uses  the logarithm of a value instead of the value itself. 
Logarithmic scales have been explicitly designed to reduce  large range of values to a more manageable range. 
The logarithmic scale used here is one in "factors of te n" which makes use of so-called decades on the vertical 
axis.  A decade is a factor of 10 difference between two numbers (as order of magnitude difference) rather than 
unit steps. 
 
EN 17   EN Error rate Reduction in control costs 
3,5% €8 million 
4% €15,5 million 
4,5% €22 million 
     Table 2 : predicted higher error rates corresponding to reduced costs of controls 
6.3. The implications of the analysis for a TRE level  
The analysis above shows that gu aranteeing an error rate of 2%  would be prohibitive in cost 
terms. Likewise, any reduction in the current erro r rate could be costly to achieve and would 
be unlikely to be cost effective. Of course, the error rate will evolve year-on-year as the balance of framework programmes and other activ ities change: and statistical sampling also 
has its own uncertainties which can affect the error rate in a give n year. The Commission 
services also believe that Framework Programme  7 (for which most payments to date have 
been low-risk pre-financing) could be more error-prone than FP6 when it comes to treating 
cost claims due to the reduction in the numbers of audit certificates and the low number of 
beneficiaries eligible for the acceptance of their personnel costing methodologies (which 
would permit the claiming of aver age personnel costs rather than  actuals, thus reducing the 
risk of error). As a result, it would be pr udent to fix a range fo r TRE. This could be 
reconsidered in the light of any major changes in the control environment. 
It is clear from the above that the TRE level should be fixed in the current "yellow" zone of 
the Court: analysis by DG Research has indicat ed that reducing the mu lti-annual error rate for 
its payments alone would cost some €126 million for FP7 (2,100 audits multiplied by €60,000 unit cost). The analysis above indicates that reducing the current MLE significantly would be 
likely to be costly. Similarl y, the 2008 error rate  of 2.99% may evolve as FP7 comes on-
stream and the higher inherent risk may influence the error rate upwards. 
In this light, the Commission proposes a TR E level in the yellow range (2%-5%). A DAS 
error rate around the middle of this range would be acceptable and justified. Beyond 
this level, additional action would be taken to reduce the error rate through increased controls addressing the major causes of erro r and taking into account the multi-annual 
nature of the expenditure. This rate should  cover all Research Framework Programmes 
irrespective of the Chapter of the Court's annual report in which they are included as 
well as all activities included in this anal ysis even if they are reallocated to other 
Chapters of the Court's annual report in future.  
Fixing a tolerable rate of error higher than 2%  could allow a fundament al refinement of the 
current control strategy. In particular the ma gnitude of controls (a udit coverage) and focus 
could be addressed with a view to lower the control burden both for the major beneficiaries 
(where this is cost-effective) and the Com mission (and thus reduc e the disincentive to 
beneficiaries to participate in Union programmes). Whilst maintaining an adequate level of representative contro ls (to support the regular need  for assurance on sound financial 
management), such a revised control strategy w ould allow to focus less on audit coverage and 
more on targeted risk based aud its and fraud prevention controls. On such basis, a sufficiently 
solid accountability framework could be ensured whilst maintaining a sufficient deterrent to 
 
EN 18   EN irregular use of EU research f unds. As a result, scarce financial and human resources could be 
reoriented to other priori ty research policy areas. 
 
EN 19   EN Annex 1: Glossary  
 
Audit certificates:  independent certification of the co sts incurred and claimed under the FP6 
project for a given period. Their purpose was to give the Commission reasonable assurance 
that eligible costs are claimed in accordance with the relevant provisions of the contract. The 
auditor must be competent and independent from the contractor. The Certificate on the 
Financial Statements (CFS) in FP7 replaced the FP6 Audit Cer tificates. FP7 introduced also 
additional notions of audit certificates. 
Ex-ante verifications:  all checks put in place by the authorising officer responsible in order 
to verify operational and financial aspects of an operation prior to its execution. The purpose 
of this verification is to ascertain compliance with the applicable provi sions and the principle 
of sound financial management.  
Framework programmes:  since 1984, research and innova tion activities of the EU are 
grouped in one large programme called th e Framework Programme (FP). Research 
framework programmes are the EU's main financ ial and legal instruments to implement the 
European Research Area. The current FP7 ( 2007-2013) is designed as a key contribution to 
the EU's strategy for growth and jobs. 
FP6/7 audit strategy:  audit strategy established join tly by the Commission's research 
services in order to assess the legality and regularity of fina ncial transactions under FPs and 
provide the basis for corrective and recovery mechanisms and, in particular, to eliminate 
errors occurring in the cost statements provide d by the beneficiaries. It  aims at reaching the 
residual multiannual level of error of  2% by the end of the Programme. 
Full time equivalent (FTE):  ratio used to measure a person's involvement in an activity. FTE 
= 1 means that the person is equivalent to a full-time employee; whereas an FTE = 0.5 
indicates that the employee wo rks only half-time on a certain  project (e.g. on performing a 
control activity). 
Most Likely Error (MLE) : an error that is equivalent to the best estimation using statistical 
techniques of an error that can be found in a population having a normal distribution. It is 
calculated based on the errors found in an audited sample. 
Monetary Unit Sampling (MUS) : a method of statistical samp ling used to assess the amount 
of monetary misstatement in a population. It is the most co mmonly used sampling method in 
auditing. 
Non-systematic portion of an error rate:  the part of an error rate  detected during control 
that cannot be applied to other payments to th e same beneficiary because its nature is random. 
Non-quantifiable errors:  errors that do not have direct measurable financial impact on the 
value of a transaction. These may be errors which concern non-observance of a condition for 
payment or other compliance issue for exampl e non-respect of deadlines set by a regulation 
but which does not affect the amount paid. 
 
EN 20   EN Normal distribution : a symmetrical distribution in statis tics, represented on a graph by a bell 
curve, in which the class with the highest freq uency is at the centre and the classes with the 
lowest frequency are at each end. 
On-the-spot control:  an optional control (distinct from both the ex-ante ve rification and the 
initiation of an operation) that contributes to an increase in the assurance of the authorising officer by delegation concerning the use of EU funds (for which the Commission is 
accountable). On the spot control checks the proper implementation of operations already 
financed (a posteriori - intermediate or final payment) as regards their regularity, conformity 
and sound financial management. These checks  are done on documents and/or on-the-spot 
and can be selected based on a st atistical sample, judgementally by means of a risk analysis or 
randomly. 
Quantifiable errors:  errors that affect the eligibility  conditions and th at have direct 
measurable impact on the value of the underl ying transactions (error rate), for example 
reimbursement errors resulting from ineligible  items included in the cost statements by a 
beneficiary. 
Systematic portion of an error rate:  a part of an error rate that can be applied to other 
payments to the same beneficiary as its underlyi ng causes are of the same nature and they are 
highly likely to be repeated in other cost statements.  
 
EN 21   EN Annex 2: Definition of the dataset on payments:  
 
Budget lines description Reasons for exclusion 
Payments < 20.000 euro Small grants and paym ents to natural persons (as experts) were 
eliminated. Whereas they could be as error prone as bigger payments the financial impact of errors detected therein is overall less material. 
Contributions from third 
countries (payments received 
from EEA countries) These amounts are used to fina nce expenditure related to 
Framework Programmes in the future, so their inclusion would 
result in double counting 
Budget title 10 – direct research Direct research is managed by the Joint Research Centre. This 
spending consists mainly of a large number of small value 
transactions in a form of direct actions, which are quite distinct from other research spending and require different control strategies. 
Administrative expenditure (budget chapters XX 01) The proportion of administrative expenditure was indirectly 
included in the calculation of cost  of controls by using average 
costs for the estimates on “Human resources".  Administrative 
expenditure will be analysed sep arately in the upcoming second 
wave of Commission's Communication on concrete TRE proposals.  
Framework programme 4 and preceding FPs Payments related to Framework programmes prior to FP5 were 
not taken into consideration as these programmes are now phasing out. 
Media programme Management of this pr ogramme was transferred to a different 
Directorate General, which falls into another chapter of the ECA's annual report. (To assure the comparability, costs of 
controlling this programme were not  included on the costs side).