Feasibility study on the evaluation of the state of the EU customs union : draft final report.
Date: 2016-06-28
Institutions: Directorate-General for Taxation and Customs Union
Subjects: customs formalities, customs harmonisation, customs regulations, customs union, feasibility study, single market

Draft Final Report 

Framework contract 
DIGIT/R2/PO/2009/027 ABC II 

Feasibility study on the evaluation of the state of the EU 
Customs Union  

May 15th, 2012 

Deloitte  
Berkenlaan 8C 
1831 Diegem 
Belgium 

KP-02-16-400-EN-N 

i 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Contents 

Executive summary ................................................................................................................. 6 

1 

2 

3 

Introduction ..................................................................................................................... 9 

Study objectives ............................................................................................................. 10 

Evaluation standards and methodology .......................................................................... 11 

3.1 

3.2 

3.3 

3.4 

3.5 

Introduction ........................................................................................................... 11 

Commission evaluation standards .......................................................................... 11 

Intervention logic ................................................................................................... 12 

Evaluation criteria ................................................................................................... 14 

Analytical framework .............................................................................................. 14 

4 

Intervention logic of the Customs Union ........................................................................ 16 

4.1 

4.2 

Introduction ........................................................................................................... 16 

Needs ..................................................................................................................... 16 

4.2.1 

Needs of governments .................................................................................... 17 

4.2.2 

Needs of society .............................................................................................. 18 

4.2.3 

Needs of businesses ........................................................................................ 18 

4.3 

4.4 

4.5 

4.6 

4.7 

4.8 

Objectives .............................................................................................................. 19 

Inputs ..................................................................................................................... 20 

Activities ................................................................................................................ 22 

Outputs .................................................................................................................. 26 

Outcomes (effects) ................................................................................................. 26 

Impacts .................................................................................................................. 27 

5 

Evaluation questions ...................................................................................................... 29 

5.1 

5.2 

5.3 

Overview of the evaluation questions considered ................................................... 29 

Relationship between evaluation questions and evaluation criteria ........................ 30 

Relationship between the evaluation questions and the intervention logic ............. 31 

6  Analytical framework for the evaluation ......................................................................... 32 

6.1 

6.2 

Introduction ........................................................................................................... 32 

Judgement criteria and indicators ........................................................................... 33 

ii 

 
 
6.2.1 

Judgement criteria ........................................................................................... 33 

6.2.2 

Indicators ........................................................................................................ 34 

6.3 

Data sources .......................................................................................................... 36 

6.3.1 

Approach followed to define the data sources per indicator ............................ 36 

6.3.2 

Suggested data sources for the evaluation ...................................................... 36 

7 

Proposed budget and timeframe .................................................................................... 44 

7.1 

7.2 

Introduction ........................................................................................................... 44 

Estimation of mandays, budget and timeframe ...................................................... 46 

8  Conclusions ................................................................................................................... 49 

9  Annexes ......................................................................................................................... 52 

9.1 

9.2 

9.3 

9.4 

9.5 

Annex 1: Overview of tasks performed for the study .............................................. 52 

Annex 2: Reference list........................................................................................... 53 

Annex 3: List of interviewees .................................................................................. 55 

Annex 4: Glossary of terms and concepts used ...................................................... 56 

Annex 5: Definitions of evaluation criteria applied for the feasibility study ............ 59 

9.6 
Annex  6:  Mapping  of  the  evaluation  questions  with  the  evaluation  criteria  and  the 
intervention logic .............................................................................................................. 61 

9.7 
Annex  7:  Long  list  of  data  to  be  collected  per  data  source  (priority  1  judgement 
criteria).............................................................................................................................. 63 

9.8 

Annex 8: Suggested step-approach for setting up a web-based survey ................. 79 

9.9 
Annex  9:  Suggested  indicators  to  monitor  changes  in  the  Customs  Union’s 
environment ...................................................................................................................... 83 

9.10  Annex  10:  Overview  of  workshops  held  with  DG  Taxud  and  other  stakeholders  in 
the framework of the study ............................................................................................... 85 

9.10.1  Workshop 27 February 2012 ........................................................................... 85 

9.10.2  Workshop 23 March 2012 ............................................................................... 85 

iii 

 
 
Table of figures 

Figure 1: An illustration of the components of an intervention logic .................................... 14 

Figure  2:  Identified  needs  of  governments,  society  and  businesses  vis-à-vis  the  Customs 
Union .................................................................................................................................... 17 
Figure 3: Identified strategic objectives ................................................................................ 20 
Figure 4: Identified inputs .................................................................................................... 22 
Figure 5: Activities of the Customs Union – core processes .................................................. 23 
Figure 6: Activities of the Customs Union – supporting processes ........................................ 25 
Figure 7: Identified outputs of the Customs Union ............................................................... 26 
Figure 8: Intended outcomes of the Customs Union ............................................................. 27 
Figure 9: Identified intended impacts on businesses, governments and society ................... 28 
Figure 10: Relationship between evaluation questions and the intervention logic ................ 31 
Figure 11: Project phases ..................................................................................................... 52 
Figure 12: Stages associated with implementing online surveys ........................................... 80 
Figure 13: Example of a web questionnaire .......................................................................... 82 

iv 

 
 
 
 
Table of tables 

Table 1: Mapping of judgement criteria priorities per evaluation question ........................... 34 

Table 2: Mapping of number of indicators per data source per evaluation question (priority 1)
 ............................................................................................................................................. 38 
Table 3: Differences between scenarios for the budget estimation ....................................... 45 
Table 4: Estimation of man days per scenario ...................................................................... 47 
Table 5: Suggested milestones for the 2012 evaluation ....................................................... 48 
Table 6: People interviewed during the preparatory phase ................................................... 55 
Table 7: Participants to the brainstorming session on needs of businesses .......................... 56 
Table 8: Definitions of evaluation criteria applied for the feasibility study  ........................... 59 

Table 9: Overview of evaluation questions, criteria and relationship on the intervention logic
 ............................................................................................................................................. 61 
Table 10: Long list of data to be collected at central Member State level (priority 1) ............ 63 
Table 11: Indicator table for survey among customs authorities .......................................... 69 
Table 12: Indicator table for survey among businesses ........................................................ 74 
Table 13: Indicator table for survey among non-customs authorities ................................... 78 

v 

 
 
 
 
Executive summary 

The  objective  of  this  study  is  to  design  a  comprehensive,  detailed  and  feasible  framework  that 

will  allow  the  European  Commission  (in  particular  DG  TAXUD)  to  evaluate  the  state  of  the 

Customs Union. The overall aim of the evaluation is to improve the Customs Union’s processes 

and its implementation as seen by the stakeholders – taking into account the strategic objectives 

of the Customs Union. 

The  approach  followed  for  this  study  is  in  line  with  the  Commission’s  evaluation  standards  – 

bearing  in  mind  that  the  Customs  Union  entails  many  more  aspects  than  a  single  policy 

intervention. 

In a first step, an intervention logic has been established to create a common understanding of 

the main rationale behind the Customs Union as it stands today. The intervention logic presents 

the  needs  of  the  stakeholders,  the  strategic  objectives,  inputs,  activities,  outputs,  effects 

(outcomes)  and  impacts  of  the  Customs  Union.  The  fact  that  common  agreement  could  be 

reached on the intervention logic of the Customs Union may be considered as an achievement in 

itself of this project. 

Following this, significant efforts were made to prioritise the many possible evaluation questions, 

based on an assessment of the extent to which the answers on the questions could be used to 

inform decision makers. The following evaluation questions have been retained to be assessed 

in terms of feasibility: 

•  To  what  extent  are  the  strategic  objectives  of  the  Customs  Union  still  relevant  and 

sustainable for the next decade?  

•  To what extent are the core processes of the Customs Union uniformly  applied  across the 

EU  Member  States 

(clearance  processes,  enforcement,  control  processes,  data 

management, economic operator processes, risk management)?  

•  Are the core processes of the Customs Union being efficiently  performed? Are there great 

differences  across  Member  States  in  the  efficiency  (clearance  processes,  enforcement, 

control processes, data management, economic operator processes, risk management)?  

•  What  have  been  the  effects/outcomes  of  the  major  recent  policy  initiatives  in  the  field  of 

Customs Union for the Customs Union and its stakeholders?  

6 

 
•  What is the  quality  of the  service  provided  by  the Customs Union to  its main stakeholders 

(involvement  of  stakeholders  in  decision-making,  reachable,  costs  of  service  provision, 

paperless environment, skills and competences of staff, treatment of AEO)? 

•  Has  the  Customs  Union  achieved  its  strategic  objectives?  Has  it  generated  its  intended 

effects?  To  what  extent  has  the  Customs  Union  in  particular  contributed  to  these  effects? 

Are there any unintended effects and how significant are they?  

•  How has the Customs Union absorbed the enlargement of the EU? 

In  terms  of  identification  of  judgement  criteria  for  the  evaluation  questions,  the  feasibility 

assessment has lead to the conclusion that it is feasible to translate all evaluation questions into 

judgement criteria, with the exception of evaluation question 1, dealing with the sustainability of 

the  strategic  objectives  of  the  Customs  Union  for  the  next  decade.  For  the  other  evaluation 

questions,  no  less than  187 (sub-) judgement criteria have  been  identified. Trying to assess all 

(sub-)  judgement  criteria  in  one  single  evaluation  study  is  not  feasible.  Therefore  a  further 

prioritisation of (sub-) judgement criteria will need to take place, focussing on those aspects that 

DG TAXUD deems most relevant to be evaluated.  

In terms of identification of indicators per (sub-) judgement criteria, the feasibility assessment has 

lead  to  the  conclusion  that  it  is  feasible  to  translate  each  (sub-)  judgment  criterion  into  several 

quantitative  and/or  qualitative  indicators.  The  number  of  indicators  is  in  direct  relation  with  the 

number  of  (sub-)  judgement  criteria.  The  187  (sub-)  judgement  criteria  have  given  way  to  the 

identification  of  no  less  than  570  possible  indicators.    In  particular  the  evaluation  questions 

dealing  with  the  uniformity  and  the  efficiency  of  the  Customs  Union’s  core  processes  are 

characterised by a high number of indicators. Trying to collect data for each of these indicators in 

one  single  evaluation  study  is  virtually  impossible.  It  is  recommended  to  focus  the  prioritisation 

exercise  at  the  level  of  (sub-)  evaluation  questions  and  (sub-)  judgement  criteria,  to  allow 

sufficient data basis for conclusion making per (sub-) judgment criterion.   

For  a  prioritised  set  of  (sub-)  judgement  criteria  the  study  team  has  defined  suggested  data 

sources  for  each  indicator.  For  the  evaluation  we  propose  using  the  following  data  collection 

methods:  

•  Desk research of existing data; 
•  Collection of additional data from the Member States; 
•  Three  different  web-based  surveys  targeting  businesses,  customs  authorities  and  non-

customs authorities; 

In-depth interviews with customs authorities and businesses 

• 
•  Case studies. 

7 

 
 
In  terms  of  data  availability,  it  appears  that  some  relevant  data  are  already  available,  but  that 

significant efforts will be required to collect additional data from the Member States and through 

web-based  surveys.  For  more  detailed  data  the  use  of  case  studies  is  suggested.  In-depth 

interviews are recommended to complement the data collected and to gain further understanding 

on the reasons behind the findings.  

The budget needed for the actual  evaluation  will  depend on the outcome of the prioritisation  of 

(sub-) evaluation questions and (sub-) judgement criteria – information which is not yet available 

at this stage. Three scenarios have been developed for the budget estimation of the evaluation, 

with varying levels of data collection activities. It should be emphasised that for each of the three 

scenarios it is assumed that there will be a significant reduction in the number of indicators to be 

considered. Depending on the scenario chosen, the total cost of man days for the evaluation is 
estimated  to  be  between  €150,000  and  €242,000.  (travel  budget  and  translation  budget 

excluded).  

If the evaluation project were launched by mid May 2012, it is estimated that the results would be 

available to DG TAXUD by mid November 2012.  

8 

 
 
1 Introduction 

This draft final report is structured as follows:  

•  Executive summary 

•  Chapter 1 presents the structure of this report.  

•  Chapter 2 recalls the objective and outputs of the study, as well a view on the scope of 

the assignment.  

•  Chapter 3 presents the evaluation standards that apply for Commission evaluations and 

explains how this report contributes to those.  

•  Chapter 4 contains the intervention logic. 

•  Chapter 5 includes the evaluation questions.  

•  Chapter 6 includes the analytical framework for the evaluation. 

•  Chapter 7 includes an assessment of the required budget and time frame to conduct the 

evaluation study. 

•  Chapter  8  presents  our  conclusions  on  the  feasibility  to  conduct  an  evaluation  of  the 

state of the Customs Union.  

•  The annexes are presented at the end of the report  

9 

 
2 Study objectives 

As  requested  by  the  Terms  of  Reference,  the  general  objective  of  the  study  is  to  design  a 

comprehensive,  detailed  and  feasible  framework  that  will  allow  the  European  Commission  (in 

particular  DG  TAXUD)  to  evaluate  the  state  of  the  Customs  Union.  The  overall  aim  of  the 

evaluation is to improve the Customs Union’s processes and its implementation as seen by the 

stakeholders – taking into account the strategic objectives of the Customs Union. 

The specific objectives of the project are: 

•  To design and conduct a study to propose a feasible evaluation methodology, focussing 

on the  activities  of import,  export and transit. The evaluation framework should  support 

DG  TAXUD  in  designing  a  stable  mechanism  to  support  the  improvements  of  the 

Customs  Union’s  main  processes  and,  ultimately,  contribute  to  organisational  reform 

and/or refining or re-engineering of the EU customs policy; 

•  To design this framework in a way which will allow to: 

o 

reveal pain points and shortcomings and thus support their effective targeting; 

o  support  the  choice  of  policy  methodologies,  sourcing  strategies  at  the  EU  and 

Member State level;  

o  support planning and improve resource allocation and skills;  

o  enable  development  of  methodologies 

for  measuring 

the  business  process 

performance.  

•  To contribute to survey design which will allow the development of SMART1 indicators to 

make  an  informed  judgement  of  the  functioning  of  the  main  processes  of  the  EU 

Customs Union. 

The main beneficiary of this feasibility study will be the European Commission, specifically DG 

TAXUD, which will design and manage the final evaluation to follow 

1 SMART stands for: specific, measurable, accepted, realistic and time-bound 

10 

 
                                                      
3 Evaluation standards and 

methodology 

3.1 

Introduction  

This  chapter  situates  the  approach  and  output  of  the  project  towards  the  Commission’s 

evaluation standards. Furthermore, it presents the evaluation criteria for the evaluation study, as 

well  as  the  main  systematic  tools  that  were  elaborated  during  this  study:  the  intervention  logic 

and the analytical framework. 

3.2  Commission evaluation standards 

This section presents the evaluation standards that should be used for any type of evaluation – 

and  therefore  also  the  evaluation  of  the  current  state  of  the  Customs  Union.  The  proposed 

evaluation  standards  are  building  further  upon  the  Commission  evaluation  standards,  as 

specified in the Commission Secretariat-General report, called “Evaluation Standards – Annex to 
Communication "Responding to Strategic Needs: Reinforcing the use of evaluation”2. 

The  Commission’s  evaluation  standards  aim  to  ensure  relevant  and  timely  evaluations  of  high 

quality. The standards are an integral part of the Commission’s Internal Control Standard n°23 on 

evaluation.  They  are  binding  and  the  way  they  are  implemented  can  be  audited.  The  evaluation 

standards apply to Commission evaluation of policy instruments such as expenditure programmes, 

legislation  and  other  non-spending  activities,  as  well  as  to  the  different  temporal  types  of 

evaluations (ex-ante, mid-term, ex-post). 

The Commission distinguished five categories of evaluation standards.  

A.  Resources and organisation of evaluation activities (A1-A3): for example, each DG should 

have  an  evaluation  function  with  a  clearly  defined  responsibility  for  coordinating  and 

monitoring  evaluation  activities  of  the  DG;  each  DG  should  ensure  that  human  and 

financial resources are proportionally allocated for  the evaluation. 

B.  Planning  evaluation  activities  (B1-B5):  for  example,  an  annual  evaluation  plan  and  an 

indicative multi-annual evaluation programme should to be prepared by the evaluation unit 

in consultation with the other units in the DG. 

2 http://ec.europa.eu/dgs/secretariat_general/evaluation/docs/standards_c_2002_5267_final_en.pdf 

11 

 
                                                      
C.  Designing  evaluations  (C1-C3):  evaluation  design  needs  to  set  clear  and  specific 

objectives and appropriate methods and means for managing the evaluation.  

D.  Conducting evaluations (D1-D5):  

•  The  evaluation  must  be  conducted  in  such  a  way  that  the  results  are  supported  by 

evidence and rigorous analysis. 

•  All actors involved in evaluation activities must comply with principles and rules regarding 

conflict of interest.  

•  The final evaluation reports must as a minimum set out the purpose, context, objectives, 

questions, information sources, methods used, evidence and conclusions. 

•  The  quality  of  the  evaluation  must  be  assessed  on  the  basis  of  the  pre-established 

criteria  throughout  the  evaluation  process  and  the  quality  criteria  must  as  a  minimum 

relate  to  relevant  scope,  appropriate  methods,  reliable  data,  sound  analysis,  credible 

results, valuable conclusions and clarity of the deliverables. 

•  The  evaluation  questions  should  reflect  the  following  evaluation  issues  whenever 

relevant: 

effectiveness, 

efficiency/cost-effectiveness, 

relevance, 

coherence, 

sustainability, utility and/or community added value, and where relevant the contribution 

to broader strategic objectives. Additional evaluation issues may also have to be added 

to the terms of reference. 

E.  Dissemination and utilisation of evaluation results (E1-E5): for example, evaluation results 

must  be  communicated  effectively  to  all  relevant  decision-makers  and  other  interested 

stakeholders/parties. 

This feasibility study has as objective the definition of the methodology to perform the evaluation of 

the current state of the Customs Union. In particular it is translating the abovementioned standards 

with  regard  to  “conducting  evaluations”  (Category  D  of  the  evaluation  standards)  to  a  concrete 

methodology for the evaluation of the Customs Union.  

3.3 

Intervention logic 

The first stage of an evaluation study usually consists in examining the intervention strategy, in 

other  words,  the  rationale  of  a  policy  intervention,  and  its  logic.  In  the  intervention  logic,  the 

needs  or  problems  of  stakeholders  are  taken  as  a  starting  point.  Based  on  these  needs,  the 

objectives of the policy intervention are defined. In the next step, the intervention logic identifies 

the  inputs,  activities,  outputs,  outcomes  and  (expected  and  unexpected,  direct  and  indirect) 

impacts of the policy intervention. It is described by using causality assumptions.  

Whereas  the  Customs  Union  entails  many  more  aspects  than  a  single  policy  intervention,  an 

intervention  logic  has  been  established  at  the  offset  of  the  project  to  create  a  common 

12 

 
 
understanding  of  the  main  rationale  behind  the  Customs  Union  as  it  stands  today.  The 

intervention logic has been built up in line with the figure below. 

13 

 
 
Figure 1: An illustration of the components of an intervention logic 

3.4  Evaluation criteria 

Based on the intervention logic, a prioritisation has taken place of the main areas of the Customs 

Union that the evaluation study should focus upon as well as the evaluation criteria that would be 

most relevant to assess the function of each of these areas.  

The most commonly used evaluation criteria for evaluation studies are as follows: 

•  Relevance;  

•  Effectiveness;  

•  Efficiency;  

•  Utility and impact; 

•  Coherence; 

•  Sustainability. 

The Glossary in Annex 9 contains a more detailed description of these evaluation criteria. 

The prioritisation of areas of the intervention logic and evaluation criteria has helped to define a 

list of evaluation questions to be assessed in terms of feasibility. These evaluation questions are 

presented in the next chapter. 

3.5  Analytical framework 

14 

 
 
 
 
An analytical framework organises the evaluation questions into a single structure and assigns to 

each evaluation question one or several judgement criteria. This procedure makes sure that all 

the evaluation questions are fully covered in terms of scope and identifies the indicators that will 

feed the analysis. The analytical framework can therefore be defined as a logical roadmap of this 

part of the study that guides the consultants in their analysis.   

The analytical framework allows the evaluator to: 

identify the judgement criteria and indicators to be collected; 

•  map and organise all the study questions to be answered; 
• 
• 
•  build a common vision of the work to be performed; 
• 

identify the sources of information; 

structure the tasks to be performed, the findings and the reporting. 

The establishment of a solid analytical framework for evaluation is built on the intervention logic 

and the first set of evaluation questions. For each evaluation question, the evaluation framework 

aims to provide: 

• 

the judgement criteria to allow the evaluator to formulate a judgment and cover all of the 

relevant ground relating to each question/criterion; 

•  potential indicators that will be used to feed the judgment; 
• 

the  information  sources  that  will  be  used  to  collect  relevant  qualitative  and  quantitative 

data;  

15 

 
 
 
4 Intervention logic of the Customs 

Union 

4.1 

Introduction  

At  the  offset  of  the  project  an  intervention  logic  has  been  established  to  capture  the  rationale 

behind the current functioning of the Customs Union.  

It is clear that an intervention logic is an instrument that simplifies reality, and that the reality of 

the  functioning  of  the  Customs  Union  is  much  more  complex.  The  schematic  representation  of 

the  intervention  logic  however  allows  to  better  evaluate  the  interventions  of  all  players  and  to 

better monitor and inform decision-making.  

4.2  Needs 

We  started  with  defining  the  main  needs  of  the  main  stakeholders  that  have  underpinned  the 

Customs Union as it exists today. A distinction has been made between the needs of:  

•  Governments, both at EU level and at national and regional level;  

•  Society (including non-governmental organisations, citizens, etc); 

•  Businesses. 

The needs have been allocated to the stakeholders for which they apply most, but it is clear that 

some of the needs allocated to one type of stakeholder may apply to the other stakeholders as 

well. For example, the collection of revenues and protection of financial interests (identified as a 

government needs in the intervention logic below) is also in the interest of the society as a whole, 

as collected revenues will eventually be used to finance infrastructure (roads, railways, etc.) for 

businesses  and  citizens.  Alternatively,  also  businesses  and  governments  are  part  of  and  are 

operating into a wider society, so naturally they have the same safety needs or law enforcement 

needs as citizens have.  

To enhance traceability, the strategic objectives of the Customs Union have been linked to each 

of these needs in between brackets.  

16 

 
 
Figure 2: Identified needs of governments, society and businesses vis-à-vis the Customs Union 

4.2.1  Needs of governments 

The  following  needs  of  governments  at  EU,  national  and  regional  level  vis-à-vis  the  Customs 

Union were identified:  

•  The collection of revenue by customs authorities ;  

•  The correct implementation of customs, trade policies and legislation at land, sea and in 

the air. Customs and trade policies and legislation could cover anti-dumping measures, 

embargos, etc. 

•  The correct implementation of (parts of) non-customs policies and legislation by customs 

authorities.  Non-customs  policies  and  legislation  for  example  include  health  and  safety 

legislation.  Although  customs  authorities  are  not  responsible  for  law  making  in  non-

17 

 
 
 
 
 
customs  fields,  they  have  an  important  role  in  the  enforcement  and  application  of  non-

customs legislation. 

•  To collect information on trade flows and on trade actors for policy-making purposes. 

Finally, it is clear that governments do not only want these needs to be met effectively, but also in 

an efficient way.  

4.2.2  Needs of society 

The identified needs of society are the following:  

•  Protection against terrorism and criminal organisations: society needs protection against 

all kinds of illegal activities, including money laundering. 

•  Protection  of  human  and  animal  health,  preventing  the  inflow  of  illegal  drugs,  unsafe 

products or foods, etc. 

•  Enforcement of respect for the rule of law;  

•  Protection of the interest of EU taxpayers: this concern overlaps with the specific needs 

of government to ensure proper revenue collection;  

Increase economic prosperity, for example through trade with other parties from outside 

the EU or outside Europe. 

Implementation  of  foreign  policy  of  the  EU,  for  example  the  enforcement  of  trade 

• 

• 

embargos, ensuring the non-proliferation of nuclear or chemical weapons, etc. 

•  Protection  of  cultural  heritage,  meaning  preventing  the  illegal  export  of  EU  cultural 

heritage goods;  

•  Protection of the environment.  

Also society wants that these needs are not only met in an effective way, but also in an efficient 

way.  

4.2.3  Needs of businesses 

Expectations of EU businesses vis-à-vis the Customs Union that were identified are:  

•  Protect the level-playing field for companies and goods in equal situations across the EU: 

in order to ensure fair competition, businesses wish to be treated on equal terms as their 

competitors in the same situations; 

18 

 
 
 
 
•  Ensure minimum necessary physical and procedural intrusion of trade flows, for example 

through the introduction of simplified procedures, single window, or one-stop shops; 

•  Support  the  competitiveness  of  EU  companies,  for  example  through  tariff  suspensions: 

this need is very much linked to the EU trade policies; 

•  Protect the intellectual property;  

•  Protect the free movement within the Single Market;  

•  Transparent, swift and uniform information on working procedures, customs master data 

(data relating to classification, valuation and origin), trade measures, standards & norms 

to be complied with (phyto-sanitary requirements), legislation and jurisprudence. 

Finally, also businesses want these needs not only to be met in an effective way, but also in an 

efficient way.  

4.3  Objectives 

The  strategic  objectives  for  the  Customs  Union  aim  to  tackle  the  identified  needs  of 

governments,  society  and  businesses.  The  previous  figure  on  the  needs  specifically  links  all 

identified needs to one of the strategic objectives of the Customs that are identified below.  

The  strategic  objectives  were  taken  from  the  EC  Communication  on  the  Strategy  for  the 

Evolution  of  the  Customs Union  and  have  been  considered  as  a  given  fact  by  the  study  team. 

These strategic objectives are:  

•  1. To protect the EU, specifically:  

o  1A. Protection of society:  

o  1B. Protection against crime and terrorism;  

o  1C. Protection of the financial interests of the EU and its Member States. 

•  2. To support the competitiveness of EU businesses, specifically:  

o  2A. Protection of the business environment in Europe;  

o  2B. Provision of quality customs service 

o  2C. Facilitation of trade 

19 

 
 
Figure 3: Identified strategic objectives 

Source: COM (2011)922, 21/12/2011 

4.4 

Inputs 

In order to realise these strategic objectives, a range of inputs is made available to the Customs 

Union. Without these inputs, the Customs Union would not be able to perform its activities. Three 

types of inputs were identified: 

•  Rules, covering:  

o 

International agreements such as the World Trade Organisation agreements and 

the  World  Customs  Organization  conventions  (e.g. 

the  Revised  Kyoto 

Convention);  

o  Customs legislation and policies, such as the Community Customs Code (CCC) 

and its consolidated Implementing Provisions (CCIP)3 as well as amendments to 

the CCC, in particular the Security and Safety Amendment to the Customs Code4 

providing  for  full  computerisation  of  all  operating  procedures  related  to  security 

and safety, and export; Common Customs Tariff (Combined Nomenclature5 and 

Tariff  Measures)  and  duty  relief  legislation6,  Special  legislation  dealing  with 

border  enforcement  and  customs  formalities  and  controls  on  specific  types  of 

goods:  goods  infringing  IPR,  drug  precursors,  cultural  goods,  cash;  residual 

national rules for the implementation of EU customs legislation; national rules on 

customs offences and penalties or individual customs decisions. 

3 The Community Customs Code Regulation 2913/92 (Community Customs Code) and Regulation  2454/93 

(Implementing Provisions)  

4 Regulation N° 648/2005 of 13 April 2005 - OJ L117, 4 May 2005. 

5 Council Regulation (EEC) No 2658/87 of 23 July 1987 on the tariff and statistical nomenclature and on the Common 

Customs Tariff 

6 Council Regulation (EU) No 1186/2009 

20 

 
 
                                                      
o  Non-customs  legislation  and  policies,  such  as  REACH,  EU  data  protection 

legislation and national legislation in non-customs fields; 

o 

International and EU customs and non-customs case law;  

•  Guidelines and working procedures, aiming to make customs and non-customs rules 

practical and concrete for trade actors and customs authorities. For example, guidelines 

on how to clear and release goods, on how to collect and store customs data, or on how 

to  conduct  risk-assessments.  These  guidelines  and  working  procedures  could  have  a 

national, EU or international character.  

•  Resources,  covering  the  budget  spent  on  the  functioning  of  the  Customs  Union  at 

national  and  EU  level,  the  input  of  human  resources  at  the  EU  and  national  customs 

authorities, ICT systems and ICT investments, as well as other forms of equipment (for 

example scanners).  

By means of labelling (red-orange-green), the study team aimed to visualise the extent to which 

DG  TAXUD  can  manage  and  influence  the  inputs  identified.  The  EU  level  (and  DG  TAXUD  in 

particular)  has  only  limited  impact  on  the  inputs  marked  in  red  (e.g.  international  agreements, 

specific national customs legislation, non-customs legislation, international case law). This does 

not mean that it is impossible to influence these aspects, but rather that significant efforts would 

be  required  to  change  these  elements  (e.g.  a  new  EU  legislative  proposal  to  replace  national 

legislation is a time and resource intensive effort, which cannot be realised in a short term). On 

the inputs in green the possibilities of DG TAXUD to make modifications are much higher, such 

as for example on the EU Customs guidelines and working procedures. With regard to the inputs 

marked  in  amber,  DG  TAXUD  has  a  certain  extent  of  influence,  but  many  other  actors  would 

need to be involved as well (e.g. any revision of REACH, etc).  

21 

 
Figure 4: Identified inputs 

4.5  Activities 

The identified inputs are used by the Customs Union to perform its various activities.7   

The figure below presents the core processes that are performed by the Customs Union, being 

three “vertical” core processes (clearance, control and enforcement) and three “horizontal” core 

processes  (economic  operator  management,  data  management  and  risk  management  and 

intelligence) which contribute to the vertical processes.  

7 For the purpose of this study, the identification of the activities performed by the Customs Union primarily took place by 

making use of the findings from the Deloitte study on the strengths and weaknesses of the Customs Unions operating 

model (DG TAXUD, Final Report on Task 2.2 – Strengths and weaknesses of the current organisation model, 2011). It 

is recognized that other possible approaches exist in this respect and that therefore both the activities identified and the 

taxonomy used could be debated. However, with the aim of having a stable basis that is required for the analysis to be 

conducted by the current study, it was agreed to use this representation.   

22 

 
 
 
                                                      
Figure 5: Activities of the Customs Union – core processes 

The  clearance  process  involves  all  the  formalities  of  dealing  with  import/export/transit/other 

declarations8. The control process refers to the application of various types of control, including 

controls related to the clearance process, and to safety and security.  

The clearance process includes:  

•  Processing of pre-arrival/departure notifications;   
•  Processing of import/export/transit/warehouse and other declarations; 
•  Applying of economic customs procedures (bonded warehousing, etc);  
•  Applying classification, tariff, valuation, origin rules;  
•  Applying prohibitions and restrictions;  
•  Collection  of  customs  duties  and  other  revenues  (such  as  excise  duties,  import  VAT, 

etc.); 

The control process includes:   

•  Examination of declarations and supporting documents ('documentary controls');  
•  Examination  of  goods  ('physical  controls',  including  x-raying,  scanning,  using  customs 

dogs etc); 

•  Taking samples for laboratory analysis or more detailed examination of goods; 
•  Post-release controls; 

8 The details of these processes will differ, but the main components are considered to be similar. 

23 

 
 
 
                                                      
Inspection of means of transport; 

•  Examination of accounts and other records of economic operators; 
• 
•  Laboratory analyses; 
• 

Inspection of baggage and other goods carried by or on persons (incl. cash controls); 

•  Carrying out official enquiries and other similar acts. 

The enforcement process includes: 

•  Conducting of criminal and civil investigations; 
•  Conducting of criminal and civil prosecution; 
• 

Imposing and collecting fines and other penalties. 

The three “horizontal” core processes contribute to each of these “vertical processes”.  

The economic operator (trader) management process includes: 

•  Registration of traders; 
•  Prior audit of economic operators; 
•  Taking  and  managing  customs  decisions  (incl.  handling  applications  for  and  granting 

authorisations,  like  for  AEO;  issuing  Binding  Tariff  Information  and  Binding  Origin 

Information,  granting  right  to  be  heard;  amending,  suspending,  revoking,  annulling, 

monitoring, and re-assessing decisions); 

•  Appeals against customs decisions; 
•  Guarantees; 
•  Provision of information and advice related to the processes mentioned above;  
•  Regular dialogue with economic operators; 
•  Development of expertise and relationship management; 
•  Treatment of companies’ failure to comply (administrative settlements, penalties, criminal 

charges, investigation, cross-border examinations etc). 

Risk management9 and intelligence process includes:  

•  Risk assessment; 
•  Risk analyses, targeting; 
•  System monitoring; 
•  Development of intelligence. 

Data management process includes: 

9http://ec.europa.eu/taxation_customs/customs/customs_controls/risk_management/customs_eu/ 

index_en.htm  

24 

 
 
 
                                                      
•  Collection of data: external and internal; 

•  Data storage;  

•  Dissemination of data: external and internal. 

To perform the core processes, a number of supporting processes are executed as well. They 

are:  

•  Planning and monitoring processes;  
•  Human capital processes;  
• 
ICT processes;  
•  Finance processes;  
•  Procurement processes;  

• 

Internal and external communication. 

Figure 6: Activities of the Customs Union – supporting processes 

25 

 
4.6  Outputs 

The  Customs  Union  has  been  created  to  generate  certain  outputs.  On  a  meta  level,  these 

outputs are:  

•  Clearances of goods provided by customs authorities (import/export/transit);  
•  Seizures,  confiscations,  re-exportation  and  goods  destroyed  by  customs  authorities, 

covering prohibited goods, restrictions, Intellectual Property Rights infringements; 

•  Collected  revenues  through  customs  authorities,  including  collected  customs  duties, 

other taxes and levies;  

•  Overall treatment of non-compliance, including enforcement; 
• 

Information provided to stakeholders.  

It is clear that through performing the various core and supporting processes, the Customs Union 

generates  many  additional  intermediate  outputs.  For  example,  the  risk  management  activities 

produce various different risk analyses. These risk analyses can be considered as intermediate 

outputs and not as final outputs as such.  

Figure 7: Identified outputs of the Customs Union 

4.7  Outcomes (effects) 

The  outputs  of  the  Customs  Union  are  intended  to  generate  certain  effects  or  outcomes.  The 

desired outcomes or the likely achieved short-term and medium-term effects of the outputs of the 

Customs Union are:  

• 

Illicit,  restricted  and  prohibited  goods  destined  for  the  EU  are  being  stopped  at  the  EU 

borders, within the EU or in third countries;  

•  Fraud and tax evasion is being detected;  
•  Customs duties, other taxes and levies are correctly and efficiently collected 
•  Customs  and  non-customs  legislation  and  working  procedures  are  correctly  and 

uniformly applied across the EU;  

•  Unnecessary  administrative  burden  on  businesses  for  trade-related  formalities  is  being 

reduced.  

26 

 
 
 
 
 
Figure 8: Intended outcomes of the Customs Union 

4.8 

Impacts  

The achievements of the outcomes identified above will logically lead to the following impacts on 

society, business and governments in the EU. Ideally, the intended impacts match with the needs 

that were initially identified (actually implying an improvement of the starting situation).  

The intended impacts of the Customs Union on governments include: 

•  Availability  of  public  financial  resources  from  revenue  collected  by  customs  authorities 

that  can  be  spent  by  governments  in  order  to  implement  economic,  social,  cultural  or 

environmental policies;  

•  Customs and non-customs policies and legislation are enforced and respected;  
•  … 

The intended impacts on society that were identified include: 

•  Enhanced protection of health of EU citizens or animals;  
•  Higher protection of the environment;  
•  Enhanced protection of cultural heritage;  
•  Higher protection of the EU against terrorism and organised crime;  
•  Enhanced effectiveness of EU foreign policies;  
• 
• 

Increase in economic prosperity in the EU; 

... 

The intended impacts on businesses that were identified include: 

•  Fair competition among businesses within the EU internal market;  
•  Fair competition at global level between EU businesses and non-EU businesses;  
• 
•  Facilitation of intra-EU and extra-EU trade, leading to increased economic prosperity;  
•  Free movement within a protected, single market; 

Increased protection of intellectual property;  

27 

 
 
 
 
 
 
 
• 

...  

Figure 9: Identified intended impacts on businesses, governments and society 

28 

 
 
 
5 Evaluation questions 

5.1  Overview of the evaluation questions considered 

Significant efforts were made to prioritise the many  possible evaluation questions, based on an 

assessment of the extent to which the answers on the questions could be used to inform decision 

makers.  

The following evaluation questions have been retained to assess in terms of feasibility: 

•  To  what  extent  are  the  strategic  objectives  of  the  Customs  Union  still  relevant  and 

sustainable for the next decade?  

•  To what extent are the core processes of the Customs Union uniformly  applied  across the 

EU  (clearance  processes,  enforcement,  control  processes,  data  management,  economic 

operator processes, risk management)?  

-  Where do the main areas of divergence emerge?  

-  What are the main reasons for differences in application of the processes or for 

non-uniformity?  

-  How  significant  are  the  consequences  of  the  differences  in  application  of  the 

processes (and for whom)? 

•  Are the core processes of the Customs Union being efficiently  performed? Are there great 

differences  across  the  EU  in  the  efficiency  (clearance  processes,  enforcement,  control 

processes, data management, economic operator processes and risk management)?  

•  What  have  been  the  effects/outcomes  of  the  major  recent  policy  initiatives  in  the  field  of 

Customs Union for the Customs Union and its stakeholders?  

-  What  have  been  the  effects  of  the  Security  Amendment  of  the  Community 

Customs Code (including AEO) for the Customs Union and its stakeholders? 

-  What  have  been  the  effects  of  the  simplified  procedures  and  other  related 

measures?  

29 

 
-  What  have  been  the  effects  of  the  e-Customs  Initiative  for  the  Customs  Union 

and its stakeholders? 

•  What is the  quality  of the  service  provided  by  the Customs Union to  its main stakeholders 

(involvement  of  stakeholders  in  decision-making,  reachable,  costs  of  service  provision, 

paperless environment, skills and competences of staff, treatment of AEO)? 

•  Has  the  Customs  Union  achieved  its  strategic  objectives?  Has  it  generated  its  intended 

effects?  To  what  extent  has  the  Customs  Union  in  particular  contributed  to  these  effects? 

Are there any unintended effects and how significant are they?  

-  With regard to detection of fraud and tax evasion? 

-  With regard to the detection of prohibited and illegal goods? 

-  With regard to the collection of customs duties and other taxes and levies? 

-  With  regard  to  the  reduction  of  administrative  burden  on  businesses  for 

formalities related to trade? 

-  With  regard  to  the  application  of  customs  legislation  and  working  procedures 

across the EU? 

•  How has the Customs Union absorbed the enlargement of the EU? 

5.2  Relationship between evaluation questions and evaluation criteria 

In the previous chapter the most commonly used evaluation criteria for policy  evaluations have 

been  presented.  Following  the  evaluation  questions  presented  above,  it  is  clear  that  for  the 

feasibility  assessment  the  following  set  evaluation  criteria  have  been  selected  for  further 

consideration10: 

•  Relevance;  
•  Effectiveness; 
•  Efficiency; 
•  Utility/impact; 
•  Uniformity 

10 Other possible evaluation criteria would have been: coherence, complementarity, sustainability, etc. Following the 

prioritization of the evaluation questions, these evaluation criteria will however not be considered for the feasibility 

study. 

30 

 
 
 
                                                      
Whereas the criterion of uniformity is not commonly used in evaluation studies, it is considered to 

constitute  a  major  corner  stone  of  the  Customs  Union  and  has  therefore  been  retained  as  an 

evaluation criterion for the feasibility study. 

Annex  5  provides  a  detailed  definition  of  each  of  these  evaluation  criteria,  as  applied  for  the 

feasibility  study.  In  addition,  Annex  6  provides  a  detailed  mapping  of  the  evaluation  questions 

retained for the feasibility study and the evaluation criteria used.  

5.3  Relationship between the evaluation questions and the intervention 

logic 

On the figure below the relationship of the evaluation questions and the components of the 

Customs Union’s intervention logic is presented in a graphical way.  

In particular, the evaluation questions deal with the following elements: 

•  The relationship between the needs of the stakeholders and the strategic objectives; 
•  The relationship between the inputs on the one hand and the outputs and outcomes on 

the other hand; 

•  The way in which the activities of the Customs Union are performed; 
•  The outcome of the Customs Union (overall and for certain specific policy initiatives) 

Figure 10: Relationship between evaluation questions and the intervention logic 

31 

 
 
 
 
 
6 

Analytical framework for the 
evaluation 

6.1 

Introduction 

The analytical framework established defines for each evaluation question of the feasibility study 

the following elements:  

• 

the judgement criteria to allow the evaluator to formulate a judgment and cover all of the 

relevant ground relating to each evaluation question; 

•  quantitative and qualitative indicators that will be used to feed the judgment; 

32 

 
•  a  suggestion  of  the  information  sources  to  be  used  to  collect  relevant  qualitative  and 

quantitative data;  

In  some  cases  also  sub-evaluation  questions  and  sub-judgement  criteria  have  been  defined, 

given the extensiveness of the scope of some of the evaluation questions.    

This  chapter  contains  the  main  conclusions  of  the  exercise  of  establishing  the  analytical 

framework,  dealing  with  the  extent  to  which  it  was  possible  to  identify  judgement  criteria  and 

indicators  for  all  aspects  of  the  evaluation  questions,  as  well  as  the  main  data  collection  tools 

suggested to use for the evaluation. 

6.2  Judgement criteria and indicators 

To define possible judgement criteria and indicators the consultants have made extensive use of 

desk  research  and  suggested  input  from  DG  TAXUD  officials,  officials  of  other  Commission 

Directorate Generals as well as participants from trade organisations and academic institutions. 

The result is presented in a separate Excel file. 

6.2.1  Judgement criteria 

In  terms  of  identification  of  judgement  criteria  for  the  evaluation  questions,  the  feasibility 

assessment has lead to the following specific conclusions: 

• 

It  is  feasible  to  translate  all  evaluation  questions  into  judgement  criteria,  with  the 

exception  of  evaluation  question  1,  dealing  with  the  sustainability  of  the  strategic 

objectives of the Customs Union for the next decade. As it is nevertheless important to 

be  continuously  aware  of  the  impact  any  changes  in  the  Customs  Union  environment 

might  have  on  the  relevance  of  the  Customs  Union’s  objectives,  it  is  recommended  to 

include  indicators  on  this  aspect  as  well  into  regular  monitoring.  Annex  9  provides 

suggestions for indicators to monitor the evolution of some key drivers for change of the 

Customs Union. 

•  Given  the  complex  nature  of  the  Customs  Union,  answering  the  other  evaluation 

questions in their full extent involves many different aspects. This has given way to the 

identification of a high number of (sub-) judgement criteria. This is in particular the case 

for the evaluation questions dealing with the uniformity and the efficiency of the Customs 

Union’s  core  processes,  as  these  processes  entail  many  different  aspects.  In  total,  the 

analytical framework contains more than 170 (sub-) judgement criteria; 

•  For  evaluation  question  7  -  dealing  with  the  how  the  Customs  Union  has  absorbed  the 

enlargement of the EU- it is suggested to use as judgement criteria the extent to which 

there  are  differences  in  the  findings  for  each  of  the  other  evaluation  questions  for  the 

EU15  versus  those  Member  States  that  have  joined  the  EU  since  2004.  Due  care  is 

33 

 
however  needed  when  interpreting  potential  differences,  as  a  number  of  these  new 

Member States cover a significant share of the EU’s land border with third countries. 

As a general conclusion in terms of feasibility of the (sub-) judgement criteria, it is however clear 

that trying to assess all (sub-) judgement criteria in one single evaluation study is highly advised 

against.  Therefore  a  prioritisation  of  (sub-)  judgement  criteria  needs  to  take  place.  To  facilitate 

the discussion, the consultants have prepared a first prioritisation of the (sub-) judgement criteria 

based on the following criteria:  

•  The relevance of the aspect in question to be considered  when making an assessment 

on the evaluation question;  

•  Extent  to  which  the  aspect  is  considered  by  various  stakeholders  to  hinder  the  current 

functioning of the Customs Union; 

As a result of this expert assessment, each (sub-) judgment criterion has received a priority from 

1 (recommended to be considered as a first priority for the evaluation) to 3 (least recommended). 

The  table  below  provides  an  overview  of  the  distribution  of  the  priority  levels  for  each  of  the 

evaluation question (for example: for Question 2, 32 judgement criteria are suggested as priority 

1; 7 judgement criteria are recommended as priority 2 and 27 judgement criteria are suggested 

as priority 3): 

Table 1: Mapping of judgement criteria priorities per evaluation question 

Priority judgment criteria/ evaluation question
Priority 1
Priority 2
Priority 3

Q2

Q3

Q4

Q5

Q6

Q7

Total

32
7
27
66

22
15
44
81

9
2
0
11

4
5
3
12

6
5
6
17

0

73
34
80
187  

6.2.2  Indicators 

In terms of identification of indicators per (sub-) judgement criteria, the feasibility assessment 

has lead to the following specific conclusions: 

• 

It  is  feasible  to  translate  each  (sub-)  judgment  criterion  into  several  quantitative  and/or 

qualitative indicators. Different types of information –if collected- will thus be available to 

the  evaluator  for  analysis.  As  such,  none  of  the  (sub-)  judgement  criteria  is  thus 

unfeasible to assess. 

34 

 
 
 
 
 
 
 
 
•  For  evaluation  question  7  -dealing  with  the  how  the  Customs  Union  has  absorbed  the 

enlargement of the EU- it has been suggested to use as judgement criteria the extent to 

which there are differences in the findings for each of the other evaluation questions for 

the  EU15  versus  those  Member  States  that  have  joined  the  EU  since  2004.  For  this 

reason  this  evaluation  question  will  not  require  separate  indicators  to  be  collected.  As 

mentioned above, due care is however required when interpreting the data collected due 

to  the  presence  of  land  borders  with  non-EU  countries  for  a  significant  number  of 

Member States that have joined the EU since 2004. 

•  The number of indicators is in direct relation with the number of (sub-) judgement criteria. 

As a high number of (sub-) judgement criteria has been identified, it should therefore not 

come  as  a  surprise  that  also  a  very  high  number  of  possible  indicators  has  been 

identified.  This  is  in  particular  the  case  for  the  evaluation  questions  dealing  with  the 

uniformity and the efficiency of the Customs Union’s core processes, as these processes 

entail  many  different  aspects11.  In  total,  more  than  570  different  indicators  have  been 

identified.  

•  Whereas the reporting of the data should happen at the level of the Customs Union as a 

whole in the evaluation report, it is highly recommended to collect the data per Member 

State,  as  this  will  allow  for  more  nuanced  analysis  and  conclusions.  It  will  also  allow 

answering the evaluation question on the differences in findings between the EU15 and 

the Member States that joined the EU since 2004. 

•  All of the indicators identified have been found to be feasible to collect as such – even for 

the evaluation planned in 2012. The only exception on this point may be the possibility to 

collect  information  for  certain  aspects  of  the  effectiveness  of  the  Customs  Union.  In 

concrete  terms,  this  deals  with  indicators  such  as  the  estimated  volumes  of  illicit, 

prohibited  etc  goods.  For  the  (sub-)  judgement  criteria  for  which  these  indicators  were 

identified, it is suggested to use  a pragmatic approach;  where possible the comparison 

between  the  actual  volumes  of  goods  seized  by  customs  should  be  compared  with  the 

total  estimated  volume  of  these  goods  entering  the  EU  based  on  official  estimations  of 

international organisations. In the case such information is however not available, a more 

qualitative  assessment  of  the  effectiveness  of  the  Customs  Union  can  be  made  –  for 

example based on in-depth interviews and feedback collected through surveys. 

As a general conclusion in terms of feasibility of the indicators, it is however clear that trying to 

collect data for each indicator in one single evaluation study is virtually impossible. Even for the 

“priority 1” (sub-) judgment indicators attempting to collect data for all indicators (being more than 

400) is deemed not feasible. DG TAXUD will therefore need to further limit the (sub-) evaluation 

questions and/or (sub-) judgement criteria that need to be considered for the evaluation study. 

11 It should be noted that some indicators are identical for both evaluation questions, be it that the way in which they need 

to be analysed will be different.  

35 

 
 
   
                                                      
To ensure sufficient basis for analysis to come to a conclusion on a (sub-) judgment criterion it is 

however  not  recommended  to  prioritise  the  indicators  per  (sub-)  judgment  criterion.  In  other 

words,  it  is  recommended  to  focus  the  prioritisation  exercise  at  the  level  of  (sub-)  evaluation 

questions and (sub-) judgement criteria.   

6.3  Data sources 

6.3.1  Approach followed to define the data sources per indicator 

The  data  sources  per  indicator  have  been  defined  based  on  the  concept  of  triangulation. 

Triangulation  is  a  technique  that  facilitates  validation  of  data  through  cross  verification  from 

multiple  data  sources  instead  of  one  single  source.  It  is  based  on  the  application  and 

combination  of several research methodologies in the study  of one subject (or  of an evaluation 

question  in  this  context),  in  order  to  support  findings  with  the  help  of  other  sources  (validation) 

and  to  complement  data  with  new  results  and  additional  pieces  of  the  overall  puzzle 

(completeness). 

Triangulation  will  increase  the  robustness  of  the  analyses  that  are  made  in  this  study.  By 

combining  multiple  observers,  methods,  and  data,  the  weakness  or  intrinsic  biases  and  the 

problems  that  come  from  one  single  method,  single-observer  and  single-theory  studies  can  be 

overcome. 

Denzin (1978) identified four basic types of triangulation:  

•  Data  triangulation:  various  data  sources  are  used  with  regard  to  time,  space,  and 

persons. For example, questionnaires could be applied at different points of time; 

• 

Investigator triangulation: involvement of multiple researchers with multiple backgrounds 

in an study; 

•  Theory 

triangulation: 

involves  using  more 

than  one 

theoretical  scheme 

in 

the 

interpretation of the phenomenon 

•  Methodological triangulation: involves using more than one method to gather data, such 

as interviews, observations, questionnaires, and documents. One can distinguish within-

method triangulation (a combination of approaches from the same type of data collection 

method)  and  between-method  triangulation  (combination  of  approaches  from  both 

quantitative and qualitative data collection methods). 

For this study, data triangulation and methodological triangulation are recommended. 

6.3.2  Suggested data sources for the evaluation 

36 

 
 
 
For  the  (sub-)  judgement  criteria  with  priority  1  the  study  team  has  already  defined  suggested 

data  sources  for  each  indicator.  For  the  evaluation  the  following  data  collection  methods  are 

proposed:  

•  Desk research of existing data; 
•  Collection of additional data from the Member States; 
•  Different web-based surveys targeting  

o  Businesses (EU and national level);  

o  customs authorities (DG TAXUD officials and national customs authorities);  

o  non-customs authorities; 

• 

In-depth interviews with:  

o  customs authorities (EU and national level); 

o  businesses (EU and national level); 

•  Case studies. 

Each of these data sources can be used for the evaluation to be performed in 2012, as well as 

any  subsequent  evaluation  work  at  a  later  stage.  Below  each  of  these  data  collection  methods 

are presented into further detail, together with an indication of information to be collected through 

the data source and points for attention.  

The  actual  information  to  be  collected  through  each  data  source  will  depend  on  the  further 

prioritisation  of  (sub-)  evaluation  questions  and  (sub-)  judgement  criteria  by  DG  TAXUD.  As 

stated above, without such further prioritisation the amount of information to be collected will not 

be  feasible.  Annex  7  provides  the  long  list  of  indicators  to  be  collected  per  data  source  for  the 

“priority  1”  judgement  criteria.  The  table  below  provides  an  quantitative  overview  of  how  many 

indicators  are  considered  for  the  “priority  1”  judgement  criteria  per  data  source  per  evaluation 

question. It  is clear that this list of indicators  will need to  be further reduced. In the  absence of 

more  prioritisation,  for  each  data  type  the  main  evaluation  criteria  for  which  data  need  to  be 

collected are presented. 

37 

 
 
 
Table 2: Mapping of number of indicators per data source per evaluation question (priority 1) 

Data source / evaluation question
Numbers of indicators for which data will be requested 
at central MS level
Numbers of indicators for which data will be collected 
through desk research performed by the evaluator
Number of indicators for which data will be collected 
through interviews with businesses
Number of indicators for which data will be collected 
through interviews with customs authorities
Numbers of indicators for which data will be collected 
by survey among businesses
Numbers of indicators for which data will be collected 
by survey among customs authorities
Numbers of indicators for which data will be collected 
by survey among non-customs authorities
Numbers of indicators for which data will be collected 
through case studies

Q2

16

5

7

7

24

49

0

Q3

49

7

1

14

7

18

0

23
131

33
129

Q4

15

2

2

2

18

24

8

0
71

Q5

7

1

0

0

3

4

0

Q6

21

8

1

2

8

6

1

3
18

19
66

Q7

Total

108

23

11

25

60

101

9

78
415

Every  type  of  data  source  has  potential  risks  and  shortcomings.  To  enhance  the  robustness  of 

data collected from each source, it is therefore recommended to take into account the attention 

points per data source presented below.   

Desk research 

Desk research by the evaluators deals with the review of available literature, as well as analysis 

of available quantitative data and databases. 

Review of available literature  

Desk research is considered a source of primary importance for this evaluation. It consists of an 

extensive  literature  review  aimed  at  understanding  the  main  points  of  the  important  internal 

documents  and  external  sources  in  order  to  set  the  “playing  field”  in  which  the  evaluation  is 

carried out. 

Extensive  review  of  available  literature  and  legislation  has  already  been  conducted  during  this 

pre-study.  The  evaluator  will  be  able  to  benefit  to  a  large  extent  from  the  work  done. 

Nevertheless,  it  is  recommended  that  the  evaluator  performs  an  in-depth  reading  of  a  set  of 

documents  at  the  start  of  the  evaluation.  The  list  of  recommended  literature  can  be  found  in 

annex 2 of this report.  

Analysis of available quantitative data  

The evaluation of the current state of the Customs Union should be evidence-based. A significant 

part  of  this  evidence  will  come  from  the  analysis  of  available  quantitative  data.  This  deals  with 

38 

 
 
 
 
 
publicly available data (e.g. annual reports of DG BUDGET, World Bank indicators etc) but also 

with information that is reported to the EU level which may not be publicly available (e.g. the data 

of the Measurement of Results).  

It  was  found  that  –whereas  existing  data  may  be  very  relevant  and  important  to  analyse-  the 

number  of  indicators  publicly  available  on  the  performance  of  the  Customs  Union  is  relatively 

limited. The available quantitative data will be required in particular with regard to the evaluation 

criteria uniformity, efficiency and for the overall effectiveness of the Customs Union.  

The main point for attention for this type of data source is that the evaluator must be cautious not 

to interpret the figures wrongly (i.e. be clear about the definition of the indicator).  

Collection of additional data from the Member States 

Although some relevant data, figures and numbers that feed the indicators that were selected to 

answer the evaluation questions is already available, the evaluator will need to collect additional 

data in order to be able to properly answer all evaluation questions.  

It  is  therefore  suggested  that  the  evaluator  collects  further  quantitative  data  from  the  customs 

authorities in the Member States. The type of information to be collected through this way should 

normally  be  feasible  to  be  provided  by  the  central  customs  authorities’  administration  in  each 

Member State.  

This data source is relevant to collect information for all evaluation questions, but it will be most 

important for the evaluation questions on efficiency, overall effectiveness, uniformity and quality 

of service of the Customs Union.  

The following attention points are suggested for this data source:  

•  Experience indicates that the number of indicators on which one can collect information 

in  this  way  is  limited.  If  Member  States  are  requested  to  provide  quantitative  data  on 

many  indicators,  the  chance  of  not  receiving  any  input  becomes  considerable.  It  is 

therefore suggested to restrict the number of indicators to be collected in this way to an 

absolute minimum; 

•  When  asking  the  Member  States  to  provide  quantitative  data,  clear  definitions  are 

required so as to avoid any differences in interpretation of what is meant by the indicator 

to be reported upon. It is suggested that the evaluation team provides clear definitions of 

the data collected through this channel; 

•  To  facilitate  the  reporting  process,  it  is  also  recommended  that  each  Member  States 

identifies  a  single  point  of  contact  who  is  responsible  for  the  data  reporting.  Also  the 

39 

 
 
evaluation team is recommended to establish a central contact person who co-ordinates 

the data collection and who can provide clarifications on the indicators when necessary. 

Web-based survey 

To  collect  feedback  from  a  large  group  of  Customs  Union  stakeholders,  three  different  web-

based surveys for data collection are proposed:  

•  A web-based survey targeting businesses;  
•  A web-based survey targeting customs authorities (EU and national);  
•  A web-based survey targeting non-customs authorities (EU and national); 

A web-based survey format is suggested, because of the significant advantages compared to a 

classic  paper-based  form  or  word-document  sent  per-email.  Web-based  surveys  have  the 

following advantages:   

•  Swift  set-up  and  operationalisation.  From  experience,  the  bulk  of  the  workload  is 

agreeing on a solid list of questions. 

•  Cost-efficient  instrument  which  enables  to  reach  a  very  large  target  group  (up  to  5,000 

respondents). 

•  Respondents  can  quickly  complete  and  easily  submit  the  questionnaire,  with  no 

administrative burden;  

•  Protection  of  data  is  guaranteed  whilst  manipulation  of  data  is  very  difficult  (through  a 

cookie system, respondents can only answer once);  

•  Reminders can be sent easily through e-mail;  
•  Results  can  be  consulted  online,  for  those  who  have  access  to  the  database,  and  one 

can keep track of the status and the response rate.  

•  More  sophisticated  online  surveys  enable  different 

language  versions  of 

the 

questionnaire,  as  well  as  differentiation  of  the  questions  depending  on  previously 

provided answers by the respondent. Both functionalities may be highly useful to reach 

all national customs authorities. 

The  web  surveys  targeting  businesses  and  (EU  and  national)  customs  authorities  will  be 

particularly useful to collect information regarding the uniformity of the Customs Union, as well as 

for the question on the quality of services of the customs authorities. 

The information to be collected through the web survey targeting non-customs authorities is more 

limited and mainly deals with the appreciation of the quality of services of customs authorities to 

non-customs authorities.  

40 

 
 
 
 
When setting up the  web-based surveys, the following points for  attention should be taken into 

account:  

•  Risk of low response rate. Strong stakeholders involvement is needed that can promote 

the  survey  and  raise  awareness.  This  is  particularly  needed  to  ensure  a  good 

geographical coverage of the respondents. 

•  Whereas a link to the survey can be provided on a web-site, having e-mail addresses of 

possible  respondents  significantly  enhances  the  response  rate.  These  could  potentially 

be  obtained  through  business  representation  bodies  or  existing  networks.  Their 

cooperation is crucial. 

•  The phrasing of survey questions is of critical importance, as misinterpretations and loss 

of  nuance  can  often  lead  to  erroneous  results.  This  is  a  far  greater  challenge  than  the 

technical  implementation of an online tool. The evaluators  will  have  to,  where  possible, 

use domain knowledge to adjust the language and question phrasing and thereby reduce 

the possibility of confusion, bias or misunderstanding in the response. 

•  The number of questions asked in an online survey should be limited. Ideally answering 

a  web  survey  should  not  take  longer  than  10  minutes  of  a  respondent’s  time.  If 

answering  the  survey  takes  too  long,  the  risk  of  respondents  abandoning  the  survey 

increases significantly; 

•  This  data  collection  instrument  is  not  suitable  if  one  wants  to  obtain  very  detailed 

information. Furthermore, the evaluator does not have the opportunity to ask for further 

details, if a particular answer is not clear. It is therefore recommended to complement the 

information with findings from in-depth interviews and other data sources. 

Annex 8 provides further information on the various steps that are recommend when setting up a 

survey, so that the risks described above could be mitigated. 

In-depth interviews 

The aim of in-depth interviews is to obtain qualitative feedback from the stakeholders related to 

the  performance  of  the  Customs  Union.  This  qualitative  feedback  will  complement  quantitative 

information  gathered  by  the  evaluators.  In  particular  it  will  help  understanding  the  “reasons 

behind” some of the findings from the quantitative and semi-quantitative (i.e. web surveys) data. 

In-depth  interviews  are  recommended  to  be  performed  in  a  semi-structured  way,  based  on  a 

questionnaire  with  open  questions  and  allowing  the  interviewee  to  go  beyond  the  questions  if 

relevant for the evaluation.  

To collect information for the evaluation, in-depth  interviews with the following target audiences 

are proposed:  

•  Customs authorities (DG TAXUD officials and national customs authorities);  

41 

 
 
•  Businesses 

involved 

in 

import/export/transit  activities,  with  a  good  balance  of 

multinational companies and SME, as well as with activities related to various domains.  

Should  it  not  be  possible  to  have  an  in-depth  interview  on  a  face-to-face  basis,  telephone 

interviewing may be considered as well. Telephone interviewing is an appropriate data collection 

tool especially for specific targets, for instance opinion leaders etc., whose professional activities 

leave them little time to participate in a face-to-face interview, or for a representative sample of 

decentralised  target  audiences  where  a  relatively  short  period  of  time  is  required  to  obtain 

answers to key questions. 

In-depth  interviews  with  customs  authorities  (EU  and  national)  will  be  particularly  useful  for  the 

evaluation  questions  on  the  efficiency  and  the  uniformity  of  the  Customs  Union.  In-depth 

interviews  with  businesses  will  be  particularly  relevant  to  gather  further  information  on  the 

uniformity of the Customs Union. 

When  setting  up  in-depth  interviews,  the  following  points  for  attention  should  be  taken  into 

account:  

•  This  data  collection  instrument  is  not  suitable  if  one  wants  to  obtain  very  detailed 

quantitative information.  

•  The  number  of  questions  asked  in  an  online  survey  should  be  limited.  Ideally  an 

interview should not take longer than 90 minutes of a respondent’s time. 

•  Face-to-face interviews incorporate a distance problem: it is relatively costly to plan face-

to-face interviews with a high number of people.  

Case studies 

The  term  “case  study”  refers  to  the  collection  and  presentation  of  detailed  information  about  a 

particular  participant  or  small  group.  The  case  study  looks  intensely  at  an  individual  or  small 

participant pool, drawing conclusions only about that participant or group and only in that specific 

context.  

Common strengths of case studies are: 

•  Flexibility.  The  case  study  approach  is  a  comparatively  flexible  method  of  scientific 

research.  Because  its  project  designs  seem  to  emphasize  exploration  rather  than 

prescription  or  prediction,  researchers  are  comparatively  freer  to  discover  and  address 

issues as they arise in their experiments. In addition, the looser formats of case studies 

allows  researchers  to  begin  with  broad  questions  and  narrow  their  focus  as  their 

experiment progresses rather than attempt to predict every possible outcome before the 

experiment is conducted. 

42 

 
 
•  Attention  for  Context.  By  seeking  to  understand  as  much  as  possible  about  a  single 

subject  or  small  group  of  subjects,  case  studies  specialize  in  "deep  data,"  or  "thick 

description"--information  based  on  particular  contexts  that  can  give  research  results  a 

more concrete face. This attention for context can help bridge the gap between abstract 

research  and  concrete  practice  by  allowing  researchers  to  compare  their  firsthand 

observations with the more generic results obtained through other methods of research. 

Case  studies  are  recommended  as  an  approach  for  the  evaluation,  in  particular  to  collect 

information on a more detailed level for a few Member States. Whereas it would not be feasible 

to  collect  this  information  for  all  Member  States,  this  quantitative  and  qualitative  information 

would provide valuable insight on specific aspects of the functioning of the Customs Union – in 

particular in the domains of efficiency, uniformity and overall effectiveness of the Customs Union.  

Points for attention of case studies are: 

• 

Inherent subjectivity. Opponents cite opportunities for subjectivity in the implementation, 

presentation,  and  evaluation  of  case  study  research.  Results  may  not  be  generalised, 

are difficult to test for validity, and rarely offer a problem-solving prescription. Simply put, 

relying on one  or a few subjects as a  basis for cognitive extrapolations runs the risk of 

inferring too much from what might be circumstance. 

•  The number of indicators that can be collected through case studies are limited. The 

success of a case study lies in the willingness of Member States to co-operate. Further 

prioritisation will therefore be needed to reduce the information to be collected in this 

way.  

•  High investment. Case studies can involve learning more about the subjects being tested 

than most researchers would care to know. Because of its emphasis on "deep data," the 

case study is out of reach for many large-scale research projects which look at a subject 

pool in the tens of thousands. 

43 

 
 
 
 
 
 
7 Proposed budget and timeframe  

7.1 

Introduction 

This chapter presents an estimation of the budget and timeframe for the evaluation of the current 

state of the Customs Union. As explained in the previous chapter, significant further prioritisation 

will  be  required  to  limit  the  scope  of  aspects  to  be  evaluated  as  collecting  all  the  “priority  1” 

information will not be feasible. The budget needed for the actual evaluation will depend on the 

outcome of the prioritisation – information which is not yet available at this stage. 

Based  on  our  experience  the  following  aspects  are  important  drivers  of  the  budget  of  an 

evaluation study: 

•  The number of Member States from which data is to be collected; 

•  The number of face-to-face interviews to be performed; 

•  The number of web-based surveys to be established; 

•  The number of case studies to be performed; 

•  The volume of information that needs to be collected; 

•  The travelling required to collect the information. 

Three scenarios for the  budget  estimation  of the  evaluation  have  been developed,  with  varying 

levels of data collection activities. The table below provides further details about the differences 

between  the  scenarios.  It  should  be  emphasised  that  for  each  of  the  three  scenarios  it  is 

assumed that there is a significant reduction in the number of indicators to be considered than is 

currently the case for the “priority 1” judgement criteria.  

44 

 
 
Table 3: Differences between scenarios for the budget estimation 

Aspect
The number of Member States from which data is to be collected 
through desk research

The number of Member States in which in-depth interviews will 
take place

The number of in-depth interviews in the Member States
    Customs authorities
    Businesses
The number of in-depth interviews at EU level
    EU officials
    EU level business associations
The number of web-based surveys to be established

The number of case studies to be performed

Further explanation

Scenario 1

Scenario 2

Scenario 3

Impact on travel costs & 
impact on interviews in 
Member States

Per Member State
Per Member State

-To businesses
-To customs authorities
-To non-customs authorities

Each case study deals with 
10-20 indicators

15

10

3
1
2
8
4
4

3

1

27

12

6
2
4
14
8
6

3

2

27

14

7
3
4
18
10
8

3

4

45 

 
 
7.2  Estimation of mandays, budget and timeframe 

The next table provides the result of the estimation of the number of workdays required for each scenario.   

46 

 
 
Table 4: Estimation of man days per scenario 

DG Taxud - Evaluation of the current state of the Customs Union 
Type of Activity

Minimum scenario Medium scenario Full-option scenario

Further comments

Planned days

Planned days

Planned days

1.

1.1
1.2
1.3
1.4
1.5

2.
2.1

Phase 1 - Inception (miminal thanks to pre-study)

Kick-off & start-up
Initial interview with Commission officials
Literature review 
Fine-tuning of data collection and analysis methodology
Establishment of Inception Report
Subtotal phase 1 days
Phase 2 - Data collection, analysis and interim report
Preparation of data collection

2.1.1 Preparation of the data collection through Member States (request for information)
2.1.2 Preparation of the in-depth interviews at EU level
2.1.3 Preparation of the in-depth interviews at MS level
2.1.4 Elaboration of 3 survey questionnaires + operationalisation of the 3 surveys 
2.1.5 Preparation of the case study design 

2.2

Data collection

2.2.1 Desk research by the evaluators
2.2.2 Support to data collection from the Member States
2.2.3 In-depth interviews at EU level
2.2.4 In-depth interviews at Member State level
2.2.6 Data collection for case studies
Data analysis and interim report

2.3

2.3.1 First Analysis and synthesis
2.3.2 Draft Interim report
2.3.3 Interim meeting

3.
3.1
3.2
3.3
3.4

4.
4.1.

Subtotal phase 2 days
Phase 3 - Summarising and concluding
Further data processing
Draft final report
Draft final report meeting
Final report
Subtotal phase 3 days
Project management
Project management
Subtotal project management days

Total days

1,0
1,0
3,0
2,0
13,0
20,0

5,0
1,0
1,0
11,0
2,0

3,0
7,5
4,0
15,0
2,0

14,0
12,5
1,5
79,5

17,0
12,0
1,5
7,0
37,5

13,7
13,7
150,7

1,0
1,0
3,0
2,0
13,0
20,0

5,0
2,0
2,0
11,0
4,0

3,0
13,5
7,0
36,0
4,0

20,0
12,5
1,5
121,5

23,0
12,0
1,5
7,0
43,5

18,5
18,5
203,5

1,0 Familiarise with the context and understanding of the study
1,0
3,0 Familiarise with the subject
2,0 Familiarise with the analytical framework
13,0 Draft inception report & final inception report
20,0

5,0 Establish template, definitions of indicators, etc 
2,0 Establishment of final questionnaire, sending out invitation for interview, etc
2,0 Establishment of final questionnaire, sending out invitation for interview, etc

11,0 Collect e-mail addresses of invitees, operationalise web survey tool

8,0

3,0 To collect quantitative data (e.g. MoR, etc)
13,5 Send requests for data, follow up and Q&A

9,0 Interview and reporting
49,0 Interview and reporting

8,0 Data collection and reporting

26,0
12,5
1,5
150,5

29,0
12,0
1,5
7,0
49,5

22,0
22,0
242,0

47 

 
 
 
The budget required for the evaluation depends on the cost per man day. At an average cost of 

€1,000/day, the total cost of man days would thus be situated between €150,000 (scenario 1) 
and €242,000 (scenario 3).  

This  budget  should  be  augmented  with  a  travel  budget.  Based  on  missions  of  on  average  2 
days/Member  State  the  travel  budget  is  estimated  to  range  between  €25,000  (scenario  1)  and 
€35,000 (scenario 3). 

Finally,  if  it  is  decided  to  make  the  online  survey  available  in  several  languages,  translation 

costs will need to be added to the budget as well. 

The  following  indicative  timeline  would  be  suggested  for  the  2012  evaluation  of  the  current 

state of the Customs Union, assuming that the project could be launched mid May 2012. 

Table 5: Suggested milestones for the 2012 evaluation 

Milestone
Kick off
Inception phase 
Data collection and analysis phase Mid June 2012 - Sept 2012
Summarising and conclusion phase Oct 2012 - Mid Nov 2012

Indicative timing
Mid May 2012
Mid May 2012 - Mid June 2012

48 

 
 
 
 
 
 
8 Conclusions  

This feasibility study has as objective the definition of the methodology to perform the evaluation of 

the current state of the Customs Union. Whereas the Customs Union entails many more aspects 

than  a  single  policy  intervention,  the  approach  followed  for  the  study  has  demonstrated  that  the 

Commission’s  evaluation  standards  can  be  applied  for  the  evaluation  of  the  Customs  Union  as 

well. 

Thus, an intervention logic has been established at the offset of the project to create a common 

understanding  of  the  main  rationale  behind  the  Customs  Union  as  it  stands  today.  The 

intervention  logic  presents  the  needs  of  the  stakeholders,  the  strategic  objectives,  inputs, 

activities, outputs, effects (outcomes) and impacts of the Customs Union. The fact that common 

agreement could be reached on the intervention logic of the Customs Union may be considered 

as an achievement in itself of this project. 

Significant efforts were then made to prioritise the many possible evaluation questions, based on 

an  assessment  of  the  extent  to  which  the  answers  on  the  questions  could  be  used  to  inform 

decision  makers.  The  following  evaluation  questions  have  been  retained  to  assess  in  terms  of 

feasibility: 

•  To  what  extent  are  the  strategic  objectives  of  the  Customs  Union  still  relevant  and 

sustainable for the next decade?  

•  To what extent are the core processes of the Customs Union uniformly  applied  across the 

EU  Member  States 

(clearance  processes,  enforcement,  control  processes,  data 

management, economic operator processes, risk management)?  

•  Are the core processes of the Customs Union being efficiently  performed? Are there great 

differences  across  Member  States  in  the  efficiency  (clearance  processes,  enforcement, 

control processes, data management, economic operator processes, risk management)?  

•  What  have  been  the  effects/outcomes  of  the  major  recent  policy  initiatives  in  the  field  of 

Customs Union for the Customs Union and its stakeholders?  

•  What is the  quality  of the  service  provided  by  the Customs Union to  its main stakeholders 

(involvement  of  stakeholders  in  decision-making,  reachable,  costs  of  service  provision, 

paperless environment, skills and competences of staff, treatment of AEO)? 

49 

 
 
•  Has  the  Customs  Union  achieved  its  strategic  objectives?  Has  it  generated  its  intended 

effects?  To  what  extent  has  the  Customs  Union  in  particular  contributed  to  these  effects? 

Are there any unintended effects and how significant are they?  

•  How has the Customs Union absorbed the enlargement of the EU? 

When comparing these evaluation questions with the aspects identified in the intervention logic, it 

appears that in particular the objectives, inputs, activities and effects (outcomes) of the Customs 

Union will be subject to the evaluation. The following evaluation criteria have been deemed to be 

most relevant when assessing the performance of the Customs Union: relevance, effectiveness, 

efficiency, utility/impact and uniformity.  

In  terms  of  identification  of  judgement  criteria  for  the  evaluation  questions,  the  feasibility 

assessment has lead to the conclusion that it is feasible to translate all evaluation questions into 

judgement criteria, with the exception of evaluation question 1, dealing with the sustainability of 

the strategic objectives of the Customs Union for the next decade. As it is nevertheless important 

to  be  continuously  aware  of  the  impact  any  changes  in  the  Customs  Union  environment  might 

have  on  the  relevance  of  the  Customs  Union’s  objectives,  it  is  recommended  to  include 

indicators on this aspect as well into regular monitoring. It is however clear that trying to assess 

all  (sub-)  judgement  criteria  in  one  single  evaluation  study  is  not  feasible.  Therefore  a  further 

prioritisation of (sub-) judgement criteria will need to take place, focussing on those aspects that 

DG TAXUD deems most relevant to be evaluated.  

In terms of identification of indicators per (sub-) judgement criteria, the feasibility assessment has 

lead  to  the  conclusion  that  it  is  feasible  to  translate  each  (sub-)  judgment  criterion  into  several 

quantitative and/or qualitative indicators. Different types of information –if collected- will thus be 

available  to  the  evaluator  for  analysis.  The  number  of  indicators  is  in  direct  relation  with  the 

number  of  (sub-)  judgement  criteria.  As  a  high  number  of  (sub-)  judgement  criteria  has  been 

identified,  it  should  therefore  not  come  as  a  surprise  that  also  a  very  high  number  of  possible 

indicators has been identified. This is in particular the case for the evaluation questions dealing 

with the uniformity and the efficiency of the Customs Union’s core processes, as these processes 

entail many different aspects12. In total, more than 570 different indicators have been identified. 

Trying  to  collect  data  for  each  of  these  indicators  in  one  single  evaluation  study  is  virtually 

impossible. It is recommended to focus the prioritisation exercise at the level of (sub-) evaluation 

questions and (sub-) judgement criteria, to allow sufficient data basis for conclusion making per 

(sub-) judgment criterion.   

12 It should be noted that some indicators are identical for both evaluation questions, be it that the way in which they need 

to be analysed will be different.  

50 

 
 
 
                                                      
For  a  prioritised  set  of  (sub-)  judgement  criteria  the  study  team  has  defined  suggested  data 

sources  for  each  indicator.  For  the  evaluation  the  following  data  collection  methods  are 

suggested:  

•  Desk research of existing data; 
•  Collection of additional data from the Member States; 
•  Three  different  web-based  surveys  targeting  businesses,  customs  authorities  and  non-

customs authorities; 

In-depth interviews with customs authorities and businesses 

• 
•  Case studies. 

In  terms  of  data  availability,  it  appears  that  some  relevant  data  are  already  available,  but  that 

significant efforts will be required to collect additional data from the Member States and through 

web-based  surveys.  For  more  detailed  data  the  use  of  case  studies  is  suggested.  In-depth 

interviews are recommended to complement the data collected and to gain further understanding 

on the reasons behind the findings.  

The budget needed for the actual  evaluation  will  depend on the outcome of the prioritisation  of 

(sub-) evaluation questions and (sub-) judgement criteria – information  which  was not available 

at  the  time  of  writing.  Three  scenarios  for  the  budget  estimation  of  the  evaluation  have  been 

developed, with varying levels of data collection activities. It should be emphasised that for each 

of  the  three  scenarios  it  is  assumed  that  there  will  be  a  significant  reduction  in  the  number  of 
indicators to be considered. At an average cost of €1,000/day, the total cost of man days would 
be estimated to be between €150,000 (scenario 1) and €242,000 (scenario 3). The travel budget 
is  estimated  to  range  between  €25,000  (scenario  1)  and  €35,000  (scenario  3).  In  addition, 

translation costs may be required if it  is decided to  make the online survey available in various 

languages. 

If  the  evaluation  project  could  be  launched  by  mid  May  2012,  it  is  estimated  that  the  results 

would be available to DG TAXUD by mid November 2012.  

51 

 
 
 
 
9 Annexes 

9.1  Annex 1: Overview of tasks performed for the study 

Due  to  the  complexity  of  the  topic,  the  project  of  evaluating  the  current  state  of  the  Customs 

Union was divided into three phases. The first phase (of which this report is the final deliverable) 

involves  defining  the  scope  and  developing  the  methodology  of  the  evaluation,  including 

identification  of  relevant  evaluation  questions  and  an  inventory  of  possible  data  sources  to  be 

used. The second phase, the actual evaluation phase, will involve data collection and synthesis 

of results. The third phase will involve follow-up actions, possibly including a Communication on 

the State of the Customs Union, a more politically oriented document based on the results of the 

evaluation.  

Figure 11: Project phases 

I

A
C
T
V
T
E
S

I

I

A
C
T
O
R

P
E
R
O
D

I

2 

PHASE  1:  Feasibility 
study 

PHASE 2: Carrying out of the evaluation  

PHASE 3: Follow up 

(cid:129) Intervention logic 
(cid:129) Evaluation questions 
(cid:129) Analytical framework:  
(cid:129)  Judgment criteria; 
(cid:129)  Indicators; 
(cid:129)  Data sources 

(cid:129)  Feasibility of 

evaluation in the given 
timeframe; 

(cid:129) Estimated budget  

(cid:129)  Data collection: 

(cid:129)  Follow up actions  

(cid:129)  Desk research; 
(cid:129)  In-depth interviews; 
(cid:129)  Surveys to relevant stakeholders; 
(cid:129)  Etc; 

(cid:129)  Analysis of the data; 
(cid:129)  Conclusions based on the findings  

Deloitte 

Managed by TAXUD; Steering Group; contrator  under 
framework contract PWC (tbc)   

Commission (TAXUD) 
and /or Member States  

Jan 2012 – May 2012 

May 2012 – Dec 2012 

After December  2012 

52 

 
 
 
 
 
 
 
 
 
 
 
9.2  Annex 2: Reference list 

Evaluation guidelines 

•  European Commission, Evaluation Standards: 

http://ec.europa.eu/dgs/secretariat_general/evaluation/docs/standards_c_2002_5267_final

_en.pdf 

•  Evaluation in the European Commission: 

http://ec.europa.eu/dgs/secretariat_general/evaluation/index_en.htm 

•  DG Budget, Evaluating EU Activities: A practical guide for Commission services, 2004; 
•  OECD DAC Network on Development Evaluation: Glossary  of Key Terms in Evaluation 

and Results Based Management, 2002 

EU reports 

•  European  Commission,  Report  from  the  Commission  to  the  Council  and  the  European 

Parliament.  Report  on  Progress  on  the  Strategy  for  the  Evolution  of  the  Customs  Union, 

Brussels, 2011  

•  European  Commission  (conducted  by  Deloitte),  Future  business  architecture  for  the 

Customs Union and cooperative model in the taxation area in Europe, Brussels, 2010/2011 

(TAXUD/R3/VDL D(2010) 433216) 

•  European  Commission,  DG  Taxation  and  Customs  Union,  Management  Plan  2012, 

Brussels, 2012 

•  European  Commission,  Commission  Staff  Working  Paper, 

Impact  Assessment 

accompanying the proposal for a regulation of the European Parliament and of the Council 

establishing an action programme for customs and taxation in the EU for the period 2014-

2020 

(FISCUS)  and 

repealing  Decisions  N°1482/2007/EC  and  N°624/2007/EC, 

COM(2011), Brussels, 9.11.2011 

•  European Commission, Communication from the Commission to the Council, the European 

Parliament and the European Economic and Social Committee. Strategy for the evolution 

of the Customs Union, COM(2008) 169, Brussels, 1.4.2008 

•  European Commission. Mid-term evaluation of Fiscalis 2013. Brussels, 2011 
•  Dennis,  D.J.  &  Zainal  Aznam,  Y.,  Developing  Indicators  of  ASEAN  Integration  ‘  A 

Preliminary Survey for a Roadmap. Final Report, August 2003 

•  Gubin, A., Developing performance measurement for the Russian Customs Service, World 

Customs Journal, Vol.5, n°2 

•  Lawrence,  R.Z.,  Drzeniek  Hanouz,  M.,  Doherty,  S.  &  Moavenzadeh,  J.,  The  Global 

Enabling Trade Report 2010, World Economic Forum, Geneva, 2010 

53 

 
 
 
 
•  US AID – Jordan, Developing Service Standards and Performance Indicators for Customs. 

AMIR II Achievement of Market-Friendly Initiatives and Results, April 2006 

•  European Commission (DG for Internal Policies). Implementation of the Modernised 

Customs Code, Brussels, 2012 

•  Fournel, J., Directeur général des douanes et droits indirects (French Customs Authority), 

Rapport Annuel de performances. Bilan stratégique du rapport annuel de performances, 

2011 

•  Fournel, J., Directeur général des douanes et droits indirects (French Customs Authority), 

Rapport Annuel de performances. Bilan stratégique du rapport annuel de performances, 

Annex 4-5: Liste des indicateurs de CPP 2012-2013 et valeurs cibles associées, 2011 

•  European Commission (DG Taxud), Performance Measurement Project: Annex 2: Detailed 

table with critical areas, indicators and remarks/aspects to be measured, 2011 

•  World Bank, Connecting to compete 2010: Trade Logistics in the global economy. The 

Logistics Performance Index and its indicators, 2010: 

http://siteresources.worldbank.org/INTTLF/Resources/LPI2010_for_web.pdf 

•  ASEAN Customs Capacity Building Working Group (CCBWG), Compilation of Key 

Performance Indicators of Customs Services. Strategic Plan of Customs Development on 

Customs Reforms and Modernisation, xxx: 

http://www.asean.org/documents/AEC/customs/DG20-Annex21.pdf 

•  Asia-Pacific Economic Cooperation, Suggested Key Performance Indicators (KPIs). 

2008/SMO1/SSCP/004, 2007: 

http://aimp.apec.org/Documents/2008/SCCP/SCCP1/08_sccp1_004.pdf 

•  Tullverket (Swedish Customs Auhority), Årsredovisning 2011, Stockholm, 2011: 

http://www.tullverket.se/download/18.2a2db13b135a4da826ab9/Tullverket_%C3%85rsred

ovisning_2011.pdf 

•  European Commission (DG Taxud), Measurement of Results, Guidance notes, Brussels, 

Revision January 2010 

•  European Commission (OLAF), Eleventh operational report of the European Anti-fraud 

Office, Brussels, 2011: 

http://bookshop.europa.eu/en/eleventh-operational-report-of-the-european-anti-fraud-

office-pbOBAC11001/?CatalogCategoryID=Z5AKABstzL8AAAEjw4YY4e5K 

•  European Commission (DG Taxud and DG Enterprise and Industry), Drug precursors, EU 

summary report 2011 –statistics from 2010-, Brussels, 2011: 

http://ec.europa.eu/taxation_customs/customs/customs_controls/drugs_precursors/seizure

s/index_en.htm 

•  European Commission (DG Taxud), Report on EU customs enforcement of intellectual 

property rights, Results at the EU border – 2010, 2011: 

http://ec.europa.eu/taxation_customs/customs/customs_controls/counterfeit_piracy/statistic

s/index_en.htm 

54 

 
 
9.3  Annex 3: List of interviewees 

The following people have been interviewed during the preparatory phase: 

Table 6: People interviewed during the preparatory phase 

Name 

Unit 

Theo Vassiliadis 

DG TAXUD - Customs systems and IT operations 

Date of 
interview 

19/01/2012 

19/01/2012 

Jacky Marteau 

Patrick Wallez 

DG TAXUD - Tariff economics, relief from customs duties, 
TARIC, outermost regions 

DG  TAXUD  -  Tariff  quotas  management  &  surveillance 
requirements 

19/01/2012 

Renato Lazzaroni 

DG TAXUD - TARIC 

19/01/2012 

Dimitrios Vardakis  DG  TAXUD  -  Autonomous  tariff  provisions,  anti-dumping 

19/01/2012 

duties, customs duty reliefs, outermost regions 

Maria Manuela  

DG TAXUD - Risk management and Security 

19/01/2012 

Cabral 

Dagmar Kolinkova  DG TAXUD - Customs policy & evaluation 

19/01/2012 &  

Dieter Kaspereit 

DG TAXUD - Customs policy & evaluation 

06/02/2012 

19/01/2012 

Sophie De Coster 

DG TAXUD - Customs processes & project management 

19/01/2012 

Michael Lux 

DG TAXUD - Customs processes & project management 

19/01/2012 

Karlheinz Kadner 

DG  TAXUD  -  Protection  of  citizens  and  enforcement  of 
IPR 

19/01/2012 

Caroline Edery 

DG  TAXUD  -  Protection  of  citizens  and  enforcement  of 
IPR 

19/01/2012 

Tomas Kucirek 

DG TAXUD - Customs policy 

Peter Timmermans  DG TAXUD - Risk management and Security 

23/01/2012 

23/01/2012 

Iosif Dascalu 

DG  TAXUD 
programmes 

- 

Information  and  management  of 

26/01/2012 

Jean-Michel Grave  DG TAXUD - Customs legislation 

Jozef Hupperetz 

DG  BUDG  -  Control  of  traditional  own  resources  and 
assistance to the candidate countries 

26/01/2012 

26/01/2012 

Antti Suortti 

DG  BUDG  -  Control  of  traditional  own  resources  and 
assistance to the candidate countries 

26/01/2012 

Tino Eggermont 

DG  BUDG  -  Control  of  traditional  own  resources  and 
assistance to the candidate countries 

26/01/2012 

Paul-Hervé 
Theunissen 

DG TAXUD - Taxation systems and IT compliance 

26/01/2012 

55 

 
 
 
 
The following individuals have attended the brainstorming session on the needs of businesses:  

Table 7: Participants to the brainstorming session on needs of businesses 

Name 

Function 

Ives Taelman 

Head of Customs operations Europe 

Gordon Wright 

Senior manager, Cargo Security Process and Standards IDFS Cargo 

Steven Pope 

European Head of Customs and Regulatory Affairs 

Graham Austin 

Public Affairs Manager, kappé international 

Axel Klein 

Manager Handling Competence Customs, Authorities & Committees 

Mark Van de Perre  

Clecat 

Christian Piaget 

World Shipping Council 

9.4  Annex 4: Glossary of terms and concepts used  

•  Activity: Actions taken or work performed through which inputs are mobilized to produce 

specific outputs. 

•  Analytical Framework: an analytical framework organises the evaluation questions into 

a  single  structure  and  assigns  to  each  evaluation  question  one  or  several  judgement 

criteria. This procedure makes sure that all the evaluation questions are fully covered in 

terms  of  scope  and  identifies  the  indicators  that  will  feed  the  analysis.  The  analytical 

framework  can  therefore  be  defined  as  a  logical  roadmap  of  this  part  of  the  study  that 

guides the consultants in their analysis.   

•  Data collection: In the analytical framework, a detailed programme is laid out, specifying 

interview  schedules,  surveys  and  case  studies,  etc.  A  distinction  is  made  between 

primary  and  secondary  data  sources.  Primary  data  collection  could  keep  (pleasant  or 

unpleasant) surprises in store, which should be mitigated. The difficulties of primary data 

collection  concern  lack  of  cooperation  from  informants,  or  the  absence  or  weakness  of 

data being looked up (low comparability, various biases, self-censorship, etc.). In order to 

minimise  data  collection  costs,  one  could  rely  on  existing  secondary  data,  such  as 

published statistical resources, previous evaluation reports, or other studies. Issues that 

might arise from using secondary data sources include the accuracy of data, their age, 

comparability, the  extent  to which  data is of an  aggregated or disaggregated  nature, or 

the definitions of certain concepts. 

56 

 
 
 
 
•  Effectiveness:  evaluates  the  extent  to  which  the  objectives  of  the  policy  interventions 

were  achieved,  or  are  expected  to  be  achieved,  taking  into  account  their  relevant 

importance. It should be noted that effectiveness is also used an aggregate measure of 

the merit or worth of an intervention (OECD definition). 

•  Efficiency:  evaluates  how  economically  resources  or  inputs  (budget,  expertise,  time, 

etc.)  are  transformed  into  outputs  (OECD  definition).  For  the  definition  of  “efficiency”  in 

this  study,  the  term  as  it  is  considered  by  the World  Economic  Forum  "Doing  business 

Survey" and the World Bank’s Logistics Performance Index (LPI) are considered, which 

define  speed,  simplicity,  predictability  and  cost-effectiveness  as  the  main  efficiency 

concerns. 

• 

• 

• 

Impact: Positive and negative, primary and secondary long-term effects produced by an 

intervention, directly or indirectly, intended or unintended. 

Indicators: in order to be able to judge whether a policy intervention is successful or not 

against a give criterion, indicators need to be formulated to structure the data that should 

be collected. They precisely specify which data are to be collected and avoid collecting 

an excessive amount of irrelevant data and to focus the process on the questions asked. 

ndicators  could  have  a  quantitative  or  qualitative  nature  and  may  be  formulated  with 

regard to inputs, outputs, outcomes or impacts. 

Input: The financial, human, and material resources used for the policy intervention. 

•  Judgement criteria:  in the analytical framework, the  evaluation questions are linked to 

clear judgement criteria that specify which aspects of the questions are being analysed. 

Each evaluation question is being made more concrete through one or more judgement 

criteria, in order to avoid subjectivity and to make judgments on pre-defined and agreed 

terms; to improve the transparency of the evaluation by making the judgment explicit; to 

structure  the  answers  to  the  evaluation  questions,  as  the  judgement  criteria  will  define 

the indicators and the nature of the data collected. 

•  Outcomes: 

the 

likely  or  achieved  short-term  and  medium-term  effects  of  an 

intervention’s outputs. 

•  Outputs: the products, capital goods and services which result from an intervention 

•  Relevance:  evaluates  the  extent  to  which  the  objectives  of  the  policy  intervention  are 

consistent with the beneficiaries’ or stakeholders’ requirements and needs. It should be 

noted  that  retrospectively,  the  question  of  relevance  often  becomes  a  question  as  to 

whether the objectives of an intervention or its design are still appropriate given changed 

circumstances or evolutions (OECD definition). 

57 

 
 
•  Uniformity:  a  state  or  condition  in  which  everything  is  regular,  homogeneous,  or 

unvarying 

•  Utility/impact:  evaluates  the  positive  and  negative,  primary  and  secondary  long-term 

effects  produced  by  a  policy  intervention,  directly  or  indirectly,  intended  or  unintended 

(OECD definition). 

58 

 
 
9.5  Annex 5: Definitions of evaluation criteria applied for the feasibility 

study 

The table below contains definitions of the evaluation criteria that have been retained for the 

evaluation of the Customs Union. With regard to their definitions and description, a ‘best of both 

worlds”  approach  has  been  followed,  combining  elements  from  the  following  major  strategic 

evaluation methodologies:  

•  DG Budget, Evaluating EU Activities: A practical guide for Commission services, 2004; 
•  OECD DAC Network on Development Evaluation: Glossary  of Key Terms in Evaluation 

and Results Based Management, 2002 

Table 8: Definitions of evaluation criteria applied for the feasibility study 

1314

Evaluation criterion 

Definition 

Relevance 

Effectiveness 

Efficiency 

Utility/impact 

Uniformity 

Relevance  evaluates  the  extent  to  which  the  objectives  of  the 
policy  intervention  are  consistent  with  the  beneficiaries’  or 
stakeholders’ requirements and needs. It should be noted that 
retrospectively,  the  question  of  relevance  often  becomes  a 
question  as  to  whether  the  objectives  of  an  intervention  or  its 
design  are  still  appropriate  given  changed  circumstances  or 
evolutions (OECD definition). 

Effectiveness  evaluates  the  extent  to  which  the  objectives  of 
the  policy  interventions  were  achieved,  or  are  expected  to  be 
achieved,  taking  into  account  their  relevant  importance.  It 
should  be  noted  that  effectiveness  is  also  used  an  aggregate 
measure  of  the  merit  or  worth  of  an  intervention  (OECD 
definition).  

Efficiency  evaluates  how  economically  resources  or  inputs 
(budget,  expertise,  time,  etc.)  are  transformed  into  outputs 
(OECD definition).  

For  the  definition  of  “efficiency”  in  this  study,  the  term  as  it  is 
considered  by  the  World  Economic  Forum  "Doing  business 
Survey"  and  the  World  Bank’s  Logistics  Performance  Index 
(LPI)  have  been  used,  which  define  speed,  simplicity, 
predictability  and  cost-effectiveness  as  the  main  efficiency 
concerns.  

Utility  evaluates  the  positive  and  negative,  primary  and 
secondary long-term effects produced by a policy intervention, 
directly or indirectly, intended or unintended (OECD definition).  

A state or condition in which everything is regular, 
homogeneous, or unvarying 

13 OECD, DAC, Working Party on Aid Evaluation:glossary of key terms in evaluation and results-based 

management  

14 EC evaluation guidelines: http://ec.europa.eu/europeaid/evaluation/methodology/glossary/glo_en.htm 

59 

 
 
 
 
                                                      
60 

 
 
 
9.6  Annex 6: Mapping of the evaluation questions with the evaluation 

criteria and the intervention logic 

The table below provides a mapping of the evaluation questions retained for the feasibility study, 

as well as the evaluation criteria that apply and the elements of the intervention logic considered: 

Table 9: Overview of evaluation questions, criteria and relationship on the intervention logic 

Evaluation question 

Evaluation criteria 

Intervention logic 
relationship 

•  To what extent are the 
strategic objectives of 
the Customs Union still 
relevant and 
sustainable for the next 
decade?  

•  To what extent are the 
core processes of the 
Customs Union 
uniformly applied 
across the Customs 
Union? 

•  Are the core activities 
of the Customs Union 
being efficiently 
performed? Are there 
great differences 
across MS in the 
efficiency? 

•  Sustainability-relevance 

•  Needs (cid:217) objectives 

needs 

and 
(linking 
objectives  of  the  Customs 
Union) 

•  Uniformity 

•  Objectives  (cid:217)  inputs  (cid:217) 
activities (cid:217) outputs 

•  Efficiency 

•  Cost-effectiveness 

• 

• 

Inputs  (cid:217)  activities  (cid:217) 
outputs 

Inputs  (cid:217)  activities  (cid:217) 
outcomes 

•  What is the quality of 

•  Effectiveness  

•  Objectives (cid:217) outcomes 

the service provided by 
the Customs Union to 
its main stakeholders? 

•  Efficiency 

• 

Inputs  (cid:217)  activities  (cid:217) 
outputs 

•  What have been the 

•  Effectiveness 

•  Objectives (cid:217) outcomes 

effects/outcomes of the 
major recent policy 
initiatives in the field of 
Customs Union for the 
Customs Union and its 
stakeholders? 

•  Has the Customs Union 
achieved its strategic 
objectives? Has it 
generated its intended 
effects? 

•  Effectiveness 

•  Objectives (cid:217) outcomes  

•  How has the Customs 
Union absorbed the 

•  Effectiveness 

•  All  aspects  identified  for 
evaluation 

other 

the 

61 

 
 
 
 
enlargement of the EU? 

questions 

62 

 
 
9.7  Annex 7: Long list of data to be collected per data source (priority 1 judgement criteria) 

Table 10: Long list of data to be collected at central Member State level (priority 1) 

Evaluation question 

Indicators for which data should be collected at MS level 

Evaluation Question 2: To what extent are the core 
processes of the Customs Union uniformly applied 
across the Customs Union 
To what extent are the control processes uniformly applied 
across the Customs Union? What are the main reasons for 
differences  in  application  of  the  control  procedures  or  for 
non-uniformity?  How  significant  are  the  consequences  of 
the  differences  in  application  of  these  processes  as 
identified? 

To  what  extent  are  the  enforcement  processes  uniformly 
applied across the Customs Union? 

To  what  extent  are  the  economic  operator  processes 
uniformly applied across the Customs Union? What are the 
main reasons for differences in application of the economic 
operator  processes  or  for  non-uniformity?  How  significant 
are  the  consequences  of  the  differences  in  application  of 
these processes as identified? 

•  Proportion of physical controls by means of scanner versus total of physical controls 
• 

 Number  of  controls  by  means  of  scanner  vis-à-vis  total  number  of  customs-related  goods 
movements / intelligent target selection 

•  Relative number of risk-based controls (vis-à-vis total number of controls) 
•  Number of physical controls 

•  Number of prosecutions per domain 
•  Average height of penalty per infringement across EU Member States 
•  Variety in amount of unpaid fines across the Member States 
•  Number of cases where a fine/penalty was not paid/executed 
•  Actions taken to recover amounts/to execute penalties 
•  Percentage  of  cases  where  a  fine  or  other  penalty  is  imposed  in  relation  to  the  number  of 

prosecutions 

•  Percentage  of  uncollected  fines    /  penalties  not  executed  in  relation  to  the  number  of 

fines/penalties imposed 

•  Number of prior audits performed compared to the number of applications across the Customs 

Union 

•  Number of prior audits performed compared to the number of AEO authorizations across the 

Customs Union 

•  Differences in items checked during prior audits across the Customs Union 
•  Business perception on the main reasons for differences/non-uniformity 
•  Customs authority perception of the main reasons for differences / non-uniformity 

63 

 
 
 
 
 
Evaluation question 3: Are the core activities of the 
Customs Union being efficiently performed? Are there 
great differences across MS in the efficiency 

Are current clearance processes performed efficiently?    

•  Number of AEO suspensions compared to the number of post-audits 
•  Number of AEO suspensions compared to the number of AEO authorizations 

•  Average time between declaration of goods and acceptance of the declaration 
•  Average time between declaration of goods and release of the goods (in case of control / not 

in case of control)  

•  Average time between customs declaration and release of goods for those goods coming from 

AEO businesses and for those goods coming from non-AEO businesses 

•  Percentage  of  import/transit/export  declarations  under  normal  clearance  procedures  vs. 

simplified procedures 

•  Percentage of import/transit/export declarations made electronically vs. paper-based 
•  Number of actions to be taken by businesses to go through the pre-arrival/departure process 

and customs declaration/clearance process 

•  Average  revenues  collected  per  import  declaration  compared  to  average  number  of  hours 

performed by customs authorities on declaration processes per import declaration 

•  Average  number  of  hours  spent  by  customs  officials  per  customs  declaration  

(import/export/transit) compared to trade flow (import/export/transit) 

•  Average  number of hours spent per customs declaration  (import/export/transit) compared to 

GDP 

•  Evolution  in  the  unit  cost  per  declaration  (total  number  of  hours  spent  on  declarations/total 

number of declarations) 

•  25% of customs duties retained by the Member States  compared with the means provided by 

national authorities to the custom authorities 

•  Customs  duties,  other  taxes  and  levies  collected  by  the  Member  States  via  customs 
authorities  compared with the national customs authorities' cost relating to import clearance 
formalities an collection of customs duties, other taxes and levies 

•  Total  customs  agency  costs  compared  to  customs  duties,  other  taxes  and  levies  collected 

Are current control processes performed efficiently? 

(Cost of customs operations vs. revenue collected per MS) 

•  Total number of import/export/transit declarations (normal and simplified procedure) subject to 

documentary control 

•  Proportion of documentary controls in the supply chain (declarations subject to control vs total 

number of declarations) 

•  Total numbers of officials performing documentary controls vs. hit rate  
•  Total numbers of hours devoted to documentary controls vs. hit rate  

64 

 
 
 
•  Total numbers of officials performing documentary controls  vs.  Additional revenues collected 

due to documentary controls 

•  Total  numbers  of  hours  devoted  to  documentary  controls  vs.  Additional  revenues  collected 

due to documentary controls 
•  Unit cost per documentary control 
•  Total  number  of  import/export/transit  declarations  (normal  and  simplified  procedures)  subject 

to physical control 

•  Proportion of physical controls in the supply chain (travellers controls, goods controls, on-site 

controls) 

•  Total  numbers  of  officials  performing  physical  controls  vs.  revenues  collected  due  to  these 

physical controls 

•  Total numbers of hours devoted to physical controls vs. Hit rate  
•  Total numbers of officials performing physical controls vs. hit rate  
•  Total resources (HR, equipment, etc) invested in physical controls vs revenues collected due 

to these physical controls 

•  Total resources invested in lab tests vs hit rate 
•  Unit cost per physical control  
•  Number of significant investigations/seizures  (per 100.000) 
•  Number  of  customs  officials  involved  in  criminal/civil  investigations  compared  to  the  total 

number of criminal/civil investigations 

•  Number  of  investigations  leading  to  court  cases,  administrative  or  other  consequences  (fine, 

penalties, destruction) 

•  Time spent on appeals and investigations (both by customs agencies and for businesses) 
•  Cost  per  prosecution  (hours  performed  by  customs  authorities/public  prosecutors)  on 

prosecution vs. Number of prosecutions) 

•  Number of prosecutions initiated compared to number of prosecutions leading to convictions 
•  Value of monetary penalties 
•  Time/resources  spent  on  imposing  and  collecting  fines  compared  to  the  amounts  of  fines 

collected 

•  Value of imposed fines that were actually collected (%) 
•  Time spent on appeals and investigations (both by customs agencies and for businesses) 
•  Cost  per  prosecution  (hours  performed  by  customs  authorities/public  prosecutors)  on 

prosecution vs. Number of prosecutions) 

•  Number of prosecutions initiated compared to number of prosecutions leading to convictions 
•  Value of monetary penalties 
•  Time/resources  spent  on  imposing  and  collecting  fines  compared  to  the  amounts  of  fines 

65 

Are current enforcement processes performed 
efficiently? 

 
 
Are current risk management processes performed 
efficiently?    

Are current data management processes performed 
efficiently?    

Are current authorisations (AEO and permit) 
management processes performed efficiently?    

Evaluation question 4: What is the quality of the 
service provided by the Customs Union to its main 
stakeholders? 

Do customs authorities sufficiently involve their main 
stakeholders in decision-making and rule-making and 
communicate adjustments? 

Are the services provided by the customs authorities 
customer-oriented? 

collected  

•  Value of imposed fines that were actually collected (%) 
•  Amount of resources (time/money) invested in risk assessment and analysis 
•  Amount of resources (time/money) invested in system monitoring and analysis 
•  Amount of resources (time/money) invested in intelligence development 
•  Amount of resources (time/money) invested in data collection and exchange 
•  Amount of resources (time/money) invested in data storage 
•  Amount of resources (time/money) invested in dissemination of data 
•  Cost of communication and information activities compared to the size of the target group  
•  Average  time  between  application  for  registration  by  a  trader  and  issuance  of  the  EORI 

number 

•  Average  time  between  application  for  authorizations  /  AEO  by  a  trader  and  issuance  of  the 

certificate / decision not to issue the certificate 

•  Relative number of companies with multiple registrations 
•  Amount of customs resources (time/money) invested in AEO authorization process compared 

to number of businesses going through the application process 

•  Amount of customs resources (time/money) invested in AEO authorization process compared 

to number of businesses successfully going through the application process 

•  Average time between submitting an appeal and the appeal decision 
•  Average time between start and conclusion of prior audit 

•  List  of  MS  in  which  the  government  offers formal  consultation  with  stakeholders  on  policy  or 

legislative proposals (and what these MS represent in terms of trade volume) 

•  List  of  formal  stakeholder  platforms  (EU  and  national  levels)  between  customs  and 

stakeholders exists  

•  Classification  of  the  type  of  stakeholders  part  of  these  stakeholders  platforms  (e.g.  non-

customs authorities, farmers, producers, traders, SMEs, NGOs, academics 

•  List of channels open for businesses to contact customs (EU and per MS) or the responsible 

Ministry, per MS (e.g. website, hotline, etc) 

•  Number  of  MS  in  which  customs  have  established  a  single  point  of  contact  for  businesses 

(and what these MS represent in trade volume) 

•  List  of  MS  in  which  declarations  can  be  filed  in  advance  of  arrival  (and  what  these  MS 

represent in trade volume) 

•  List  of  MS  in  which  consignments  can  be  cleared  24h/day  (and  what  these  MS  represent  in 

66 

 
 
 
Do customs authorities sufficiently support a 
paperless customs environment? 

Do Custom officials have the right competences and 
skills to provide a quality service? 

Evaluation question 5: Effects of policy initiatives 

What have been the effects of the Safety and Security 
Amendment of the Community Customs Code 
(including AEO) for the Customs Union and its 
stakeholders? 
What have been the effects of the simplified 
procedures and other related measures? Are there 
differences with regard to the impact of simplified 
procedures per stakeholder type and per Member 
State? Are there differences in impact according to the 
type of simplified procedure? 
What have been the effects of the e-Customs Initiative 
for the Customs Union and its stakeholders? 

Evaluation question 6: Has the Customs Union 
achieved its strategic objectives? Has it generated its 
intended effects? 

Did the Customs Union succeed to detect fraud and tax 
evasion? 

trade volume) 

•  Differences in the amount of time that declarations need to be stored 
•  List of MS in which documents in English are accepted for declarations (DG TAXUD to judge if 

this may be a politically sensitive indicator) 

•  Delays per MS between publication in the Official Journal and communication at national level 

(suggested by the workshop participants, included for reflection) 

•  List of communication channels (at EU and national level) offered by customs 
•  Percentage of emergency procedures in relation to the number of electronic declarations 

•  Ratio of number of training courses in customs legislation followed per staff (per MS) 
•  Budget  spent  per  MS  on  training  the  customs  officials  in  customs  legislation.  Results  of 

employee satisfaction surveys of national customs authorities 

•  Results of employee satisfaction surveys of national customs authorities 

•  Variance in risk selection criteria across the EU (uniformity) 
•  Existence of risk selection criteria in all EU Member States 

•  Time savings (spent on administration) from simplified procedures vis-à-vis baseline scenario 

per type of simplified measure  

•  Time savings (spent on administration) from simplified procedures vis-à-vis baseline scenario 

per type of business 

•  Time savings (spent on administration) from simplified procedures vis-à-vis baseline scenario 

per Member State 

•  Data exchange between customs authorities (import-transit-export) 

•  Number of cases of tax fraud, customs fraud or smuggling successfully prosecuted each year 
•  Hit rate (cases of detected fraud) per total number of controls and per type of control (physical, 

documentary or post-clearance) 

67 

 
 
 
 
 
•  Cost of fraud detection activities, compared to number of cases of fraud detected 
•  Quality of common risk profiles 
•  Risk-based control hit rate vis-vis random control hit rate 
•  Hours performed on fraud detection, compared to number of cases of fraud detected 
•  Number of specimens confiscated and entrusted to a competent authority 
•  Number of infringements of illegal export of cultural goods 
•  Number of confiscated illegal weapons and firearms 
•  Number and value of confiscated illegal pharmaceutical goods  
•  Number and value of confiscated alcohol, tobacco and drugs 
•  Number of confiscated hazardous waste 
•  Number of infringements against food safety regulation 
•  Estimated  relative  value  and  volume  of  stopped  illegal  goods:  comparison  between  stopped 

illegal goods and estimated total number of illegal goods in EU Member States 

•  Evolution in investments to protect against all types of illegal goods over the last 5 years 
•  Total amount of duties charged vs. duties effectively collected 
•  Percentage of duties collected based on control or audit on the total amount 
•  Number of cases with misapplications of origin procedures 
•  Number of cases with misapplications of classification procedures (BTI misapplications) 
•  Number of cases with misapplications of valuation procedures (BTI misapplications) 
•  Costs (EUR per container) to import/ to export: fees levied on a 20-foot container, including all 

kinds of charges but excluding tariffs or trade taxes 
•  Time (in days) needed (for import versus for export) 

Did the Customs Union succeed to stop prohibited and 
illegal goods from entering the market? 

Did the Customs Union succeed to collect customs 
duties, other taxes and levies effectively and correctly? 

Did the Customs Union succeed to reduce the 
administrative burden on businesses for formalities 
related to trade? 

68 

 
 
Table 11: Indicator table for survey among customs authorities 

Evaluation question 

Evaluation Question 2: To what extent are the core 
processes of the Customs Union uniformly applied 
across the Customs Union 

To  what  extent  are  the  clearance  processes  uniformly 
applied  across  the  Customs  Union?  What  are  the  main 
reasons  for  differences  in  application  of  the  clearance 
processes  or  for  non-uniformity?  How  significant  are  the 
consequences  of  the  differences  in  application  of  these 
processes as identified? 

To what extent are the control processes uniformly applied 
across the Customs Union? What are the main reasons for 
differences  in  application  of  the  control  procedures  or  for 
non-uniformity?  How  significant  are  the  consequences  of 
the  differences  in  application  of  these  processes  as 
identified? 

Indicators  for  which  data  should  be  collected  through  a  survey  among 
customs authorities 

•  Number of  emergency procedures used compared to the total number of declarations 
•  Customs  authorities    perception  of  performance  of  the  clearance  system  /  application  of  the 

emergency procedure 

•  Differences  in  conditions  and  prerequisites  for  simplified  procedures  in  various  Member 

States: compliance record, years of activity, guarantee to be set 

•  Differences in list of available simplified clearance at MS level 
•  Customs authority perception of the main reasons for differences / non-uniformity 
•  Perception of national customs officials with regard to uniformity in interpretation of provisions 

regarding origin 

•  Perception of national customs officials with regard to uniformity of interpretation of provisions 

regarding classification 

•  Differences in the proof required for obtaining preferential certificates of origin  between MS 
•  Differences  in  the  proof  required  for  obtaining  non-preferential  certificates  of  origin  between 

MS 

•  Perception of national customs officials with regard to uniformity in interpretation of provisions 

regarding valuation 

•  Extent  to  which  consequences  of  the  difference  in  application  of  these  clearance  processes 

are significant for business and other stakeholders 

•  Comparison between national guidelines for documentary controls: what are the areas that are 

to be checked in documentary controls? 

•  Feedback from customs officials on the scope of their documentary controls 
•  Feedback from businesses on the scope of the documentary controls in their MS 
•  List of MS where documentary controls of risk-based  (and what these MS represent in terms 

of trade volume) 

•  Comparison between national priorities for documentary controls 
•  Perception  of  national  customs  officials  with  regard  to  priority  and  focus  of  documentary 

controls 

•  Perception  of  national  customs  officials  with  regard  to  relative  importance  of  documentary 

69 

 
 
 
controls 

•  Customs authority perception of the main reasons for differences / non-uniformity 
•  Proportion of physical controls by means of scanner versus total of physical controls 
•  Differences  in  aspects  actually  controlled  (fiscal  control,  counterfeit,  IPR,  illegal  persons  and 

• 

goods) 
 Number  of  controls  by  means  of  scanner  vis-à-vis  total  number  of  customs-related  goods 
movements / intelligent target selection 

•  Comparison between national priorities for physical controls 
•  Relative number of risk-based controls (vis-à-vis total number of controls) 
•  Perception of national customs officials with regard to priorities and focus of physical controls 
•  Number of physical controls 
•  Perception of national customs officials with regard to relative importance of physical controls 
•  Customs authority perception of the main reasons for differences / non-uniformity 
•  Uniformity regarding the elements that trigger  an  investigation  (tip, random, etc.) across the 

To  what  extent  are  the  enforcement  processes  uniformly 
applied across the Customs Union? 

To  what  extent  are  the  risk  management  processes 
uniformly applied across the Customs Union? What are the 
main  reasons  for  differences  in  application  of  the  risk 
management  processes  or 
for  non-uniformity?  How 
significant  are  the  consequences  of  the  differences  in 
application of these processes as identified? 

To  what  extent  are  the  economic  operator  processes 
uniformly applied across the Customs Union? What are the 
main reasons for differences in application of the economic 
operator  processes  or  for  non-uniformity?  How  significant 
are  the  consequences  of  the  differences  in  application  of 
these processes as identified? 

Customs Union 

•  Uniformity regarding the entity that has the power to initiate an investigation 
•  Perception of national customs officials with regard to the scope of customs investigations 
•  Perception of national customs authorities wrt priorities and focus of investigations 
•  Types of prosecution that exist: administrative, civil, penal, etc 
•  Perception of national customs authorities wrt priorities and focus of prosecution 
•  Variety across Member States with regard to criteria to determining the height of penalties 
•  Perception  of  national  customs  authorities  wrt  uniformity  regarding  coordination  among  risk 
management  and  intelligence  units  and  between  enforcement  units  and  risk  management 
units 

•  Variety in system monitoring across the EU 
•  Variety in analysis of systems across the Customs Union  
•  Customs authority perception of the main reasons for differences / non-uniformity 
•  Customs authority perception of the main reasons for differences / non-uniformity 
•  Extent  to  which  consequences  of  the  difference  in  application  of  these  processes  are 

significant for business and other stakeholders 

•  Differences in items checked during prior audits across the Customs Union 
•  Customs authority perception of the main reasons for differences / non-uniformity 
•  Differences in Items checked during post-audits across the Customs Union 
•  Customs authority perception of the main reasons for differences / non-uniformity 

70 

 
 
Evaluation question 3: Are the core activities of the 
Customs Union being efficiently performed? Are there 
great differences across MS in the efficiency 

Are current clearance processes performed efficiently?    

Are current data management processes performed 
efficiently?    

Are current authorisations (AEO and permit) management 
processes performed efficiently?    

Evaluation question 4: What is the quality of the 
service provided by the Customs Union to its main 
stakeholders? 

Do customs authorities sufficiently involve their main 
stakeholders in decision-making and rule-making and 
communicate adjustments? 

•  List of MS where pre-arrival declaration of goods is possible 
•  Extent to which customs declaration processes are aligned between Member States 
•  Number of actions to be taken by businesses to go through the pre-arrival/departure process 

and customs declaration/clearance process 

•  Number of changes to the pre-arrival/departure process and the customs declaration process 

over the past three years 

•  Perceived predictability by businesses with regard to clearance processes 
•  Use of the possibility foreseen in article 221 §2  by MS Customs authorities 
•  Amount of resources (time/money) invested in data collection and exchange 
•  Data shared with other customs authorities / other authorities 
•  Amount of resources (time/money) invested in data storage 
•  Extent to which the same data is stored more than once 
•  Amount of resources (time/money) invested in dissemination of data 
•  Cost of communication and information activities compared to the size of the target group  
•  Possibility and use of electronic registration  
•  Existence of registration guidelines  
•  Average  time  between  application  for  authorizations  /  AEO  by  a  trader  and  issuance  of  the 

certificate / decision not to issue the certificate 

•  Average time between submitting an appeal and the appeal decision 
•  Average time between start and conclusion of prior audit 
•  Average time between start and conclusion of post audit 
•  Average time between reception of a request for information and provision of the information  

•  List  of  formal  stakeholder  platforms  (EU  and  national  levels)  between  customs  and 

stakeholders exists  

•  Classification  of  the  type  of  stakeholders  part  of  these  stakeholders  platforms  (e.g.  non-

customs authorities, farmers, producers, traders, SMEs, NGOs, academics etc) 

•  Appreciation  of  national  customs  authorities,  non-customs  authorities,  businesses  and  other 
stakeholders on the extent to which changes in rules and legislation are notified in advance 
•  List  of  MS  in  which  the  government  offers formal  consultation  with  stakeholders  on  policy  or 

legislative proposals (and what these MS represent in terms of trade volume) 

71 

 
 
 
 
•  Appreciation of EU officials, national customs authorities, non-customs authorities, businesses 
and  other  stakeholders  on  the  extent  to  which  the  (EU  and  national)  stakeholders  platforms 
have an impact on decision making and rule making 

•  Appreciation of EU officials, national customs authorities, businesses and other stakeholders 
on  the  extent  to  which  the  (EU  and  national)  stakeholders  platforms  have  an  impact  on 
decision making and rule making 

•  Comparison of opening hours of customs per MS 
•  List of channels open for businesses to contact customs (EU and per MS) or the responsible 

Ministry, per MS (e.g. website, hotline, etc) 

•  Number  of  MS  in  which  customs  have  established  a  single  point  of  contact  for  businesses 

(and what these MS represent in trade volume) 
•  List of MS in which in-house customs is available 
•  List  of  MS  in  which  declarations  can  be  filed  in  advance  of  arrival  (and  what  these  MS 

represent in trade volume) 

•  List  of  MS  in  which  consignments  can  be  cleared  24h/day  (and  what  these  MS  represent  in 

trade volume) 

•  List of MS in which documents in English are accepted for declarations (DG TAXUD to judge if 

this may be a politically sensitive indicator) 

•  Delays per MS between publication in the Official Journal and communication at national level 

Are the services provided by the customs authorities 
customer-oriented? 

(suggested by the workshop participants, included for reflection) 

•  List  of  countries  in  which  a  communication  strategy  exists  to  actively  translate  EU  customs 
policy and legislation into national communication campaigns (and what these MS represent in 
trade volume) 

•  List of communication channels (at EU and national level) offered by customs 
•  Comparison  of  opening  hours  of  customs  for  companies  with  AEO  status  versus  companies 

Do Customs authorities provide additional quality service 
for businesses with an AEO status? 

versus companies without AEO status 

•  Comparison  of  the  quality  of  additional  customs  services  related  to  clearance  offered  to 

without AEO status 

•  Comparison  of  the  extent  to  which  customs  can  be  reached  for  companies  with  AEO  status 

Do Custom officials have the right competences and skills 
to provide a quality service? 

companies with AEO status versus companies without AEO status 

•  Comparison  of  the  extent  to  which  communication  and  information  is  adapted  to  needs  of 

companies with AEO status versus companies without AEO status 

•  Ratio of number of training courses in customs legislation followed per staff (per MS) 
•  Budget spent per MS on training the customs officials in customs legislation 
•  Appreciation  of  customs  officials  on  the  level  of  knowledge  and  expertise  on  customs 

legislation by customs officials in their country 

72 

 
 
Evaluation question 5: Effects of policy initiatives 

What have been the effects of the Safety and Security 
Amendment of the Community Customs Code (including 
AEO) for the Customs Union and its stakeholders? 
What have been the effects of the e-Customs Initiative for 
the Customs Union and its stakeholders? 
Evaluation question 6: Has the Customs Union 
achieved its strategic objectives? Has it generated its 
intended effects? 

Did the Customs Union succeed to reduce the 
administrative burden on businesses for formalities related 
to trade? 

•  Variance in risk selection criteria across the EU (uniformity) 
•  Existence of risk selection criteria in all EU Member States 

•  Data exchange between customs authorities (import-transit-export) 

•  Average number of documents required for import versus for export 
•  Costs (EUR per container) to import/ to export: fees levied on a 20-foot container, including all 

kinds of charges but excluding tariffs or trade taxes 
•  Time (in days) needed (for import versus for export) 
•  Administrative fees for customs clearance 

73 

 
 
 
 
Table 12: Indicator table for survey among businesses 

Evaluation question 

Evaluation Question 2: To what extent are the core 
processes of the Customs Union uniformly applied 
across the Customs Union 

To  what  extent  are  the  clearance  processes  uniformly 
applied  across  the  Customs  Union?  What  are  the  main 
reasons  for  differences  in  application  of  the  clearance 
processes  or  for  non-uniformity?  How  significant  are  the 
consequences  of  the  differences  in  application  of  these 
processes as identified? 

To what extent are the control processes uniformly applied 
across the Customs Union? What are the main reasons for 
differences  in  application  of  the  control  processes  or  for 
non-uniformity?  How  significant  are  the  consequences  of 
the  differences  in  application  of  these  processes  as 
identified? 

To  what  extent  are  the  enforcement  processes  uniformly 
applied across the Customs Union? 

To  what  extent  are  the  risk  management  processes 
uniformly applied across the Customs Union? What are the 
main  reasons  for  differences  in  application  of  the  risk 
for  non-uniformity?  How 
management  processes  or 

Indicators  for  which  data  should  be  collected  through  a  survey  among 
businesses 

•  Business  perception  of  performance  of  the  clearance  system  /  application  of  the  emergency 

procedure 

•  Business perception on the main reasons for differences/non-uniformity 
•  Perception  of  businesses  with  regard  to  uniformity  in  interpretation  of  provisions  regarding 

origin 

•  Perception  of  businesses  with  regard  to  uniformity  in  interpretation  of  provisions  regarding 

valuation 

•  Perception  of  businesses  with  regard  to  uniformity  of  interpretation  of  provisions  regarding 

classification 

•  Business perception on the main reasons for differences/non-uniformity 
•  Extent  to  which  consequences  of  the  difference  in  application  of  these  processes  are 

significant for business and other stakeholders 

•  Perception of business with regard to priority and focus of documentary controls 
•  Perception of business with regard to relative importance of documentary controls 
•  Business perception on the main reasons for differences/non-uniformity 
•  Perception of business with regard to priorities and focus of physical controls 
•  Perception of business with regard to relative importance of physical controls 
•  Business perception on the main reasons for differences / non-uniformity 
•  Perception of business with regard to the scope of customs investigations 
•  Perception of business wrt priorities and focus of investigations 
•  Perception of business wrt priorities and focus of prosecution 
•  Business perception on the main reasons for differences/non-uniformity 
•  Extent  to  which  consequences  of  the  difference  in  application  of  these  processes  are 

significant for business and other stakeholders 

74 

 
 
 
 
 
significant  are  the  consequences  of  the  differences  in 
application of these processes as identified? 
To  what  extent  are  the  economic  operator  processes 
uniformly applied across the Customs Union? What are the 
main reasons for differences in application of the economic 
operator  processes  or  for  non-uniformity?  How  significant 
are  the  consequences  of  the  differences  in  application  of 
these processes as identified? 
Evaluation question 3: Are the core activities of the 
Customs Union being efficiently performed? Are there 
great differences across MS in the efficiency 

Are current clearance processes performed efficiently?    

Are current authorisations (AEO and permit) management 
processes performed efficiently?    

Evaluation question 4: What is the quality of the 
service provided by the Customs Union to its main 
stakeholders? 

Do  customs  authorities  sufficiently  involve  their  main 
stakeholders  in  decision-making  and  rule-making  and 
communicate adjustments? 

Are the services provided by the customs authorities 
customer-oriented? 

•  Business perception on the main reasons for differences/non-uniformity 

•  Perceived time-efficiency by businesses with regard to customs declaration process 
•  Extent to which customs declaration processes are aligned between Member States 
•  Number of actions to be taken by businesses to go through the pre-arrival/departure process 

and customs declaration/clearance process 

•  Perceived predictability by businesses with regard to clearance processes 
•  Perceived simplicity by businesses on registration procedure 
•  Average  time  between  application  for  authorizations  /  AEO  by  a  trader  and  issuance  of  the 

certificate / decision not to issue the certificate 

•  Average cost for businesses to obtain AEO status / an authorization  

•  List  of  formal  stakeholder  platforms  (EU  and  national  levels)  between  customs  and 

stakeholders exists  

•  Classification  of  the  type  of  stakeholders  part  of  these  stakeholders  platforms  (e.g.  non-

customs authorities, farmers, producers, traders, SMEs, NGOs, academics etc) 

•  Appreciation  of  national  customs  authorities,  non-customs  authorities,  businesses  and  other 
stakeholders on the extent to which changes in rules and legislation are notified in advance 
•  Appreciation of EU officials, national customs authorities, non-customs authorities, businesses 
and  other  stakeholders  on  the  extent  to  which  the  (EU  and  national)  stakeholders  platforms 
have an impact on decision making and rule making 

•  Feedback  of  traders  on  extent  to  which  customs  opening  hours  match  the  arrival  of 

shipments/their needs 

•  Feedback of the businesses on the reachability of customs (EU and national level) 
•  Feedback of businesses on the quality  of the communication and information available (user 

75 

 
 
 
 
 
Do Customs authorities provide additional quality service 
for businesses with an AEO status? 

Do customs authorities sufficiently support a paperless 
customs environment? 

Do Custom officials have the right competences and skills 
to provide a quality service? 

Evaluation 5: What have been the effects/outcomes of 
the major recent policy initiatives in the field of 
Customs Union for the Customs Union and its 
stakeholders? 
What have been the effects of the simplified procedures 
and other related measures? Are there differences with 
regard to the impact of simplified procedures per 
stakeholder type and per Member State? Are there 
differences in impact according to the type of simplified 
procedure? 
Evaluation question 6: Has the Customs Union 
achieved its strategic objectives? Has it generated its 
intended effects? 
Did the Customs Union succeed to detect fraud and tax 
evasion? 
Did the Customs Union succeed to reduce the 

friendliness, accuracy, up to date, etc) 

•  Comparison  of  opening  hours  of  customs  for  companies  with  AEO  status  versus  companies 

without AEO status 

•  Comparison  of  the  extent  to  which  customs  can  be  reached  for  companies  with  AEO  status 

versus companies without AEO status 

•  Comparison  of  the  quality  of  additional  customs  services  related  to  clearance  offered  to 

companies with AEO status versus companies without AEO status 

•  Comparison  of  the  extent  to  which  communication  and  information  is  adapted  to  needs  of 

companies with AEO status versus companies without AEO status 

•  Feedback  of  companies  with  AEO  status  on  other  areas  in  which  there  is  additional  quality 

service 

•  Feedback of the businesses and non-customs authorities on the quality of paperless customs 

services in their MS (for each aspect as identified in the judgment criterion above) 

•  Feedback of the customs officials, businesses and non-customs authorities on the (perceived) 

security level of the electronic customs systems 

•  Appreciation of businesses on the level of knowledge and expertise on customs legislation by 

customs officials in their country 

•  Transparency scores per MS (client perception of integrity of customs officials) 
•  Appreciation of businesses on the customer-orientation of the customs officials in their country 

•  Time savings (spent on administration) from simplified procedures vis-à-vis baseline scenario 

per type of simplified measure  

•  Time savings (spent on administration) from simplified procedures vis-à-vis baseline scenario 

per type of business 

•  Time savings (spent on administration) from simplified procedures vis-à-vis baseline scenario 

per Member State 

•  Perception of business regarding risk of fraud detection and on severity of consequences 

•  Average number of documents required for import versus for export 

76 

 
 
 
 
administrative burden on businesses for formalities related 
to trade? 

•  Costs (EUR per container) to import/ to export: fees levied on a 20-foot container, including all 

kinds of charges but excluding tariffs or trade taxes 
•  Time (in days) needed (for import versus for export) 
•  Administrative fees for customs clearance 
•  Cost of delay for businesses 
•  Average number of time spent by companies with customs officials 

77 

 
 
 
Table 13: Indicator table for survey among non-customs authorities 

Evaluation question 

Indicators for which data should be collected through a survey 
among non-customs authorities 

Evaluation question 4: What is the quality of the service provided 
by the Customs Union to its main stakeholders? 

Do  customs  authorities  sufficiently  involve  their  main  stakeholders  in 
decision-making and rule-making and communicate adjustments? 

Are the services provided by the customs authorities customer-
oriented? 

Do customs authorities sufficiently support a paperless customs 
environment? 

Do Custom officials have the right competences and skills to provide a 
quality service? 

•  Appreciation of national customs authorities, non-customs authorities, businesses 
and  other  stakeholders  on  the  extent  to  which  changes  in  rules  and  legislation 
are notified in advance 

•  Appreciation  of  EU  officials,  national  customs  authorities,  non-customs 
authorities,  businesses  and  other  stakeholders  on  the  extent  to  which  the  (EU 
and national) stakeholders platforms have an impact on decision making and rule 
making 

•  Feedback  of  non-customs  authorities  on  the  quality  of  the  communication  and 

information available (user friendliness, accuracy, up to date, etc 

•  Feedback  of  the  businesses  and  non-customs  authorities  on  the  quality  of 
paperless  customs  services  in  their  MS  (for  each  aspect  as  identified  in  the 
judgment criterion above) 

•  Feedback  of  the  customs  officials,  businesses  and  non-customs  authorities  on 

the (perceived) security level of the electronic customs systems 

•  Appreciation of businesses on the level of knowledge and expertise on customs 

legislation by customs officials in their country 

•  Appreciation  of  other  authorities/agencies  on  the  customer-orientation  of  the 

customs officials in their country 

78 

 
 
 
 
 
9.8  Annex 8: Suggested step-approach for setting up a web-based survey 

Setting-up the web-based survey 

By sending an e-mail to the target population, which can be very large in number, respondents 

can  be  invited  to  complete  the  questionnaire  by  clicking  on  a  hyperlink  that  refers  them  to  the 

web-based questionnaire. Different language versions of the questionnaire could be foreseen, if 

desirable,  linked  to  the  same  database.  When  the  respondents  electronically  submit  the 

questionnaire in their chosen language version, the data is stored in the global database.  

Confidentiality  and  anonymity  of  the  data  collection  is  an  important  aspect  that  needs  to  be 

considered  in  order  to  ensure  a  successful  survey  and  a  high  response  rate.  An  example  of  a 

typical web questionnaire is provided below. 

There are a number of important contextual factors to consider regarding the scope and design 

of a data gathering instrument to be used as the basis for online deployment.  

The success of any online data collection may be largely determined by the characteristics of the 

underlying instrument or set of questions – how complex the questions are (e.g. questions that 

may have dependencies or piping from one question to another), how many questions there are, 

and the mode of the questions (e.g. open-ended vs. Likert Scale). Certain measures can mitigate 

these  risks,  for  instance  following  up  survey  invitees,  carefully  considering  survey  design  with 

regard to usability, and utilising previous experience in conducting online fieldwork.  

The various steps associated with implementing online tools are described below. 

79 

 
 
 
 
Figure 12: Stages associated with implementing online surveys 

Stage 1

Review 
survey

Technical 
knowledge

Stage 2a

Review 
technical 
applicability

Stage 2b

Review 
phrasing of 
questions

Domain 
knowledge

Stage 4

Stage 5 

Test 
internally

Deploy 
online tool

Stage 3

Implement 
online tool

Stage 4a

Conduct 
pilot

Stage 6

Stage 7

Follow up 
responden
ts

Collate and 
hand-over 
data

Optional

Stage 7a

Conduct 
analysis

Continuous task: Regular liaison with client to keep informed of progress

Stage  1:  Review  survey:  initially,  the  data-gathering  instrument  is  reviewed  to  answer  the 

following  questions:  What  is  the  structure  of  the  online  instrument  required?  How  many 

sections/questions  does  it  contain?  How  complex  are  the  questions?  How  many  stakeholder 

groups are expected to be addressed and are there different surveys for each? The answers to 

these  questions  will  impact  both  on  the  details  of  implementation  and  the  expected  response 

rate. In general, as online data collection tools get longer and more complex, the response rate 

drops. It is good practice not to try to deploy something that will take longer than a maximum of 

15–20 minutes for a response, otherwise the response rate declines considerably.  

Stage  2a:  Review  technical  applicability:  the  online  instrument  is  reviewed  and  refined, 

bearing  in  mind  the  technical  advantages  and  disadvantages  of  such  tools.  For  example,  web-

based  surveys  can  be  constructed  with  yes/no,  radio-button,  drop-down  list  and  open-text 

questions,  questions  where  user  information  is  “piped”  from  one  question  to  the  next  and 

“condition”-based  questions  that  are  only  presented  if  certain  preceding  answers  have  been 

given.  Conditioned  questions  are  of  critical  importance,  as  they  can  be  used  to  reduce  the 

effective length of a survey. A web interface also limits the type of questions that can be used. 

For example, complex, graphical questions may need to be adjusted to allow for answers to be 

supplied with a drop-down list. 

80 

 
 
 
 
Stage  2b:  Review  phrasing  of  questions:  the  phrasing  of  survey  questions  is  of  critical 

importance, as misinterpretations and loss of nuance can often lead to erroneous results. This is 

a far greater challenge than the technical implementation of an online tool. The aim in this phase 

is, where possible, to use domain knowledge to adjust the language and question phrasing and 

thereby reduce the possibility of confusion, bias or misunderstanding in the response. This stage 

is critical to ensuring that the survey gets the results that the study team require. 

Stage 3: Implement online tool: the survey is then implemented on the online platform. Various 

factors  should  be  considered,  including  the  particular  requirements  of  the  project  (e.g.  whether 

respondents will be expected to complete the questionnaire in one sitting, how invitations will be 

distributed),  the  complexity  of  the  instrument  (e.g.  multiple  choice  questions  or  those  requiring 

non-textual  interaction),  robustness  of  the  selected  platform  (e.g.  number  of  version),  and 

experience of those required to use and implement the instrument in the platform.  

Stage 4: Test internally: once the survey is implemented, internal testing is conducted. It is at 

this  stage  that  email  invitations  and  embedded  hyperlinks  are checked,  the  performance  of  the 

instrument in  various  web-browsers (e.g. Opera, Internet  Explorer and Firefox) is checked, and 

data collection verified (whether the instrument is recording data in the appropriate way). These 

checks will reduce the occurrence of problems with the survey both at the pilot and deployment 

stages. When this stage is completed, a link to the final survey instrument is sent to the project 

team for a final opportunity to review. 

Stage  4a: Conduct pilot: following internal testing,  the online instrument is then piloted  with  a 

small  sample  to  test  understanding  amongst  likely  respondents  and  iron  out  any  final  technical 

issues.  The  objective  here  is  to  ensure,  to  the  greatest  degree  possible,  that  the  respondents 

understand  the  questions  in  the  same  way  as  the  creators.  The  piloting  is  used  to  validate  the 

internal testing, to check phrasing of the questions and to address, where possible, any technical 

interface issues. This stage may be omitted or shortened depending on the number of intended 

participants. 

Stage  5:  Deploy  online  tool:  after  successful  piloting,  the  instrument  is  deployed  across  the 

sample  of  stakeholders.  Deployment  must  be  considered  via  intermediary  organisations  (e.g. 

membership  organisations  which  count  stakeholders  as  members),  that  can  act  as  force 

multipliers  for  the  distribution  of  the  instrument.  It  is  always  good  practice  to  establish  a  clear 

feedback mechanism for technical support queries. 

Stage 6: Follow up respondents: once the survey has being deployed for a short time period, 

non-respondents are followed up. Depending on the number of responders outstanding, this will 

be either via email or telephone. This follow-up is intended to maximise response rates.  

81 

 
 
Stage 7: Collate and hand over data: once the survey is completed, the data can be collated 

and  exported  in  a  suitable  format,  either  electronic  (eg  .csv,  .xml,  SPSS,  .xls,  .rtf)  or  paper-

based. 

Stage 7a: Conduct analysis: using the data collated from the survey, analysis is conducted to 

extract results and conclusions. The numerical data can also be  presented in a  graphical form, 

allowing for easy understanding. It is useful to provide a workbook of tabulated results, indicating 

the questions, responses and analysis in a logical manner. 

An example of a typical web questionnaire is provided below. 

Figure 13: Example of a web questionnaire 

82 

 
 
 
9.9  Annex 9: Suggested indicators to monitor changes in the Customs 

Union’s environment 

In the framework of the framework contract DIGIT/R2/PO/2009/027  ABC II regarding the future 

business  architecture  for  the  Customs  Union  and  cooperative  model  in  the  taxation  area  in 

Europe, a number of trends/challenges for the European Customs Union were identified: 

•  Growing globalization/liberalization of trade 
•  The need for more safety and security  
•  Developments of new markets/shift in economic powers 
•  Changes in methods and speed of the movement of goods (increased efficiency) 
•  Environmental challenges 
•  So called ‘Digital globalization’ (e-commerce) and taking full advantage of any beneficial 

technologies (e-customs)  

Increase of tasks appointed to customs authorities 

• 
•  Limited public resources/customs revenues (increased emphasis on efficiency) 
•  Erosion of the political and social support for market integration in Europe 
•  Enlargement of the Customs Union 

For each of these trends, we have developed a number of indicators, as follows: 

Trend/challenge 

Indicators 

Growing globalization/liberalization of trade 

The need for more safety and security  

Developments  of  new  markets/shift 
economic powers 

in 

Changes  in  methods  and  speed  of  the 
movement of goods (increased efficiency) 

•  Number  of  import/export  transactions  in  the 

EU 

•  Number of EORI numbers in the EU 
•  Volume  of  trade  (total  value  of  import  and 

export) in the EU 

•  Total customs duty revenues in the EU 
•  Number  of  countries  with  which  the  EU  has 

Free Trade Agreements in place 

•  Number  of  terrorist  attacks/attempts  in  the 

EU 

•  Number of terrorist organizations identified in 

the EU 

•  Number  of  (discovered) 

food/medicine/… 

safety infringements in the EU  

•  Evolution of GDPs of other global economies 

(BRIC, US, Japan, etc.) 

•  EU  GDP  compared  to  GDPs  of  other  global 

economies 
•  Trade balances 
•  Figures 
on 

transport 

(inbound/outbound/transit) 
mode (air/sea/land; container/bulk/parcel) 

per 

volumes 
transport 

•  Figures 

on 

average 

lead 

times 

83 

 
 
 
 
Environmental challenges 

(inbound/outbound and transit flows)  

•  CO2  emissions  caused  by  intra-  and  extra-

EU traffic of goods 

•  Volume  of 
waste 
•  Volume  of 

import/export  of 

(hazardous) 

import/export  of  hazardous 

called 

‘Digital  globalization’ 

So 
(e-
commerce)  and  taking  full  advantage  of 
any beneficial technologies (e-customs)  

products (chemicals, etc.) 

•  Volume (value) of e-commerce in the EU 
•  Number of e-commerce companies in the EU 
•  Percentage  of  EU 
import/export/transit 

declarations processed via e-customs 

Increase  of  tasks  appointed  to  customs 
authorities 

Limited 
revenues 
efficiency) 

public 
(increased 

resources/customs 
on 

emphasis 

Erosion  of  the  political  and  social  support 
for market integration in Europe 

Enlargement of the Customs Union 

•  Number  of 

to  customs 
tasks  appointed 
authorities  across  EU  Member  States 
(determination  based  on  a  predefined  list  of 
identified tasks)  

•  Appreciation  of  national  customs  authorities, 
non-customs  authorities,  businesses  and 
other  stakeholders  on  the  extent  to  which 
tasks  appointed  to  customs  authorities  have 
increased/decreased 

•  Total  amount  of  public  resources  spent  on 

customs activities across the EU 

•  Total  number  of  Full  Time  Equivalents 
employed  at  customs  authorities  (and  DG 
Taxud) across the EU 

•  Total  amount  of  customs  duties  collected 

across the EU 

•  Appreciation  of  national  customs  authorities, 
non-customs  authorities,  businesses  and 
other  stakeholders  on  the  extent  to  which 
further  market  integration  /  unification  in 
Europe is desirable 

•  Number of Customs Union member states 
•  Number of Customs Union inhabitants 
•  Total GDP of Customs Union  member states 

The indicators reflected above should allow getting a clear picture of the evolution of mentioned 

trends and challenges. 

Subsequently  it  could  be  considered  to  measure  the  impact  of  the  above  trends  on  the 

(perceived) importance of the Customs Union’s strategic objectives (positive, negative or neutral 

impact, for instance on a 5 point scale). Also it can be considered to measure to what extent (it is 

perceived  that)  the  current  Customs  Union’s  strategic  objectives  sufficiently  address  these 

trends/challenges.    Such  measurements  could  be  performed  via  a  survey  or  a  series  of 

interviews with relevant stakeholders (such as authorities, business and society).  

84 

 
 
 
9.10  Annex 10: Overview of workshops held with DG Taxud and other 

stakeholders in the framework of the study 

9.10.1 Workshop 27 February 2012 

Objectives 

•  Present the objectives of the evaluation and of this project; 

•  Present  the  Intervention  Logic  of  the  Customs  Union  and  collect  any  feedback  of  the 

participants; 

•  Explain  how  the  evaluation  questions  will  be  translated  into  judgment  criteria  and 

indicators; 

•  Brainstorm with the participants on which areas of the Intervention Logic would need to 

be looked at by the evaluation & first identification of possible evaluation questions  

Activities performed 

•  Break-out  sessions  were  held  in  order  to  further  refine  the  intervention  logic  and  to 

formulate evaluation questions 

Outcomes 

•  A widely accepted intervention logic  

•  A first set of evaluation questions. 

9.10.2 Workshop 23 March 2012 

Objectives 

85 

 
 
 
 
 
 
 
 
 
The  objective  of  this  workshop  was  to  further  designing  the  methodology  for  the  forthcoming 

evaluation of the Customs Union.  In particular, this second workshop, based on the intervention 

logic  (revised  according  to  the  first  workshop)  and  the  set  of  draft  evaluation  questions,  both 

developed  in  the  course  of  the  project,  focussed,  mainly,  on  the  validation  and  further 

identification of:  

•  Evaluation criteria for the respective evaluation questions  

• 

Indicators and possible data sources  

Further, the  workshop also had the aim of coming to  a first assessment of the feasibility of the 

developed evaluation questions and evaluation criteria. 

Activities performed 

•  Presentation  of  the  objectives  and  scope  of  the  evaluation,  the  current  project  and  the 

workshop ; 

•  Explanation of the Intervention Logic of the Customs Union developed in the framework 

of the project  

•  Presentation of the short list of evaluation questions that were identified in the framework 

of the project  

•  Explanation of the project approach of using an analytical framework  

•  Break-out  sessions  to  validate  or  modify  existing  and  identify  additional  judgement 

criteria, indicators and possible data sources (post-it exercise and discussion) 

•  Feedback of the break-out sessions and group discussion 

•  Conclusions and next steps 

Outcomes 

The participants to the workshop showed themselves very interested in the project at hand, and 

in the idea of conducting a thorough evaluation of the diverse aspects of the Customs Union.  

As  such,  all  participants  actively  participated  to  the  break-out  sessions  and  group  discussions, 

which on the one hand led to a number of valid comments and remarks relating to the evaluation 

criteria  for  the  respective  evaluation  questions  as  they  were  already  developed.  On  the  other 

86 

 
 
 
 
hand, the workshop participants also came up with some new insights, which in their turn led to a 

number of new evaluation criteria and indicators.   

The input that was gathered during the various stages of the workshop (via the post-it exercise 

and  group  discussions)  was  discussed  with  the  project  team,  after  which  it  was  consolidated, 

processed and integrated into the Analytical Framework. 

87