COMMUNICATION FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS Strengthening the foundations of Smart Regulation – improving evaluation
Date: 2013-10-02
Institutions: European Commission
Subjects: application of EU law, collective interest, consolidation of EU law, drafting of EU law, qualitative analysis













EUR-Lex - 52013DC0686 - EN











|



52013DC0686

COMMUNICATION FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS Strengthening the foundations of Smart Regulation – improving evaluation /* COM/2013/0686 final */









1.     
Introduction
Responding to the need
to continually improve and strengthen governance, the European Commission has
developed a comprehensive Smart Regulation policy aimed at facilitating the
achievement of public policy objectives at minimum cost and improving the added
value of EU intervention[1].
Evaluation is a key Smart Regulation tool, helping the Commission to assess
whether EU actions are actually delivering the expected results and ultimately
improving conditions for European citizens and businesses and contributing to
the EU's global role.  
In
its 2012 EU Regulatory Fitness Communication[2], the
Commission re-iterated its commitment to making "the identification of
unnecessary costs and areas for performance improvement an integral and
permanent part of its policy-making and programming across all EU
legislation." Better evaluation practices will play a crucial role in
delivering on this promise.  
The
reforms outlined in this Communication aim to bring the Commission's evaluation
system up-to-date with developments in Smart Regulation policy and make it more
transparent and comprehensive. EU policies need to be regularly and
systematically evaluated, including critical consideration of whether they are
fit for purpose, produce the expected changes and avoid unnecessary regulatory
burden. Better evaluation should increase learning and accountability and improve
stakeholders' participation.
A
recent consultation[3]
showed that whilst stakeholders support the general concept of evaluation, they
see scope for improvement. They call for a more transparent and accessible
evaluation system, with clearer planning and consistent analysis. This would
help them provide relevant information and feedback and make it easier to
understand what may happen after an evaluation finishes.
Drawing
on existing good evaluation practice within the Commission and organisations
undertaking similar analysis, this communication identifies the main areas for
improvement and the changes the Commission is considering to strengthen all aspects
of evaluation performance. It concentrates on retrospective evaluations. Revised
evaluation guidelines, based on the suggestions presented in this
communication, will shortly be put out to public consultation.
2.         
Evaluation in the Commission: state of play
Evaluation does more
than just assess what has happened - it also looks at what caused the change
and how much might reasonably be credited to EU action. It seeks answers to
questions like:
·       
Have
the objectives been met?  (Effectiveness)  
·       
Were
the costs involved justified, given the changes which have been achieved?
(Efficiency) 
·       
Does
the action complement other actions or are there contradictions? (Coherence)
·       
Is
the EU action still necessary? (Relevance)
·       
Can
or could similar changes have been achieved without EU action, or did EU action
make a difference? (EU added value).
By
looking at if, how and why the initial situation has changed, evaluation
provides evidence and informs the debate on current performance and possible
changes. Evaluation plays an important part in organisational learning,
identifying and sharing different practices. It provides an opportunity to
receive stakeholder feedback and requests for change. Thorough evaluation also
identifies unintended and unexpected consequences, which also need to be taken
into account. By publishing evaluation findings, the Commission is publicly
taking responsibility for its actions, acknowledging how an intervention is
performing and inviting further feedback. 
2.1.  
Progress
The Commission has a
long history of evaluating its spending programmes and is committed to evaluating
all its activities, including legislative and other non-financial interventions[4].
Whilst there is often a fairly consistent and standard model that applies to
the planning and timing of spending programme evaluations, the practice for
evaluating regulatory and other policy actions is more varied. The Commission will
from now on fulfil its commitment to "evaluate first"[5] and systematically
ensure that all
significant proposals for a revision are backed up by a robust evaluation of
the performance of existing EU action.  
Fitness
Checks[6] are
comprehensive policy evaluations covering more than one piece of legislation. 
They assess whether the regulatory framework for a policy area is "fit for
purpose" and if not, what should be changed. Pilot Fitness Checks have
been conducted, testing the concept and providing valuable insights for future
development. On this basis, new fitness-checks have been designed and launched
in various Commission services. The REFIT Communication[7] indicates
further Fitness Check planning.
The 2012 report of the Impact Assessment Board (IAB)[8] noted that
"a significant number of impact assessments did not include the results
from ex post evaluations".  The Commission intends to make full use in
prospective evaluations (impact assessments) of available retrospective
evaluation results. The IAB committed itself to verify that such evaluation has
been conducted and that its conclusions have been used in any related impact
assessment. It also encouraged further strengthening of the identification
within impact assessments of future evaluation and monitoring requirements. The
Commission will follow-up on impact assessments in its subsequent evaluation of
the performance of the actions adopted, thereby completing the evaluation and
assessment cycle.
2.2.  
The current organisation
Across the
Commission the structures are in place to deliver evaluation. The Commission's
organisational framework for evaluation is decentralised. Individual
Directorates General are responsible for the evaluation of their activities and
can structure their evaluation functions to reflect the Directorate General’s
needs and requirements. Evaluation functions co-ordinate and support a Directorate
General's evaluation activities and promote the well-founded and objective
character of evaluation. The Secretariat-General is responsible for supporting
measures, including the provision of guidelines[9], and for ensuring
that the evaluation standards[10] are respected.
The vast majority of evaluations are out-sourced i.e. the
Commission hires consultants to collect and analyse the relevant data and
answer a set of "evaluation questions".  Individual evaluations are
followed by steering groups, which provide guidance and support to the
evaluations and take part in ensuring their quality.
The Commission's evaluation standards are comprehensive and are
similar to those applied by other comparable international organisations. They
cover the full evaluation process: from resourcing, organisation and planning
of evaluation activities to design, conduct and ultimately the dissemination
and use of evaluations. They commit the Commission to conducting high quality
evaluations, contributing both to better regulation and better integration of
evaluation findings in the Commission’s Strategic Planning and Programming
Cycle.
3.      The drivers for change
Current
evaluation practice should be improved for the following reasons:
3.1.  
Embedding an "evaluate first" culture and improving
quality
There can be a tendency to look forward and focus on new
initiatives. But changes are costly and take time to implement – so they need
to be justified and greater attention needs to be paid to looking back before
moving forward. There is a need to confirm the place of evaluation in the
framework for EU action, to commit the appropriate resources and make sure that
evaluations are conducted before proposing further action or change.  
The link between evaluation and impact assessment needs
strengthening. This should be a continuous loop: a good evaluation should be
influenced by the quality of the preparation which went into an intervention
(i.e. the impact assessment); good impact assessments should draw on the
lessons learnt from evaluations, which should identify problems, deficiencies,
challenges and successes.  
Although evaluations often present the state of play, they do not
always include sufficient analysis of why something has happened, whether some
of the change can be attributed to EU action and the extent to which the change
fulfilled all initial expectations. Despite the commitment to evidence-based
policy making, evaluations are not always supported by sufficient data and stakeholder
opinion. The Commission often depends on data to be provided by Member States. Greater
attention also needs to be paid to the limits of the evaluation process itself
and how that affects the ability to draw sound conclusions. More consideration
could also be given to the risks faced and their possible impact on the overall
success of the EU action.
Evaluation often starts with a presumption that EU action will
continue and that changes will be made. Some evaluations don't focus enough on
the problems and difficulties encountered – which, in a way, are more important
than promoting what is working. The main focus of an evaluation should be a
critical drive to improve performance, which means looking both for the bad
practices and the good, providing robust and objective evidence to feed
decisions as to whether EU action should continue as-is, be changed or even
stop. The REFIT screening exercise[11] showed that progress is being made,
but there is room for improvement. More can be done to challenge the status quo
and improve the management of the EU acquis.
3.2.  
Need to update and improve the consistency of approach
The current
evaluation guidelines need to be updated to reflect the latest developments in Smart
Regulation policy and standardise minimum practices across the Commission.
Over time, there have been both consistent and diverging trends in
Directorates General evaluation practices as they have evolved to accommodate
new demands and developments, as well as the nature of different policy areas. This
has led to differences in the planning, scope, method, content and final quality
of evaluations.  Not all EU legislation contains a clear commitment to conduct
future retrospective evaluation, realistically scheduled so that the necessary
data and information should be available. 
There is significant room for improved longer-term planning, more
transparency and greater advance warning and predictability in the nature and
timing of evaluation work.  Planning is a first and critical step in any
evaluation process - poor planning can lead to evaluation results not being
available in a timely manner, reducing the information and evidence available
to policy making. Better advance planning is particularly necessary if
evaluation is to fulfil its' role in the Smart Regulation cycle and provide a
robust and timely starting point for consideration of whether further action is
needed.  Currently, the scope and range of such planning varies between
Commission Services and it is not always clear how priorities are identified. The
link between the performance of existing actions and proposed options for
change is not always clear. Stakeholders and external parties find it difficult
to provide timely inputs.  
It can be difficult to decide when to conduct an evaluation. Often
trade-offs need to be made between when the evaluation results are available
(so that they can feed into the decision making process) and the amount of data
on hand (needed to provide a robust evidence base and conduct analysis). Different
policies take different lengths of time to deliver the desired changes, causing
problems with data availability and making it difficult to standardise when to
evaluate. Such limitations need to be taken into account when evaluation or review
clauses are drafted and evaluations are first planned.
Apart from any available statistical evidence, much of the important
information needed to perform a good evaluation comes from stakeholders and
actors who are (directly and indirectly) affected – whose daily lives may be
altered as a result of EU policies and subsequent Member State actions. These
stakeholders have valuable insights into what is actually happening on the ground
– yet often lack of advance notice makes it difficult for them to provide input
in a timely and well-organised manner.  
3.3.  
Quality assurance and quality assessment
A clear and
common scrutiny process is needed to ensure that the evaluations carried out by
the Commission are conducted to a suitably high standard, adopt
a sufficiently critical approach and are free from bias (independent).
At present, quality assurance is undertaken by the Directorate
General's evaluation function and/or the steering group. Sometimes Directorates
General draw on a wider group of resources, for example, by involving external
stakeholders in steering groups or by involving academics in the quality assurance
of a given evaluation.  
According to the evaluation standards, the steering group jointly produces
(together with the officials managing the evaluation) a quality assessment of
the final evaluation report. This quality assessment should critically assess
the evaluation process undertaken, the information sources used, the analysis
produced and the conclusions drawn. The quality assessment is not always
published, in contrast to the IAB opinions, which are systematically made
available alongside the final impact assessment and any associated Commission
proposal.
4.      Blueprint for the new approach
The
improvements outlined below build on the existing foundations and good
practices, aiming to strengthen evaluation at the Commission. 
4.1.  
Improving consistency and clarity: providing a common definition
and process
The new
guidelines will outline and define what constitutes a robust evaluation as well
as provide pointers to what a good evaluation report might contain. The
appropriate level of (proportionate) analysis is defined based on the policy
importance, the complexity of the EU action and its stage in the policy cycle. 
A major effort will be needed to strategically manage the studies, reviews and
assessments to optimise resource use and ensure that they feed into an  evaluation
framework that provides a timely and relevant input to the decision making
process. Within the Commission, the new definition of evaluation could be: a
critical, evidence-based judgement of whether EU action(s) has met the needs it
aimed to satisfy and actually achieved its expected effects. It will go beyond
an assessment of whether something happened or not, and look at causality –
whether the action taken by a given party altered behaviours and led to the
expected changes and/or any other unintended changes.  
All
evaluations should look at the effectiveness, efficiency, relevance, coherence
and EU added value of EU action or justify why this is not the case.
Although
each evaluation needs to be tailored to meet the needs of a given set of
circumstances, elements of a common process can
still be applied across the wide range of EU actions (for example, by using
standard templates for the key stages of evaluation[12]).

4.2.  
Embedding Fitness Checks[13]
Fitness Checks
are subject to the applicable evaluation standards and guidance. Their aim is
to identify excessive administrative/regulatory burdens, overlaps, gaps,
inconsistencies and/or obsolete measures which may have appeared over time for,
launching a discussion of the effectiveness, efficiency, relevance, coherence
and EU added value of the actions being considered. 
In comparison to the evaluation of an individual action, Fitness Checks
should cover groups of actions that serve similar or overlapping objectives
rather than independent and discrete aspects which have no inter-action or
relationship between them. They should not be used just to fill gaps in
evaluations undertaken before of an individual action and should thus provide a
more comprehensive picture than could be obtained by combining several separate
evaluations of individual actions. 
The pilot exercises have shown that such an investment can be a
cost-effective way of collecting evidence for better policy making. Further use
of Fitness Checks can help to identify the cumulative impacts of EU actions and
the scope for simplification and burden reduction. 
4.3.  
Promoting
an evaluation culture  
A further culture
shift is required to complete the Smart Regulation cycle in practice and ensure
that timely evaluation results are fed into the policy making process. Good
policy must be based on robust evidence and evaluation can make a major contribution
in this respect. The 'evaluate first' principle should be further promoted.  
The necessary resources need to be dedicated to
the organisation of evaluation and to ensure involvement of all levels of
management and greater ownership within a Directorate General of the final
evaluation results. Evaluation needs to become a fully integrated part of the
policy making cycle, with clear recognition being given to the important and
necessary role its findings and conclusions must play when considering the need
for any possible future action.
Irrespective of whether an evaluation has been
conducted by the Commission Services or by an external contractor, to improve
accountability, the Commission Services responsible for the evaluation should publish
their analysis and planned follow-up. 
4.4.  
Planning
Building on
the REFIT mapping, the planning process should be streamlined and all
Directorates General should produce indicative rolling evaluation plans covering
a five year period that should be updated annually. The planning should clearly
link up the various commitments already made to evaluate and identify the
degree of evaluation expected within each item on the plan[14].

Evaluation planning and design needs to be carefully considered
and developed at the start of an action or following major change[15]. Since each action has a different
life cycle, better consideration should be given to the implementation stage and
likely data availability at a given point of time, to ensure that evaluations
provide a more timely input to decision making[16]. 
A framework of data monitoring and collection needs to be
established from the start of the action[17], so that
when the time comes to evaluate, the most useful information is readily
accessible. This framework should also indicate where Member State authorities are expected to assist with data collection, monitoring and evaluation. Consideration
also needs to be given to an initial check of the performance of a measure,
with a view to prioritising and scheduling proportionate evaluation dependent
on the initial identification of performance and associated levels of risk. 
In principle, all policy and programme Directorates General should
be conducting at least one evaluation or one Fitness Check each year.
To improve transparency evaluation planning should be
publicly available on a central website and annexed to the management plan of
the Directorate General. 
4.5.   Getting
the design right: delivering more relevant, robust and rigorous evaluations
Evaluation
design should be sound and realistic about what can be evaluated at a given
point in time and an approach selected which maximises the use of readily
available evidence. The scope and purpose of the evaluation should be clearly
defined at the beginning of the process, setting the objectives of the
evaluation and confirming that the five key evaluation criteria (effectiveness,
efficiency, relevance, coherence, EU added value) will be analysed or,
exceptionally, explaining why this is not the case.  Consideration should also
be given as to how best to analyse other key elements relevant to the given
policy area or activity, such as competitiveness (including international
competitiveness) and implementation costs, bearing in mind possible
methodological and data constraints.
From the start of the process it should be clear how the
evaluation will be conducted and put to use. Such planning should be publicly summarised
in an evaluation mandate making the intended scope, purpose and timetable of an
evaluation more transparent and accessible to all interested parties at all
steps of the process. Where external contractors are involved, the Terms of
Reference for their work should be published centrally on the same website as
the evaluation mandate. This should make the evaluation process more
transparent and accessible, facilitating timely contributions from
stakeholders.
Evaluation results and preliminary assessment need to be
communicated to all stakeholders and actively discussed and debated. The
Commission Services in charge of the evaluation should identify their follow-up
actions within six months of the completion of the final evaluation report. As
useful, regular progress updates of the follow up should be provided to senior
management.  
Within an evaluation, consultation should comply with the Commission's
minimum standards for consultation. Respecting these standards, the actual
timing and method of consultation should be defined on a case by case basis,
reflecting the need for proportionality.   
Greater focus needs to be placed on preparations for the implementation
in Member States of actions adopted at the EU level. Just as the Commission
publishes implementation plans to assist and support Member States in their
legal transposition process, so too should more attention be given to the early
preparation of monitoring and evaluation frameworks based on the adopted
legislation or action. This should be facilitated by the work already done at
the impact assessment stage. In the future, each Commission proposal for any
significant change should be accompanied, in the implementation plan, by an
evaluation framework containing the objectives and indicators for the action
and proposed programming for the monitoring and evaluation of the performance
of the action. 
4.6.  
Building in quality and developing a range of scrutiny mechanisms
The role of
the steering group should further improve the process and content of
evaluations. By overseeing a minimum set of mandatory tasks, the steering group
should make a strong contribution to the quality of an evaluation. It should also
play a strong role in ensuring robust evaluation – bringing together technical
and operational experts and other parties from related policy, representing a
wide range of opinions and influences. It should contribute to the quality assessment.

As a minimum the final report should be published and information
on completed evaluations made accessible through the EU Bookshop, alongside the
evaluation mandate and quality assessment. This wider
dissemination should make it easier for all interested parties to scrutinise
the full process, from start to finish. 
Other initiatives which might contribute to improving the quality of
evaluation results could be undertaken on an ad hoc basis. Suggestions include:
the use of academic panels; an annual review of a (random) selection of
completed evaluations to identify good practices and general improvements;
presentation of evaluations in Council working groups and Parliamentary
committees etc.
The Commission will continue to look for further ways to
collaborate with stakeholders and national/regional governments, seeking ways
to share experiences and learn from different evaluation systems. In particular,
the High Level Group on Better Regulation is assisting the Commission by
looking for pilot opportunities to develop and test approaches to "joint
evaluations" with Member States and better ways to share available data
and good practice.
5.      A joint responsibility
Evaluation
is not just a Commission responsibility. Member States require evidence based
policy making and as such, must also commit to playing their part, particularly
in providing the necessary data.  They should also do more to evaluate and
share their experiences of implementing EU legislation, or contributing to EU actions.
In this context, the Commission will strive to ensure that its proposed
monitoring requirements are as efficient and streamlined as possible,
minimising any administrative burden.
Better Regulation is anchored in the inter-institutional
agreements between the European Commission, the European Parliament and the
Council. The 2003 Inter-institutional Agreement on Better Law-Making[18] establishes a global strategy for better
law making throughout the entire EU legislative process. By further improving
the co-operation between the three institutions and working closely together to
ensure that the evaluation and monitoring requirements adopted are clear,
robust and efficient, fully corresponding to the objectives of the final
proposal, the three institutions could significantly improve the evidence base
available to decision makers. The Commission encourages the other institutions
to actively discuss evaluation results in Council working groups and Parliamentary
committees and promote an inter-institutional evaluation culture throughout the
complete cycle.
6.      The future of evaluation at the Commission
The European Commission is committed to
providing a proportionate and reliable evaluation system as part of its Smart
Regulation cycle. Robust and useful analysis, critically judging the outcomes
of EU intervention is essential. Learning from past experience, recognising the
most efficient and effective ways of operating and developing a flexible and
proportionate approach to analysing such actions will improve accountability
and make EU intervention stronger, matching actions to priority needs and
delivering the necessary high quality results.
[1] Throughout this text the words intervention and action are used as
umbrella terms to describe the wide range of different EU actions including:
expenditure and non-expenditure measures, legislation, trade agreements etc. 
[2] COM(2012) 746
final
[3] Summary of the Responses to the 2012 Stakeholder
Consultation on Smart Regulation in the EU, available at: http://ec.europa.eu/governance/better_regulation/smart_regulation/docs/sr_consultation_outcome2012_en.pdf
[4] See
Commissions Internal Control Standard 14 available at: http://ec.europa.eu/budget/library/biblio/documents/control/sec_2007_1341_annexes_en.pdf  Commission's
Financial Regulation and its implementing rules available at: http://ec.europa.eu/budget/biblio/documents/regulations/regulations_en.cfm 
[5] COM (2010)
543 final 
[6] Within this
text, wherever an evaluation is mentioned, the concept of a Fitness Check could
equally apply.  The Fitness Check concept was introduced in COM (2010) 543
final.
[7] Available at: http://ec.europa.eu/smart-regulation/better_regulation/key_docs_en.htm
[8] Available at http://ec.europa.eu/governance/impact/key_docs/docs/iab_report_2012_en_final.pdf
[9] Evaluating EU Activities: A
practical guide for the Commission Services (2004)
[10] Originally published in
Annex A of the 2004 guidelines and updated in the Commission Communication
"Responding to Strategic Needs: Reinforcing the use of evaluation" –
SEC(2007)213   
[11]  Initial
results are presented in SWD (2013) 401 final.
[12] These could
include the evaluation mandate, evaluation final report, and the quality
assessment of the evaluation final report.
[13] To date, 3 pilot Fitness
Checks have been completed by the Commission and a further 2 are on-going. The
REFIT Communication envisages more being created as this tool becomes an
established part of Commission evaluation policy.
[14] The Commission
plans to provide an outlook on the context, timing and organisation of the
reporting and evaluation work to be carried out from next year on the spending
programmes in this year's report under Article 318 of the Treaty on the
Functioning of the EU.
[15] The final
chapter of an impact assessment sets out the proposed future evaluation and
monitoring arrangements.
[16]  A point also
made in the 2011 IAB report.
[17]
Normally, such arrangements should be identified during the impact assessment
process.
[18] OJ C 321, 31.12.2003,
p. 1.