Quality of public administration : a toolbox for practitioners. Theme 1, Policy-making, implementation and innovation.
Date: 2017-11-29
Institutions: Directorate-General for Employment, Social Affairs and Inclusion
Subjects: administrative structures, fund (EU), public administration, public institution, public policy, public service, social well-being, socioeconomic conditions, user guide

Quality of Public Administration
A Toolbox for Practitioners

Theme 1: Policy-making, implementation and innovation

Governments  will  always  face  difficult  policy  choices,  even  in  times  of  peace  and  prosperity,  and  will  be  judged  on 
the outcomes they produce. Policy decision taken at all levels (supra-national, national, regional and local) will shape 
the strength of economic renewal and social well-being in the EU in the coming years. This theme poses a series of 
questions:  How  is  policy  designed  in  practice?  What  instruments  are  available?  How  can  more  creative  solutions  be 
found? In seeking answers, it explores the qualities of good policy-making, approaches to longer-term strategic planning, 
stakeholder  consultation  and  the  advent  of  co-responsibility  with  citizens  and  businesses  (co-design,  co-production, 
co-evaluation, etc.). It also sets out tools and techniques for regulatory reform. In pursuing continuous improvement, it 
emphasises the importance of systematic feedback, the value of external scrutiny in driving up standards, and innovation 
in its myriad forms.

Social 
Europe

The Quality of Public Administration “Toolbox” is developed on behalf of the European Commission’s Inter-service 
group on Public Administration Quality and Innovation. 

It is based on a joint effort by the following Commission’s services: BUDG, CNECT, DEVCO, DIGIT, ECFIN, EMPL, 
EUROSTAT, GROW, HOME, HR, JRC, JUST, NEAR, REGIO, RTD, SG, SRSS and TAXUD.

Editor: Florian Hauser, European Commission, Directorate General for Employment, Social Affairs and Inclusion 

The publication was produced with the technical assistance of the European Institute of Public Administration (EIPA) 
and Mackie O’ Sullivan Consulting Ltd.

Manuscript completed in August 2017

Neither the European Commission nor any person acting on behalf of the Commission is responsible for the use 
that might be made of the following information.

Luxembourg: Publications Office of the European Union, 2017

© European Union, 2017 

Reuse is authorised provided the source is acknowledged. 
The reuse policy of European Commission documents is regulated by Decision 2011/833/EU (OJ L 330, 14.12.2011, p. 39).

For any use or reproduction of photos or other material that is not under the EU copyright, permission must be 
sought directly from the copyright holders.

© Cover photo: Shutterstock

PDF 

doi:10.2767/879305 

  KE-04-17-933-EN-N

Contents 

Introduction ............................................................................................................................. 2 

1.1 

Qualities of good policy-making ..................................................................................... 4 

1.1.1 

Policy design .................................................................................................................. 11 

1.1.2 

Data insights to help solve policy problems ................................................................. 16 

1.1.3 

Forward thinking ........................................................................................................... 34 

1.1.4 

User-centred consultation and co-responsibility (co-creation) .................................... 39 

1.2 

Instruments of policy implementation ......................................................................... 51 

1.2.1 

Laws and the regulatory environment .......................................................................... 53 

1.2.2 

Achieving outcomes by changing behaviour ................................................................ 69 

1.2.3 

Co-production ............................................................................................................... 82 

1.3 

Continuous improvement and innovation .................................................................... 87 

1.3.1 

Monitoring, evaluation and performance audit ........................................................... 87 

1.3.2 

Encouraging external scrutiny..................................................................................... 101 

1.3.3 

Fostering innovation ................................................................................................... 103 

1.4 

Conclusions, key messages and inspiration for future action....................................... 114 

Contents 

 
 
 
 
Quality of Public Administration - A Toolbox for Practitioners 

2 

Good governance is all about policy choices: how administrations make best use of the powers and 
resources entrusted to them by the public, and manage their relationships with citizens, businesses 
and other stakeholders. This chapter: 
  Sets out the rudiments of policy design and decision-making, including the role of the evidence 
base and ‘big data’ analytics, and describes techniques to strength-test the policy process;  
  Explores  how  governments  are  looking  beyond  immediate  policy  pressures,  and  envisioning 

sustainable socio-economic development over long-term horizons; 

  Explains  the  ways  that  administrations  are  connecting  with  citizens  and  businesses  through 

consultation and co-responsibility; 

  Examines the challenges in policy delivery - managing spending within a sound fiscal framework, 
creating a regulatory  framework  that is conducive to growth,  choosing  the best  organisational 
mode (‘make or buy’), providing information, and co-producing with businesses and citizens; 
  Elaborates the growing influence of behavioural insights in policy design and implementation; 
  Recognises  the  pressures  for  continuous  improvement  in  public  policy,  and  reviews  the 

contributions of evaluation, external scrutiny and innovation. 

Introduction 

“Policy-making is the process by which governments translate their political vision into programmes 
and  actions  to  deliver  'outcomes'  -  desired  change  in  the  real  world”.  A  Practical  Guide  to  Policy 
Making, Northern Ireland Executive, 2016. 

Every  public  official  has  a  concept  of  what  ‘policy’  means  in  his  or  her  field,  but  a  quick  scan  of 
government  guidance  and  independent  research  globally  finds  no  precise  and  universally  agreed 
definition, except  perhaps  the  dictionary  consensus:  a definite  course of action.  Asked to describe 
policy in a specific domain, an official might see it as: 

  The government’s intentions (“as the minister / mayor said …”); 

  The administration’s actions (“a change in the law is being put to the assembly”, “we have 
just launched a new programme”, “the government is planning to decentralise delivery”, “the 
agency’s functions will be put out to tender”, etc.); 

  A review of alternative options (“you’ll find the policy document on our website with details 

of the government’s analysis and its proposals for discussion”). 

Rather than attempt a comprehensive definition, this Toolbox focuses instead on the characteristics 
of policy in 5Ds. Every policy should be a clear statement of direction. It should be the product of a 
robust assessment and hence deliberation over the pros and cons of prospective solutions, to enable 
a  decision  on  the  best  way  forward.  Policy  sets  out  a  course  of  action,  so  must  lead  to  delivery, 
otherwise statements of intent are just warm words. Policy-making should also be dynamic, taking 
account of changing circumstances, and flexible enough to adapt to experience and events.  

The  Toolbox  also  distinguishes  between  the  concepts  of  policy  and  strategy  (especially  as  not  all 
policies are accompanied by strategies). The direction set out in the policy might be elaborated in a 
strategy,  describing  how  resources  are  marshalled  to  achieve  the  government’s  objectives.  Policy-

Introduction 

 
 
 
 
 
 
 
 
Quality of Public Administration - A Toolbox for Practitioners 

3 

making is deciding on a definite ‘path’ to be pursued, the strategy is the ‘road map’ for getting there. 
This chapter considers how high-level objectives are articulated in policy-oriented strategies.1  

This  chapter  explores  three  fundamental  aspects of policy-making  -  designing  policy,  choosing  the 
instruments of implementation, and seeking continuous improvement - and the ways and means to 
achieve them: 

Key questions 
1.1 How is policy designed? What and who informs 
decision-making? How can governments move from 
reactive and ad hoc policy decisions to more 
reflective, long-term planning?  

1.2 What instruments are available to policy-makers 
to achieve their policy goals? What are their relative 
merits? How best should they be implemented? 

1.3 How does the administration know if the policy 
has been achieved? How can the administration 
strive for still-better performance and more 
creative solutions to established and emerging 
problems? 

Ways and tools 

  Policy fundamentals 

Insights from data analytics and visualisation 

  Forward thinking 
  Strategy preparation 
  Consultation and co-design 
  Public spending (see topic 8.1), including public 
service delivery (see theme 5) and procurement 
(see topic 8.2) and use of EU funds (see topic 8.3) 

  Laws and the regulatory framework 
  Soft policy Tools (see Better Regulation Toolbox) 
  Reforms to government structures (see theme 3) 
  Relevant Information 
  Applying behavioural insights 
  Co-production 
  Monitoring and evaluation (including co-

evaluation) 

  Performance audits 
  External scrutiny 
  Public sector innovation 
  Trust Building 

The  policy  choices  taken  by  governments  at all  levels  (supra-national,  national,  regional  and  local) 
will  shape  the  strength  of  economic  renewal  in  the  EU  in  the  coming  years  and  the  success  in 
attaining  Europe  2020  goals  of  smart,  inclusive  and  sustainable  growth.  This  is  particularly  true of 
the effect of policy decisions on public administrations. Public policy determines whether the most 
suitable  organisational  structures  (theme  3)  and  staffing  (theme  4)  are  put  in  place,  whether  the 
delivery of public services meets needs and expectations (theme 5), whether businesses are helped 
or  hindered  in  delivering  economic  prosperity  (theme  6),  whether  the  judiciary  can  operate 
independently and to the highest standards of quality and efficiency (theme 7), and whether scarce 
public  resources  are  managed  prudently  (theme  8).  Ethics  are  integral  to  good  policy-making,  but 
policy  can  also  dictate  the  extent  to  which  integrity  is  integrated  into  the  functioning  of  the 
administration itself (theme 2).  

Ultimately, every government at whatever level is judged by its policy choices and their outcomes, 
which  places  a  high  premium  on  strengthening  policy-making  as  a  process,  to try  and  achieve  the 
desired results. 

1 See theme 4 for how operational goals are translated into organisational strategies. 

Introduction 

 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

4 

1.1  Qualities of good policy-making 

is  a 

Policy design 

Forward thinking 

Analysing big data to solve policy problems 

Policy-making 
is  usually  described  as  a 
cyclical  process,  from  problem  identification 
to  programme  evaluation,  which  in  turn 
informs the next round of policy design. But 
the lack of a consensus definition reflects the 
reality  that  policy-making 
flexible 
concept,  which  in  practice  does  not  follow 
rigid  rules.  Decision-making  should  be  underpinned  by  certain  principles,  however,  that  can  be 
applied  whatever  the  context.  While  unexpected  events  mean  that  policy-making  is  sometimes 
sporadic  and  reactive  (policy  ‘on  the  hoof’),  governments  also  face  demographic,  economic  and 
environmental challenges which extend beyond the short term, and often demand pan-European or 
global solutions. In this light, many public administrations are finding the time and space for forward 
policy planning over medium-long term horizons, covering more than one electoral cycle. They are 
also increasingly looking to actively involve citizens and businesses in policy-making, rather than as 
the passive recipients of policy decisions. 

Consultation and co-responsibility 

The  ‘policy  cycle’  is  a  well-established  concept,  which  is  typically  taught  as  the  rational  model  of 
policy decision-making. While some version set out more or fewer stages to the process than others, 
and the terminology and may vary, the basic sequence of stages follows a common pattern:  

  A  problem  is  identified,  and  the  underlying  causes  and  needs  are  analysed,  to  determine 
whether there is a rationale for public policy intervention, for example due to market failure 
or government. 

  A policy response is formulated, based on setting out several scenarios that will resolve the 
problem  and  satisfy  the  needs  and  expectations  of  the  affected  parties  (whether  citizens, 
businesses, public institutions, etc.), and an options appraisal is performed that weighs the 
pros and cons of alternative approaches, usually based on a cost-benefit analysis. 

  The  preferred  solution  is  selected,  usually  at  the  political  or  senior  management  level, 
following  consultation  with  interested  stakeholders  that  will  be  directly  affected  by  the 
outcome, wherever possible. 

  The policy is implemented as agreed and subjected to monitoring, as a management tool to 
track  performance  and  measure  progress  against  the  plan,  including  any  deviations  or 
unforeseen outcomes.  

  Finally,  the  policy  is  evaluated,  to  determine  whether  it  has  been  successful  in  addressing 
the original problem and meeting the needs of affected parties. If so, the evaluation seeks to 
draw out learning points for future interventions, and if not, the evaluation notes whether 
the original objective has been overtaken by subsequent developments or recommends an 
alternative course of action, thereby feeding back into policy design. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

5 

This  traditional  policy-making  model  is  presented  below.  It  is  widely  recognised  that  this  is  an 
idealised view of the policy process, and that the model is intended to be illustrative. 

In practice, the reality is usually more complicated and sometimes chaotic too:  

  Administrations rarely start with a blank sheet of paper. The initiative for policy design can 
come from a variety of sources: political commitments made at election time, the priorities 
of  individual  elected  officials  (ministers,  mayors,  etc.),  obligations  from  EU  directives  and 
international  treaties,  public  pressure,  emerging  crises,  new  approaches  to  old  policy 
problems,  lobbying  by  think  tanks  and  associations,  and  many  more.  This  ‘frames’  the 
problem  identification  and  policy  formulation  within  a  pre-existing  set  of  ideas  and 
proposals. If allowed, the administration might wish to challenge these assumptions, in the 
interests of  policy  rigour, but  in  any case,  they  represent  the  initial  parameters  for  policy-
making most of the time.   

  The  conventional  model  suggests  that  policy-making  is  a  linear,  sequential,  end-to-end 
process, and that administrations have sufficient time to conduct each phase and reflect on 
the outcome before proceeding to the next. In practice,  the stages in the ‘cycle’ are inter-
dependent,  can  happen  simultaneously  and  often  cannot  be  separated  from  each  other. 
Elected  officials  at  any  level  (supra-national,  national,  regional  or  local)  may  require  or 
request  policy  advice  which  is  all-encompassing  and  all-at-once:  immediate  solutions  to 
current  problems,  including  scenarios,  a  recommendation  on  the  best  way  forward,  and  a 
proposal  for  how  the  policy  will  be  delivered,  including  budget  and  responsible  body.  The 
policy  decision  may  involve  several  iterations,  with  goals,  potential  actions  and  preferred 
option all evolving, often together, as new inputs or information are sought.  

  Policy-makers  can  reach  decisions  without  being  able  to  consider  all  available  options 
thoroughly,  either  because  of  limited  information  or  time  constraints.  In  many  cases,  the 
solution  is  announced  based  on  political  expediency  or  parliamentary  timetables,  rather 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

6 

than objective evidence. In the real world of unforeseen events and 24-hour news, decision-
makers  must  sometimes  make  policy  pronouncements  quickly  in  response  to  emerging 
situations. 

  The chain of supposed events in the policy cycle is easily broken, especially when there is a 
change  of  government.  Even  when  the  government  is  stable,  elected  officials  may  resign, 
retire  or  be  replaced  before  a  policy  is  fully  formulated,  implemented  or  evaluated.  Their 
successors may wish to change the direction of policy. Evaluation is often the poor relation 
of the  policy-making process,  either being neglected completely, or the findings arrive too 
late to influence changes in policy design. 

  Monitoring is more common, but not necessarily systematic (using baselines, indicators and 
benchmarks).  Elected  officials  will  often  know  when  a  policy  is  not  performing  anyway 
through the less formal channels of public opinion, critical media and business lobbying. At 
this  point,  any  stage  of  the  policy-making  process  –  goals,  delivery  options,  actual 
implementation – may need to be adjusted or even abandoned.  

For  these  reasons,  the  policy-making 
process  can  be  thorough  or  flawed, 
and  all  points  in  between.  Even  the 
best  of  intentions  can  become  ‘bad 
policy’ at the point of implementation, 
with  unexpected  and  unfortunate 
policy-making 
The 
consequences. 
process will never be an exact science, 
as  the  environment  is  ever  changing. 
public 
dynamic  world, 
In 
administrations  face  difficult  choices, 
must 
through 
a 
complicated  scenarios,  and  manage 
uncertainties  created 
in  complex 
situations (see diagram right).2  

steer 

path 

a 

Complicated scenarios - 
due to problem size, scale 
or scope, or process that 
is intricate and risky (e.g. 
international  
negotiations) 

Difficult choices - due to 
conflicting views and 
interests, lack of 
information, resources, 
capacity, political will or 
time (e.g. deficit 
reduction) 

Complex situations  - due 
to broad dispersion of 
power, high degree of 
interdependence and 
unpredictability  and some 
emergent characteristics 
(e.g  climate change) 

Good 
governance 

Public policy-making presents a dilemma to public administrations. On the one hand, governments 
should  strive  for  ‘good  policy’  that  satisfies  societal  needs  and  expectations,  which  suggests  a 
rigorous  process  that  involves  planning  and  resources.  On  the  other  hand,  the  environment  for 
policy is continually evolving, which forces decision-makers to be responsive to changing conditions 
and nimble in reacting to events.  

Policy  is  prone  to  exogenous  factors  and  its  effects  are  never  entirely  predictable,  which  puts  a 
premium on ‘adopt-and-adapt’. This iterative approach is what Professor John Kay3 has described as 
“a process of experiment and discovery. Successes and failures, and the expansion of knowledge, lead 
to  reassessment  of  our  objectives  and  goals  and  the  actions  that  result.”  Administrations  should 

2 Source: J. Bourgon with P. Milley (2010), The New Frontiers of Public Administration. 
3 J. Kay (2010), Obliquity: Why Our Goals Are Best Achieved Indirectly. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

7 

focus on high-level objectives and always keep them in sight, but remain ready to respond to events 
as  they  arise,  and  willing  to  adjust  short-term  operational  goals  and  activities  accordingly.  Policy-
making should be versatile. 

As  a  result  of  “global  tendencies  in  politics,  economy,  development  of  technologies,  migration  and 
demography, society and civic participation”, the European Public Administration Network (EUPAN) 
has described the increasingly dynamic environment for public administration in Europe in its first 
Strategy Paper as characterised by four elements, summarised as ‘VUCA’: 

  Volatility: the dynamics, scale and pace of change; 

  Uncertainty: the lack of predictability, the fact that part of future events will be surprising; 

  Complexity: the varied forces and factors that affect the operation of organisations; 

  Ambiguity:  the  difficulties  of  precisely  determining  the  connection  between  causes  and 

consequences, and hence the risk of erroneously interpreting events.   

“We have, as a rule, only the vaguest idea of any but the most direct consequences of our acts … By 
"uncertain" knowledge, let me explain, I do not mean merely to distinguish what is known for certain 
from what is only probable. The game of roulette is not subject, in this sense, to uncertainty … Even 
the weather is only moderately uncertain. The sense in which I am using the term is that in which the 
prospect of a European war is uncertain, or the price of copper and the rate of interest twenty years 
hence, or the obsolescence of a new invention … About these matters there is no scientific basis on 
which  to  form  any  calculable  probability  whatever.  We  simply  do  not  know.  Nevertheless,  the 
necessity  for  action  and  for  decision  compels  us  as  practical  men  to  do  our  best  to  overlook  this 
awkward fact”. John Maynard Keynes, The General Theory of Employment, 1937. 

The  growing  recognition  that  plans  do  not  follow  a  predictable  path  has  sparked  an  increasing 
interest in systems thinking in public administration.4 With its origins in the early 20th century and 
modern  applications  in  many  disciplines5,  the  main  value  of  systems  approaches  is  the  underlying 
philosophy  of  ‘seeing  the  bigger  picture’.  Systems  exist  in  many  forms,  e.g.  transportation, 
healthcare,  the  weather,  the  economy,  even  public  administrations  themselves.  The  connections 
and  interactions  within  systems  and  with  the  external  environment  (including  other  systems)  are 
vital  to  how  they  function,  which  means  that  focusing  on  individual  components  –  the  literal 
meaning of analysis6 – can produce misleading conclusions and lead to unintended consequences.  

4 A system can be defined as a set of interacting or interdependent parts forming a unitary (and typically complex) whole 
with a purpose. The system concept is perhaps best captured in the phrase ‘greater than the sum of its parts’. 
5 These include biology/medicine, metrology, economics/business, engineering, transportation, computing, cybernetics, 
communications, sociology and behavioural sciences. 
6 From the original Ancient Greek, meaning ‘loosening-up’, analysis can be defined as the division of the whole into its 
constituent parts for individual study (see http://www.dictionary.com/browse/analysis), and the opposite of synthesis. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

8 

“Traditionally, public policy makers have addressed social problems through discrete interventions 
that are layered on top of one another. However, these may shift the consequences from one part 
of  the  system  to  another,  or  address  symptoms  while  ignoring  causes  …  Looking  at  the  whole 
system rather than the parts allows one to focus on where change can have the greatest impact”.  
OECD, Working with Change: Systems approaches to public sector challenges, 2017. 

Systems  thinking  starts  with  acknowledging  that  a  system  exists  -  and  that  any  changes  to  one 
constituent part might change the system and/or its behaviour, but there will still be a ‘system’. It 
takes a holistic viewpoint, avoiding the error of seeing elements in isolation. It sees cause and effect 
as  not  a  straightforward  linear  process  (A  >  B  >  C  >  D),  due  to  the  dynamics  within  systems,  the 
interplay of elements and the impact of feedback (e.g. D on B), both positive and negative. Certain 
tools  are  available  to  visualise  systems  approaches,  such  as  causal  loop  diagrams,  along  with 
techniques such as framing, connecting, dwelling, prototyping, stewarding and evaluating.7 Systems 
thinking has implications for the quality of public organisations (see theme 4), and effective service 
delivery  (see  theme  5),  as  well  as  making  positive  change  happen  in  public  administration  reform 
(see theme 9). 

Most  policy  decisions  are  not  purely  technical,  however,  based  on  the  considered  merits  of  one 
course  of  action  over  another  with  respect  to  likely  outcomes.  As  the  US  Academy  of  Science 
observes, the decision-making process also involves: 

  Political considerations, insofar as policy choices influence who has and retains power; and  

  Normative considerations regarding the desirability (or undesirability) of a proposed action, 

value judgments, and considerations of legitimacy.   

Public  policy-making  is  inseparable  from  the  democratic  mandate,  so  policy  development  should 
reflect the relationship between the political priorities of elected officials (national, regional or local) 
and the ‘wise counsel’ of appointed officials (civil servants). Given the importance of evidence-based 
policy,  this  requires  clear  strategic  direction  and  leadership  from  politicians  to  be  married  to 
balanced and professional advice from the administration. 

Moreover,  as  systems  thinking  suggests,  the  policies  themselves  interact,  meaning  their  effects 
cannot be isolated from each other in assessing the best way forward. 

“Policy arguments generally constitute a package of considerations backed by reasons presented to 
persuade particular audiences of the validity of and need for a given action. The arguments consider 
not just the policy choice at hand, but how that policy interacts over time with many other policies—
does  opening  a  school  in  the  community  decrease  or  increase  housing  prices;  do  housing  prices 
affect  the  local  labor  supply;  does  the  labor  supply  affect  whether  a  chain  store  locates  in  the 
community?  Obviously,  it  is  a  complex  undertaking  to  sort  out  how  the multiple  characteristics  of 
policy  argument  function  together  to  yield  a  coherent,  valid,  and  persuasive  argument”.  US 
Academy of Sciences.8 

7 See OECD (2017), op. cit. 
8 National Research Council (2012), Using Science as Evidence in Public Policy. Committee on the Use of Social Science 
Knowledge in Public Policy, K. Prewitt, T.A. Schwandt, and M.L. Straf, Editors. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

9 

An  alternative  approach  to  the  ‘policy  cycle’  model,  which  reflects  reality  more  closely,  is  to 
concentrate  on  creating  the  right  culture,  working  environment  and  organisational  structures  for 
sound  policy  design.  This  approach  relies  on  applying  flexibly  the  qualities  of  policy-making.  The 
UK’s  Institute  for  Government  has  set  out  a  vision  of  the  policy  process  which  takes  ‘policy 
fundamentals’ as its building blocks, and the sequence in which they are assembled is a secondary 
concern.  

Policy fundamentals 

Many existing models of policy making are increasingly inappropriate in a world of decentralised services and 
complex  policy  problems.  In  the  face  of  these  challenges,  we  need  to  give  a  more  realistic  account  of  what 
good policy making should look like – and then ensure the surrounding system increases its resilience to the 
inevitable  pressures  to  depart  from  good  practice.  The  starting  point  is  our  analysis  that  there  are  certain 
fundamentals of good policy making which need to be observed at some point in the policy process: 

  Clarity on goals; 
  Open and evidence-based idea generation; 
  Rigorous policy design; 
  Responsive external engagement; 
  Thorough appraisal; 
  Clarity on the role of central government and accountabilities; 
  Establishment of effective mechanisms for feedback and evaluation. 

The fundamentals draw on elements of current policy making models, but place additional emphasis on policy 
design  and  clear  roles  and  accountabilities.  They  need  to  be  seen  alongside  the  need  to  ensure  long-term 
affordability  and  effective  prioritisation  of  policy  goals.  Overlaying  these  criteria  has  to  be  a  decision  on 
resources and resource availability. Individual policies have to be affordable over their life time and represent 
good long-term value for money. 

By policy design, we mean the stage in the process which turns policy ideas into implementable actions. Policy 
design is a fundamental yet under-developed part of the policy process. Many ideas which look good on paper 
are not feasible to implement – and it is often too late to change course when the legislation is on the statute 
book and political capital has been expended. Those failures can come from multiple causes, but one recurrent 
theme is the failure to understand the likely behaviours of those whose actions the policy is designed to affect. 
Policy  makers  need  to  be  able  to  use  prototypes  and  stress-test  policies  to  ensure  they  are  implementable, 
which will require new partnerships and a greater involvement of service users in policy development. More 
radically, policy makers (and Parliament) will need to move on from the idea that central government creates 
fixed  designs  for  policies,  and  start  creating  designs  that  are  flexible  enough  so  others  can  adapt  them  to 
changing circumstances. 

Source: Extracted from M. Hallsworth and J. Rutter, “Making policy better: improving Whitehall’s core 
business”, Institute for Government, UK  

With some minor adjustments to increase transferability to European administrations, the Institute 
for Government’s checklist is a useful mechanism to assess whether all the fundamentals have been 
achieved  during  the  policy  process,  irrespective  of  the  order  they  are  performed.  These  seven 
building blocks are explored in more detail in the topics and tools in the rest of this chapter. 

‘Fundamental’ 

Clear goals 

Key questions 

  Has the issue been adequately defined and properly framed?  
  How will the policy achieve the high-level objectives of the 

government / ministry / municipality? 

References 

Topics 1.1 and 
1.3 

Evidence-based 

  Has the policy process been informed by evidence that is high 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

10 

‘Fundamental’ 

ideas 

quality and up to date?  

Key questions 

References 

  Has account been taken of evaluations of previous policies?  
  Has there been an opportunity or licence for innovative thinking?  
  Have policy-makers sought out and analysed ideas and experience 

from the ‘front line’ or other European administrations? 

  Have policy-makers rigorously tested or assessed whether the 

policy design is realistic, involving implementers and/or end users?  

Rigorous design 

  Have the policy-makers addressed common implementation 

External 
engagement 

Thorough 
appraisal 

problems?  
Is the design resilient to adaptation by implementers? 

  Have those affected by the policy been engaged in the process?  
  Have policy-makers identified and responded reasonably to their 

views? 

  Have the options been robustly assessed?  
  Are they cost-effective over the appropriate time horizon?  
  Are they resilient to changes in the external environment?   
  Have the risks been identified and weighed fairly against potential 

benefits? 

Topics 1.1.3 
and 1.3.2 

Topic 1.2 

Clear roles and 
accountabilities 

Feedback 
mechanisms 

  Have policy-makers judged the appropriate level of (central) 

government involvement?  
Is it clear who is responsible for what, who will hold them to 
account, and how? 
Is there a realistic plan for obtaining timely feedback on how the 
policy is being realised in practice?  

  Does the policy allow for effective evaluation, even if government is 

Theme 3 

Topic 1.3.1 

not doing it? 

The  New  Synthesis  (NS)  Initiative  has  developed  an  evolving  theoretical  framework  for  helping 
governments  to  face  the  challenges  of  the  time,  whether  the  response  involves  policies, 
programmes, projects, services, structures or systems. The approach focuses on applying a series of 
techniques, summarised below: 

Technique 
Positioning  

Leveraging  

Overview 
The Power of a “Broader Mental Map”: the way we think about the role of government in 
society  and  the  way  we  frame  issues  transform  the  way  we  address  the  issues,  the 
relationship with citizens and the impact of government actions. Positioning recognises that 
public policies, programmes and agencies are instruments to serve a broader public purpose. 
They  are  important  insofar  as  they  move  society  forward  and  contribute  to  better  societal 
results.  Positioning  is  about:  exploring  the  inter-relationship  between  agency,  system-wide 
and societal results; gaining an appreciation of the ripple effects of government actions across 
government  and  across  society;  and  expanding  the  space  of  possibilities  and  the  range  of 
options open to government. Positioning is a pragmatic search for what is feasible with the 
resources and capabilities available at the time. 
The Power of Others: discovering how government can achieve the greatest possible impact 
with  the  least  amount  of  effort  and  economy  of  resources  by  enrolling  the  contribution  of 
others  to  bring  viable  and  sustainable  solutions  to  the  problems  we  face  as  a  society. 
Leveraging  is  about  pooling  capabilities  and  resources  across  multiple  boundaries  and 
interfaces  to  achieve  results  of  higher  public  value  at  a  lower  overall  cost  to  society. 
Government  does  not  need  to  do  it  all  to  serve  the  collective  interest  well.  It  can  achieve 
better results by focusing on what it is best positioned to do while building on the strength of 
others.  Government  actions  form  part  of  long  chains  of  intermediate  results  where  the 
contributions of multiple agents are necessary to achieve the desired public outcome. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

11 

Technique 

Engaging 

Overview 
The Power of People: transforming the relationship between government and citizens from 
one where government is the primary agent responsible for serving the public good to one of 
mutuality and reciprocity. Engagement is about: 

  Exploring what government is best positioned to do, what citizens can do for themselves, 

what can best be accomplished by working together and how it may all fit together;  

  Designing  public  policies,  programmes  and  services  that  give  citizens,  users  and 
beneficiaries an active role in working with public agencies to create and produce public 
results;  

  Creating  an  enabling  environment  that  encourages  and  promotes  self-organisation  and 
self-governing practices by citizens. This means encouraging citizens to work together to 
take charge of addressing issues of concern to them in a manner that also promotes the 
collective interest. 

  Building resilience by encouraging participation, shaping policy responses that reduce the 
risks  of  dependency,  building  trust  and  confidence  in  the  collective  capacity  to  invent 
solutions to the challenges we face as a society. 

Positioning  is  about  framing  the  policy  problem  and  the  response,  so  that  it  looks  beyond  the 
performance of individual organisations (‘agencies’), and lifts sights towards higher-level objectives 
and outcomes: societal results. Leveraging is about breaking down silo thinking, within and beyond 
the  public  administration,  and  seeking  new  ways  to  coordinate  and  cooperate.  Societal  problems 
increasingly  require  an  integrated  approach  that  cuts  across  several  policy  fields,  and  may 
necessitate  inter-agency  programmes  with  a  coordinating  project  leader.  In  the  Netherlands,  for 
example, such initiatives within the public administration can last several years, and thereby justify 
the setting-up of an inter-ministerial programme department. Engaging takes government into the 
often-unfamiliar territory of co-responsibility: transforming the  relationship with citizens to one of 
shared responsibility (see topic 1.1.3). 

1.1.1  Policy design 

The  strength  of  the  evidence  base  is  the 
foundation of successful policy-making, along 
with its interpretation. A solid evidence base 
strengthens  decision-makers’ 
for 
improvisation, 
challenging  pre-conceived 
ideas  and current practices in the  search for 
(more  effective)  policy  solutions.  But  this 
remains the exception, rather than the rule.  

room 

Policy design 

Analysing big data to solve policy problems 

Forward thinking 

Consultation and co-responsibility 

“It’s  fairly  obvious  that  intelligent  use  of  evidence  reduces  the  risks  of  error  and  increases  the 
prospects  of  success.  But  systematic  use  of  evidence  remains  surprisingly  rare  ...  In  most  fields  of 
government action, the evidence is too patchy; or it may show that things work in one context but 
not that they can be spread. Yet it’s vital that everyone from a police officer or head teacher to a 
policy-maker is aware of what is known … Politicians have the right to ignore evidence but not to be 
ignorant  of  it”.  NESTA,  Rewiring  the  Brain,  a  Rough  Blueprint  for  Reforming  Centres  of 
Governments, 2014 

Policy  advisors  should  cast  a  wide  net  when  thinking  about  potential  sources  and  viewpoints, 
including: official statistics; existing studies from in-house, academia, associations, think-tanks, etc.; 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
  
Quality of Public Administration – A Toolbox for Practitioners 

12 

evaluation  findings;  surveys,  panels  and  other  original  research  (if  appropriate  and  affordable); 
expert inputs; and evidence from stakeholders, both interested and affected parties. ICT can play a 
key role in evidence-based policy-making through the use of ‘big data’ (see topic 1.1.2), simulation 
and prototyping. 

One option is to outsource the gathering and assessment of evidence to a dedicated public authority 
with specific expertise in research and analysis, such as the CPB Netherlands Bureau for Economic 
Policy Analysis, which is part of the Ministry of Economic Affairs, but functions independently.  

Inspiring example: CPB Netherlands Bureau for Economic Policy Analysis 

CPB Netherlands Bureau for Economic Policy Analysis (CPB) conducts scientific research aimed at contributing 
to the economic decision-making process of politicians and policymakers. It was founded in 1945 and has been 
a part of the Ministry of Economic Affairs ever since. Its director is appointed by the Minister, in consultation 
with  other  members  of  the  government,  but  CPB  is  fully  independent  as  far  as  the  contents  of  its  work  are 
concerned.  It  also  has  its  own  legal  mandate  and  an  independent  advisory  committee.  Research  at  CPB  is 
carried  out  on CPB’s own initiative, or at the request  of the government, parliament,  individual members of 
parliament, or for example national trade unions or employers’ federations. It is largely publicly financed. To 
ensure its independence, a  maximum of  20% of its annual budget  may originate from external assignments. 
However,  CPB  is  not  allowed  to  compete  with  commercial  research  bureaus,  and  external  assignments  are 
limited to local and national governments, European institutions or international governmental organisations. 

The output for which CPB is best known includes its quarterly economic forecasts of the  development of the 
Dutch  economy.  The  main  forecasts  are  the  Central  Economic  Plan  (CEP),  published  every  spring,  and  the 
Macro  Economic  Outlook  (MEV),  which  is  published  jointly  with  the  Annual  Budget  at  the  Opening  of  the 
Parliamentary Year in September. A special forecast is the Medium-Term Forecast, which is published at the 
start of each election cycle. This forecast differs from the above-mentioned CPB forecasts by covering a four-
year period. It offers a foundation for the development of policy plans by political parties and the negotiations 
for  a  new  government  after  the  general  elections.  From  1986  onwards,  CPB  has  offered  interested  political 
parties an analysis of the economic effects of the policy proposals in their election manifestos. The plans of the 
participating parties are analysed identically, thus offering voters a comprehensive tool for comparison of the 
parties, contributing to the transparency of the election process. After the elections, CPB is often requested to 
analyse all or some of the policy proposals put forward during the negotiations for a new government. These 
analyses use the same methods as those used during the analysis of the election manifestos. 

CPB analyses policy proposals in a number of different ways and also evaluates the effects of policy measures 
that  have  already  been  implemented.  Since  the  early  1950s,  the  bureau  analyses  the  costs  and  benefits  of 
large infrastructural projects. These studies are known in Dutch by the acronym MKBA (Societal Cost Benefits 
Analysis). Examples include the Delta plan, the construction of the East Flevoland polder and  the Betuwelijn 
freight railway. CPB also conducts research into numerous other areas - for example, the economic effects of 
ageing, globalisation, health care, education, the financial crisis, or the regulation of market orders. Such work 
is sometimes co-financed externally—in particular, by Dutch ministries or the European Commission. 

For further information: http://www.cpb.nl/en 

Another  well-known  example  is  the  Productivity  Commission9,  which  is  the  Government  of 
independent  research  and  advisory  body  on  a  range  of  economic,  social  and 
Australia’s 
environmental concerns, with a mandate to help governments make better policies in the long-term 
interest of  the  Australian  community.  In  a  2009  speech,  the  former  Chairman  emphasised  heavily 

9 The Productivity Commission advises on a range of economic, social and environmental issues. Its independence is 
underpinned by an Act of Parliament. Its processes and outputs are open to public scrutiny and are driven by concern for 
the well-being of the whole community. For further information: http://www.pc.gov.au/  

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

13 

the value of the evidence base, and its contribution to avoiding false assumptions and flawed policy 
proposals, including as a check on the validity of the high-level objective. 

Inspiring example: Productivity Commission (Australia) 

“Without evidence, policy makers must fall back on intuition, ideology, or conventional wisdom — or, at best, 
theory alone. And many policy decisions have indeed been made in those ways. But the resulting policies can 
go  seriously  astray,  given  the  complexities  and  interdependencies  in  our  society  and  economy,  and  the 
unpredictability of people’s reactions to change. From the many examples that I could give, a few from recent 
Productivity Commission reviews come readily to mind: 

In our research on the economic implications of Australia’s ageing population, we demonstrated that 
common policy prescriptions to increase immigration, or raise the birth rate, would have little impact 
on  the  demographic  profile  or  its  fiscal  consequences  (indeed,  higher  fertility  would  initially 
exacerbate fiscal pressures). 

  Our report into road and rail infrastructure pricing showed that the presumption that road use was 
systematically subsidised relative to rail was not borne out by the facts (facts that were quite difficult 
to discern). 

In our inquiry into waste management policy, we found that the objective of zero solid waste was not 
only economically costly, but environmentally unsound. 

  Our inquiry into state assistance to industry showed that the bidding wars for investment and major 
events  the  state  governments  engaged  in  generally  constituted  not  only  a  negative  sum  game 
nationally, but in many cases a zero-sum game for the winning state. 

  Our recent study on Australian’s innovation system reaffirmed that, contrary to conventional opinion, 
the  general  tax  concession  for  R&D  mainly  acted  as  a  ‘reward’  for  research  that  firms  would  have 
performed anyway, rather than prompting much additional R&D. 

  Our recent draft report on parental leave, indicated that binary views in relation to whether childcare 
was a good or a bad thing were both wrong, depending on which age group you were looking at, and 
that there were many subtle influences involved. 

Now  I  am  not  saying  that  policy  should  never  proceed  without  rigorous  evidence.  Often  you  can’t  get 
sufficiently  good  evidence,  particularly  when  decisions  must  be  made  quickly.  And  you  can  never  have 
certainty in public policy. All policy effectively is experimentation. But that does not mean flying blind  — we 
still need a good rationale or a good theory. Rationales and theories themselves can be subjected to scrutiny 
and debate and, in a sense, that constitutes a form of evidence that can give some assurance about the likely 
outcomes.  Importantly  though,  all  policy  experiments  need  to  be  monitored  and  evaluated  and,  over  time, 
corrected or terminated if they turn out to be failures. These are things that Governments typically find hard 
to do — particularly the termination part.” 

Source: G. Banks (2009), “Evidence-based policy making: What is it? How do we get it?” (ANU Public Lecture 
Series, presented by ANZS OG, 4 February), Productivity Commission, Canberra.  

Officials may need to draw on fresh thinking to solve often well-established and intractable policy 
dilemmas. In seeking creative solutions, public administrations may need to look beyond their own 
internal know-how experience, and search for answers further afield – from front-line staff, affected 
stakeholders,  other  administrations,  academia  and  think-tanks,  etc.  This  can  create  insecurity,  as 
policy  officials  feel  they  are  either  ceding  responsibility  or  acknowledging  they  don’t  have  all  the 
answers, but it also empowers them by bringing different perspectives and new intelligence to the 
table.  

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

14 

Policy  design  can  embody  innovation  by  being  inventive  (entirely  new  concepts)  or  incremental 
(improving on existing practice). Ultimately, administrations may need to experiment, to find elusive 
routes  to  desired  outcomes,  when  other  ways  have  been  found  lacking,  by  launching  prototype 
actions,  evaluating  their  performance,  jettisoning  some  practices  and  expanding  others  (see  also 
topic 1.3). There are risks with experimentation, however, as the media and public can be critical of 
failure  and  what  are  viewed  as  wasted  public  resources.  This  highlights  the  value  of  shared 
ownership with citizens and businesses by co-opting all interested parties into the decision-making 
process (see also topic 1.1.3).  

One  of  the  best-known  examples  of  putting  this  principle  into  practice  is  Denmark’s  MindLab,  a 
cross-governmental and multi-disciplinary innovation unit which involves citizens and businesses in 
creating new solutions for society. MindLab is both an organisation with its own permanent staff and 
secondments, and a physical space that can provide a neutral location for exercising creativity and 
collaboration. 

Inspiring example: MindLab (Denmark) 

Established in 2002, MindLab is jointly owned by three ministries (Ministry of Business and Growth, Ministry of 
Education,  and  Ministry  of  Employment)  and  one  municipality  (Odense),  and  collaborates  formally  with  the 
Ministry for Economic Affairs and the Interior. MindLab’s mission is to work with its owners to create change 
which generates the desired value for citizens, businesses and society. MindLab is instrumental in helping key 
decision-makers  and  employees  to  view  their  efforts  from  the  outside-in  and  see  them  from  a  citizen’s 
perspective, as a platform for co-creating better ideas. MindLab has three strategic objectives: 

1.  Public sector innovation: MindLab will strengthen the outcomes of public policies through systematic 
insight into the perspective of citizens and businesses, and active involvement of the stakeholders 
which can turn new ideas into practice. 

2.  Change capacity: MindLab will build knowledge about new approaches to public problems. This 
knowledge shall enhance the owners’ competencies to take courageous change initiatives. 

3.  Visibility and legitimacy: MindLab will work actively to qualify the public sector innovation agenda and 
to share the owners’ role as co-creators of one of the world’s leading innovation environments. 

MindLab was originally created for the Ministry of Economic and Business Affairs as an internal incubator for 
invention and innovation, with five employees. At that time, the vision of an in-house laboratory as a centre of 
creativity and innovation was unique for a ministry. In the years that followed,  MindLab conducted over 300 
workshops, both within the ministry and for a broad range of other public and private organisations. In 2007, a 
new strategy and new goal were set for MindLab: its focus would be the active involvement of both citizens 
and businesses in developing new public sector solutions. At the same time, MindLab acquired two additional 
parent ministries, namely the Ministries of Taxation and Employment. In this manner, MindLab also became a 
fulcrum  of  intra-governmental  cooperation.  Finally,  the  strategy  involved  MindLab  taking  on  a  number  of 
professional researchers, with the aim of establishing a more robust methodological foundation for its work. 

Today,  MindLab  has  considerable  experience  with  innovation  processes  that  are  based  on  the  realities 
experienced  by  citizens  and  businesses,  and  which  also  promote  collaboration  across  the  public  sector. 
MindLab’s core staff consists of: 

  Seven project managers with a background in design, political science, anthropology, sociology and 

communication. 

  Seconded project managers heading up some substantial user-centred development projects within 

one or more of the parent ministries, for between six and twelve months. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

15 

  A research manager responsible for working with experts, think tanks, researchers and other 

knowledge environments to generate valuable change in MindLab’s parent ministries.  

  Trainees and students with a background in public administration, sociology, communication and 

design. 

MindLab’s strategic direction is set by the Board, which meets three to four times a year, and comprises the 
Permanent Secretaries of the three Ministries and the Chief Executive of Odense Municipality. The Board also 
gives final approval to MindLab’s portfolio of projects. An international Advisory Board has been established to 
provide  the  Board  with  expert  input  drawn  from  Denmark,  Australia,  Canada,  United  Kingdom  and  United 
States.  MindLab  resides  in  the  Ministry  of  Business  and  Growth  in  a  specially  designed  and  flexible  office 
space, which can be easily reconfigured. The space comprises several zones. The Mind is the characteristic egg-
shaped  space  lined  on  the  inside  with  whiteboards.  Architects  NORD  have  developed  the  concept  in 
collaboration with designers All the Way to Paris. They have created the Workshop Zone, which is the largest 
section,  and  is  where  most  of  MindLab’s  workshops  take  place.  The  library  consists  of  mobile  shelving  that 
contains MindLab’s literature, plus a high table that is used for meetings, lunches and informal gatherings. 

For further information:  info@mind-lab.dk, see also http://www.mind-lab.dk/en 

The European Commission itself has the Joint Research Centre (JRC), as its in-house science service, 
with  a  mission  to  provide  EU  policies  with  independent,  evidence-based  scientific  and  technical 
support  throughout  the  whole  policy  process.  Among  the  wide  set  of  services  offered  by  the  JRC 
across  the  Commission,  the  focus  on  innovation  in  policy-making  has  been  strengthened  with  the 
establishment of the EU Policy Lab. This new unit aims at providing a safe space where Commission's 
services  can  experiment  with  new  ways  of  addressing  complex  policy  challenges  as  well  as  new 
forms of engagement with stakeholders. One of the specific features of the EU Policy Lab is to bring 
together  an  interdisciplinary  team  with  competences  in  foresight,  behavioural  insight  and  design 
thinking.  The  EU  Policy Lab  is  engaged  in  a  range  of  innovation  projects  in  partnership  with other 
Commission's services, such as scenarios for the future of the collaborative economy, co-design on 
new  forms  of  engagement  for  the  implementation  of  regional  funds,  and  the  application  of 
behavioural sciences for EU policy-making.10   

The fear of failure can also be mitigated by conducting rigorous options appraisals before embarking 
in  a  new  direction,  as  a  well-established  method  and  a  crucial  component  of  impact  assessment, 
which  is  described  further  in  topic  1.2  in  the  context  of  assessing  proposed  legislation,  but  is 
applicable  to  all  policy  proposals  that  have  an  economic,  social  or  environmental  effect.  Options 
appraisal  applies  cost-benefit  analysis  (CBA)  techniques  to  several  implementation  scenarios, 
typically involving the status quo option (‘do nothing’), the proposed solution and at least one other 
alternative.  The  appraisal  must  be  genuinely  impartial  and  indifferent  to  the  options  to  add  any 
value, otherwise it is just a post hoc rationalisation of a pre-selected way forward.  

In finalising the  choice of policy instrument,  policy-makers  need to consider the  role of the public 
administration and its relationship to the chosen mode of implementation, especially if it involves 
decentralisation, outsourcing or co-production. The government may wish to devolve responsibility 
for the details of implementation to the organisation(s) tasked with delivery, especially if the policy 

10 For in-depth information on specific applications, please see also, topic 1.1.3 on foresight for EU policy-making, topic 
1.2.2 on applying behavioural insights and topic 1.3.3 on design for policy. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
  
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

16 

subject is complex and front-line providers are far better placed to determine what is best.  It may 
seek  to  pilot  a  variety  of  methodologies,  to  see  what  works  most  effectively.  It  may  wish  also  to 
select  an  array  of  providers  with  strengths  among  different  target  groups  to  promote  diversity. 
These considerations have been codified by the Institute for Government (op. cit.) as four criteria: 

Criteria 

Risk 

Uniformity 
Complexity 

Key questions 

  Does the government action need to be ‘right first time’?  

Is the priority to achieve a specific goal as efficiently or efficiently as possible, or to explore 
new possibilities?  

  What is the appetite for variety and divergence in service provision? 

Is  the  issue  so  complex  that  it  is  better  for  the  system  of  actors  to  address  it  through 
adaptation, rather than specifying a solution in advance?  

  How  likely  is  it,  that  central  direction  will  be  able  to  control  the  actors  responsible  for 

realising the policy in practice? 

Capacity 

  What  is  the  capacity  of  the  actors  in  the  system  to  address  the  policy  issue  through  their 

own agency?  
Is central government able to intervene to build such capacity?  

  To what extent is guidance or direction being requested? 

The  time  horizons  for  public  policy  extend  from  the  immediate  to  the  indefinite.  Some  policy 
challenges are ‘slow-burners’, such as the implications for care services and welfare provision of the 
ageing population, or the impact of long-term unemployment on skills and employability. Others are 
immediate,  arising  from  sudden  crises,  conflicts  and  natural  disasters.  Most  fall  somewhere  in 
between these two poles: medium-long term policy goals which are aspirational and ambitious, but 
affected  by  many  factors  outside  the  government’s  direct  influence,  such  as  improving  the 
educational  performance  of  children,  increasing  jobs  and  raising  productivity  in  the  economy,  or 
reducing the crime rate. Moreover, Governments seem to be increasingly tested by so-called ‘wicked 
problems’, such as the consequences of climate change for flooding and drought – intractable, multi-
dimensional and requiring the interaction of many stakeholders.  

1.1.2  Data insights to help solve policy problems 

Policy design 

Analysing data to solve policy problems 

In an increasingly ‘VUCA’ policy environment, 
coupled  with  advanced  technics  of  statistics, 
predictive  and  advanced  analytics,  and  data 
mining in combination with data visualisation 
technique,  governments  need  to  be  rigorous 
but responsive in their policy-making - and yet 
the  systematic  use  of  evidence  to  inform 
public policy remains relatively rare.  At the same time, we live in an increasingly data-rich society. 
The  exponential  growth  of  semiconductor capacity  over the  last  few  decades  (under  the  so-called 
Moore’s  Law11)  has  led  to  an  explosion  of  data  from  digital  sources  -  the  Internet,  social  media, 
mobile  phones,  sensors,  satellites,  static  and  wearable  devices  (e.g.  physical  fitness  and  step 
trackers,  air  pollution  monitors,  traffic  mappers,  etc.  that  contain  sensors,  processing  capabilities 
and connectivity). This has brought about the phenomenon of ‘big data’: a step-change in the scale 

Consultation and co-responsibility 

Forward thinking 

11 Moore’s Law is a computing term which originated around 1970; the simplified version of this law states that processor 
speeds or overall processing power for computers will double every two years. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

17 

and  scope  of  the  sources  of  materials  -  and  tools  for  manipulating  these  sources  -  available  in 
relation to a given object of interest.12 Opportunities now exist to strengthen evidence-based policy-
making from major ICT-led advances in data processing, by using the assembly and analysis of ‘big 
data’ to find patterns, linkages and relationships that shed fresh light on policy problems.   

‘Big data’ refers to both collections of datasets (numbers, vectors, text, images), often from bringing 
different sources together, and the tools and methods used to interrogate them (data analytics). The 
benefits of big data for public policy arise from its size and sophistication, enabling users to identify 
trends  and  patterns,  and  to  find  connections  and  correlations  across  policy  fields  by  combining 
datasets. There is no universal standard or threshold to define ‘big’, but the common approach is to 
focus on four Vs: volume (of objects, including over time), variety (of sources), velocity (of availability, 
including  real-time  access),  and  veracity  (of  quality,  through  validation  and  reduction  of  bias  and 
‘noise’).  

The immediacy of data flows in the digital era blur the boundaries between public service ‘users’ and 
‘providers’.  This  is  also  integral  to  the  concept  of  the  Internet  of  Things,  in  which  objects  in  a 
network  share  information  with each other  and  create  a smart  environment  for  people  to  extract 
services and value.  

Big data analytics has proven to bear promising insights for all policy fields. Its use is still embryonic 
but  evolving  quickly.13  Public  administrations  are  increasingly  mining  data  as  a  raw  ingredient  to 
inform  innovation  in  policy  design,  evaluation  and  implementation.  Among  Member  States14,  for 
example:  

  Germany’s  simTD  research  project,  funded  by  three  Federal  Ministries  and  involving  a 
partnership  of  the  Hessian  State  Government,  automotive  and  telecommunications 
companies, universities and research institutions, aims to increase road safety and improve 
the  efficiency  of  the  traffic  system,  through  the  use  of  car-to-car  and  car-to-road 
communication. 

  Rijkswaterstaat,  part  of  the  Dutch  Ministry  of  Infrastructure  and  the  Environment,  has 
launched an initiative with partners titled Digital Delta to use big data sources for monitoring 
purposes to prevent floods and to improve water quality and internal navigation.15 

  The  Spanish  Ministry  for  Telecommunications  and  the  Information  Society  uses  big  data 
analytics  to  support  its  understanding  of  the  structure  and  size  of  the  ICT  sector  and  to 
improve its evaluation process for awarding R&D grants, using natural language processing. 

International organisations are increasingly active in data for policy-making initiatives, including the 
OECD through its E-Leaders initiative, United Nations (UN) through its Global Pulse project and the 

12 Definition from R. Schroeder (2014), Big Data: towards a more scientific social science and humanities? in M. Graham 
and W.H. Dutton (eds.) ‘Society and the Internet’. Oxford: Oxford University Press, pp. 164–176. 
13 For example, the application of big data and open data through algorithmic regulation is an emerging field within 
regulatory reform, although at an early stage where the pros, cons and risks are still being examined. 
14 The following examples are taken from the EU-funded www.data4policy.eu. 
15 B. Rooney (2013) The Netherlands Looks to Big Data to Tackle Flooding, The Wall Street Journal 
http://blogs.wsj.com/tech-europe/2013/06/25/the-netherlands-looks-to-Big-data-to-tackle-flooding/;  

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
   
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

18 

World  Bank  through  its  ‘Innovations  in  Big  Data  &  Analytics  for  Development’  programme.    The 
European  Commission  is  pursuing  several  actions  in  the  field  of  big  data,  launching  both  research 
studies and pilot initiatives.  

EU support to big data analysis 

Under the  ISA Programme, the European Commission has been  collecting best  practices in Member States' 
public  administrations  in  analytics  and  big  data  technologies  to  support  decision-making,  along  with  the 
supporting organisational and operational processes.  

The  Commission  also  funds  big  data  related  projects  through  the  Horizon  2020  Framework  Programme  for 
Research and Innovation, not least through Co-Creation-06-2017: Policy-development in the age of big data: 
data-driven policy-making, policy-modelling and policy-implementation. 

A  task  force  in  EUROSTAT  is  exploring  how  big  data  analytics  can  complement  official  statistics  with 
information coming from various sources (e.g. mobile data, Wikipedia stats), and contributing to the business 
cases of the European Statistical System's Big Data Project. 

The Commission outlined its new strategy on big data in 2014, focused on research in innovative area, with 
several project themes connected to EU policy objectives, including: 

  Healthcare: saving lives with better diagnostics. This has led, for example, to the development of an 
ICT  tool  for  brain  trauma  patients  (TBICARE)  and  a  platform  to  promote  a  healthier  daily  life 
(DAPHNE). 

  Transport: fewer accidents and traffic jams. This has included, for example, a project on tackling road 

congestion (VIAJEAO).  

  Environment:  reducing  energy  consumption.  This  has  included,  for  example,  a  project  on  the  best 

place for wind farms (Sopcawind). 

The  Commission  has  also  funded  data4policy,  a  consortium  of  the  Technopolis  Group,  the  Oxford  Internet 
Institute (OII) and the Centre for European Policy Studies (CEPS), to explore the opportunities that innovative 
data-driven  approaches  offer  for  evidence-informed  policy  making,  including  the  relevant  data  sources  and 
technologies.  The  study  contributed  to  creating  or  linking  relevant  communities  in  the  field  (policy-makers, 
public  agencies,  NGOs,  companies  that  provide  tools  and  collect  data,  etc.),  conducted  an  inventory  of  58 
innovative  initiatives  across  the  EU,  plus  selected  non-EU  countries  and  international  organisations,  and 
organised the Big Data  for Policy conference on 15-17 June 2015. The final deliverables are available at the 
study website: State-of-the-Art  report; the workshop report, 10 cases of innovative data-driven approaches 
for policymaking at EU level; and an online ‘bee health’ demonstrator of big data linking and data visualisation 
for policy-making.  

The Commission is also developing a series of tools and projects with direct applications for EU policies. One 
example is the Tool for Innovation Monitoring (TIM), which is based on technology forecasting and exploiting 
data coming from patents, publications and news. The Commission’s Joint Research Centre (JRC) is using big 
data analytics in Earth and environmental sciences through the Digital  Earth Platform, which can help other 
Commission Services working in this policy area.  

The EU’s eGovernment Action Plan 2016-2020 (see topic 5.4) includes several actions to make the most of big 
data’s  potential.  There  is  a  growing  demand  from  business  and  citizens  to  have  access  to  high  quality, 
interoperable and re-usable data to provide new services, particularly in the field of spatial data. The use of 
spatial data for urban, land-use, traffic planning and for scientific purposes can unleash new innovations that 
respond  to  societal  needs  such  as  reducing  the  negative  impact  on  the  environment.  In  this  regard,  the 
Commission has committed to promote the development of end-user applications, including specific EU-level 
applications,  to  harvest  from  citizens  and  business  more  efficiently  the  data  provided  through  Spatial  Data 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

19 

(INSPIRE) for EU-level policy making and implementation, particularly in relation to reporting and compliance 
promotion. This will improve evidence-based policy making and support eGovernment processes where high 
quality spatial data (e.g. cadastres, maps, addresses, buildings, parks, protected sites, natural risk zones, etc.) 
is needed. 

Under the auspices of the Action Plan, the Commission has also launched the European Cloud Initiative as its 
blueprint for cloud-based services and world-class data infrastructure to ensure science, business and public 
services reap the benefits of  the big data revolution. Europe is the largest producer of scientific data in the 
world,  but  insufficient  and  fragmented  infrastructure  means  this  big  data  is  not  being  exploited  to  its  full 
potential. By bolstering and interconnecting existing research infrastructure, the Commission plans to create a 
new European Open Science Cloud that will offer Europe's 1.7 million researchers and 70 million science and 
technology  professionals  a  virtual  environment  to  store,  share  and  re-use  their  data  across  disciplines  and 
borders.  This  will  be  underpinned  by  the  European  Data  Infrastructure,  deploying  the  high-bandwidth 
networks,  large  scale  storage  facilities  and  super-computer  capacity  necessary  to  effectively  access  and 
process  large  datasets  stored  in  the  cloud,  and  will  reduce  the  cost  of  data  storage  and  high-performance 
analysis. Making research data openly available can help boost Europe's competitiveness by benefitting start-
ups, SMEs and data-driven innovation. Public services will benefit from reliable access to powerful computing 
resources and the creation of a platform to open their data and services, which can lead to cheaper, better 
and faster interconnected public services. The Commission will progressively put in place the European Cloud 
Initiative  through  a  series  of  actions  from  2016  to  2020.  The  public  and  private  investment  needed  to 
implement  the  European  Cloud  Initiative  is  estimated  at  €6.7  billion,  including  €2  billion  in  Horizon  2020 
funding. 

To  further  advance  the  eGovernment  Action  Plan,  the  Commission  has  also  established  an  engagement 
platform  (https://ec.europa.eu/futurium/en/egovernment4eu),  on  which 
ideas  can  be  proposed.  The 
platform seeks to understand stakeholders’ needs and then asks for ways to address them. Data can be used 
to justify the idea for action.  

Sources: Opportunity now: Europe’s mission to innovate; data4policy website; Commission press release and 
website  

Data analytics can help overcome the challenge of designing better informed policies that reflect and 
address the complexity of (systemic) policy problems. It can serve as a policy tool at various stages of 
intervention: problem analysis and agenda setting; ex-ante impact assessment of potential policies; 
monitoring the implementation of existing policies; or evaluating ex-post their effectiveness.  

The starting point for thinking about big data as a policy-making tool is to ask the question: for this 
policy problem,  is there a potential data-driven solution?   In other words, if we search for fresh 
data, seek answers within existing data or bring datasets together, could it shed new light or open 
up new windows of opportunity? 

Once  the  initial  decision  is  taken  to  look  to  data  for  answers,  and  the  policy  question  which  the 
policy-maker is seeking to address is clearly formulated, the next requirement is to identify data sets 
and ensure data availability, which can involve collection, capture, saving and storage (for example 
in  data  warehouses  and/or  cloud-based),  data  cleaning  to  improve  quality,  and  data  linking  to 
produce larger datasets with potential correlations, which depends on interoperability16.  

The purpose of data analysis is to uncover trends, patterns and connections that might otherwise be 
invisible,  while  data  visualisation  ensures  the  outputs  are  presented  in  a  reader-friendly  format, 

16 See topic 5.4 for an explanation of interoperability. Differences between data sets can include data formats, identifiers 
(for citizens or enterprises), storage solutions, meta-data systems, etc. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
  
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

20 

more accurately reflecting the complexity of policy options and knock-on effects and with the overall 
aim to identify insights to feed into policy decisions.   

This process can be characterised as a series of  six steps, which includes a prototype stage before 
the data processing solution to policy-making is rolled out or scaled up17. This should be viewed as a 
dynamic,  iterative  process,  as  data-driven  insights  (step  6)  can  and  should  reframe  the  policy 
question (back to step 1 or 2), by furthering the understanding of causalities, related factors, etc., or 
indeed, when data 'streams' reflect a near real-time dynamic change of the policy problem.  

6. Apply data 
innovation in 
policy-making 

5. Produce a 
working 
prototype of 
the data 
analytic tool 

4. Identify data 
sources and 
processing 
needs 

3. Translate 
policy question 
into data 
problem 

2. Formulate 
clear policy 
question 

1. Identify and 
brainstorm 
policy challenge 

Theme  7  includes  the  inspiring  example  of  Slovenia’s  Judicial  Data  Warehouse  and  Presidents’ 
Performance  Dashboards,  which  effectively  comprises  all  these  steps  and  has  successfully  helped 
reduce the backlog of pending court cases and improve efficiency.  

Moreover,  the  ongoing  example  below  of  data-driven  approach  to  tackle  illegal  fishing  and  over-
fishing activity at sea, based on the data4policy case study, illustrates this data-driven policy process 
up to step 5 (prototyping) of the above typology.  One of the interesting aspects of this case is that 
two  potential  systems  have  been  developed  in  parallel:  one  is  aimed  at  public  authorities  for 
enforcement  purposes  (‘Eyes  on the  Seas’),  while  the  other  is open  to  all  and more  suited  to civil 
society and hence external pressure and accountability purposes (‘Global Fishing Watch’).  

Step 1: Identify and brainstorm policy challenge 

Ocean governance 

Illegal  fishing  and  over-fishing  are  serious  global  challenges  from  an  environmental  and  economic  point  of 
view  that  can  permanently  destroy  ocean  ecosystems  and  jeopardise  food  security.  According  to  the  UN, 
“about a third of marine fish stocks worldwide have been overfished, and over 90% of the world’s fisheries are 
fully exploited or over-fished”. In the case of illegal fishing, the scale of the problem is well known (although it 
may be understated) - experts estimate that about 20-40% of the fish caught in the wild enters the market 
due  to  illegal  fishing  activities,  accounting  for  more  than  $20  billion  each  year.  Monitoring  and  enforcing 
protected marine reserves, fishing quotas and legislation in remote parts of the  world is a  challenge that is 
currently  not  sufficiently  addressed.  This  ‘policy  failure’  results  in  the  presence  of  industrial-scale  pirate 
fishers having a sense of impunity. There is a need to better protect the oceans from illegal activity, so that 
depleted fishing stocks can be safely preserved and rebuilt.   

Over  60%  of  the  world's  oceans  are  beyond  national  borders.  The  high  level  of  fragmentation  in  ocean 

17 This typology is based around data4policy’s readiness assessment criteria. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

21 

governance complicates monitoring activities and interventions: responsibilities and competences are spread 
among different  agencies at  regional (coastal regions),  national and international levels. For  example, there 
are  about  10  different  UN  bodies  responsible  for  marine  protected  areas.  To  have  better  knowledge  of 
activities at sea, there needs to be more transparency and especially, more collaboration between different 
institutions.  The  general  objective  of  European  Ocean  Governance  Policy  is  to  ensure  better  international 
governance of oceans and seas to the benefit of sustainable blue growth.   

The European Union is obliged to monitor illegal fishing activities as part of its role in enforcing the European 
Union’s Common Fisheries Policy (CFP). This governs access for fishing vessels inside the exclusive economic 
zone  (EEZ)  of  EU  member  states  which  stretches  200  nautical  miles  from  their  coastlines,  and  negotiates 
access in waters of third countries or governed by Regional Fisheries Management Organisations. Within their 
EEZs,  Member  States  retain  a  12-mile  zone  from  their  own  coastlines  with  exclusive  fishing  rights  for  their 
vessels. At 25 million square kilometres, the EEZ of EU countries is the largest in the world.  

In 2009 the Pew Charitable Trusts, the Coalition for Fair Fisheries Arrangements, the Fisheries Secretariat, the 
New Economics Foundation, Seas at Risk and Ecologistas en Acción established the OCEAN2012 coalition. It 
included  fishermen  organisations,  environmental  NGOs,  consumer  organisation  and  other  actors  with  an 
interest  in sustainable fisheries. Within  five years, OCEAN2012  had  grown  from  5 to 193 supporting groups 
across 24 EU Member States.  

European legislators were moving in the same direction. In May 2013, they reached an agreement to reform 
the  Common  Fisheries  Policy,  to  “restore  fish  stocks  and  to  end EU  overfishing  by  2015  where  possible  and 
2020  by  the  latest”.  The  new  Policy  came  into  force  in  January  2014.    It  allocated  greater  funding  for  data 
collection and enforcement within the European Maritime and Fisheries Fund (EMFF) which will deploy €6.4 
billion between 2014-2020. The majority will be managed by the Member States themselves but 11% will be 
managed  by  the  Commission  to  support  EU-wide  objectives  in  maritime  and  coastal  affairs,  including 
international  governance.  Of  the  total  EMFF  funds,  €580  million  have  been  allocated  for  control  and 
enforcement activities. These resources will contribute to monitoring compliance with the CFP and to protect 
a fair access to healthy fishing stocks.  

Enforcement is a responsibility of individual Member States. They can be notified of suspicious behaviour and 
must ensure that their own vessels or vessels within their waters comply with EU agreements. In severe cases 
of non-EU Member States breaching international agreements and failing to act to correct illegal fishing, the 
EU can set up embargoes to fish products from those countries. Up until now, enforcement has been based 
on the collection of data provided by the Member States and on the use of inspectors checking fishing gear 
and inspecting the registers of fish caught (and comparing them with the quotas). These checks can be done 
in  port,  at  sea,  and  using  aerial  photography.  However,  they  are  costly,  highly  resource-intensive  and  time 
consuming  (and  therefore  slow)  processes  that  do  not  sufficiently  act  as  deterrent  for  most  illegal  activity. 
Most  important,  they  do  not  allow  for  timely  information  at  the needed  scale.  As  a  result,  non-compliance 
remains a significant problem.   

Step 2: Formulate policy question  

How can we effectively identify suspicious or illegal fishing activities (to enforce the rule of law and improve 
ocean governance)? 

Step 3: Translate policy question into data problem  

Most  of  the  data  to  effectively  tackle  this  policy  question  exists  and  has  been  identified  as  being  available 
either from commercial providers or the different governments and institutions. The public sector has access 
to  vessel positioning data, which  can be  linked to other  administrative  data  sets,  to indicate possible illegal 
activity  that  can  be  followed  up  by  the  relevant  law  enforcement  agencies.  represents  for  an  improved 
conservation  of  marine  resources  and  global  ocean  governance.  There  is  the  potential  to  develop  an 
automated  early-warning  system  that  would  enable  flagging  suspicious  activities  through  the  linking  of 
available data sets combined with the use of advanced analytical techniques. Such a system would need to be 
aware of the vessel’s position over time, the administrative information detailing available quotas and fishing 
licences for each vessel, and data on the location and characteristics of fish stocks in the sea.  

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

22 

There would also be spin-off benefits. Information on the movement of (large) ships would be relevant not 
only related to illegal fishing, but also for environmental, tax and trade concerns. Coastal information, satellite 
imagery and information from aircrafts and patrol boats can be combined to form a fuller picture of activity in 
the  ocean.  More  real-time  data  on  positioning  and  patterns  of  large  ships  would  allow  more  targeted 
interventions for EU inspectors and more effective legal action. To cover the oceans, combined monitoring is 
used integrating several systems.  

Using  a  data  approach  could  therefore  contribute  to  a  more  effective  conservation  of  natural  resources 
(fishing,  deep  sea  mining,  protected  areas),  reduce  fragmentation  in  the  governance  of  sea  activities 
(fostering  better  coordination  and  collaboration  between  different 
institutions)  and  provide  more 
transparency through better knowledge of activities at sea. 

Step 4: Identify data sources and processing needs 

With regards to data sources: 

  The  Vessel  Monitoring  Systems  (VMS),  which  regularly  provides  data  on  the  location,  course  and 
speed  of  vessels  to  the  fisheries  authorities  of  flag  states  and  those  of  the  coastal  states  in  whose 
waters it is fishing. 

  Fishing vessels over 300 gross tons carry the Automatic Identification System (AIS), which broadcasts 
a signal to prevent collisions. AIS has been progressively introduced in EU fishing vessels above 15m 
and is increasingly used by Member States for monitoring the behaviour of large ships and collecting 
data on their speed, position and direction. 

  Currently, satellite imagery is used mostly in research projects. It has proved too expensive to deploy 
on  an  operational  basis.  The  main  difficulties  are  the  time  between  requesting  an  image  and 
receiving one (each satellite only passes overhead every day or so and the satellite needs time before 
it can programme its camera or sensor towards the target) and the need for manual identification of 
vessels  in  the  images  –  usually  synthetic  aperture  radar  due  to  the  ability  to  see  through  cloud. 
Improving the capability of imagery for vessel detection is the focus of the EU’s Copernicus maritime 
monitoring programme. 

What are the pros and cons of VMS and AIS? 

  Both  VMS  and  AIS  obtain  positional  information  from  GPS  signals,  but  VMS  uses  point-to-point 
satellite communication to send the information to shore whereas AIS transponders broadcast VHF 
signals that can be picked up by satellite or shore stations. 

  Communication costs make VMS expensive, so transmission occurs only every hour or so. AIS signals 

are transmitted approximately every minute.  

  Data  sharing  requirements  for  VMS  are  more  bureaucratic.  Furthermore,  VMS  information  is  only 
available to flag state and coastal state, unless special arrangements are made for sharing, whereas 
AIS information can be bought for any area of the planet.  

  There are differences between distribution of AIS signals picked up by satellite and those picked up 
by coastal stations. Satellite AIS gives global coverage of the sea but signals are only picked up when 
a  satellite  passes  above  an  area.  The  frequency  of  signals  depends  on  the  number  of  satellites 
scouting  a  specific  area.  As  it  passes,  the  satellite  can  pick  up  more  than  one  signal  so  that  vessel 
tracks can be observed. On the other hand, due to  signal  collision,  satellites do not  pick  up all the 
signals in high ship density areas, making this solution better for open ocean monitoring. In this case, 
signals picked up by coastal stations can be much more frequent but their range is more limited. 

AIS  data  is  collected  by  coastal  stations  and  satellites  and  it  can  be  bought  from  commercial  providers.  In 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

23 

terms  of  costs,  there  are  providers  of  data  from  coastal  stations  that  already  have  operational  online 
platforms  that  are  supported  by  subscription  models.  For  example,  IHS  Maritime  sells  AISLive/Sea-web 
subscription  plans  to  their  online  platform  and  marinetraffic.com  sells  subscriptions  from  €9  to  €269  per 
month, depending on the requested features. 

Once  the  data  are  available,  two  stages  of  analysis  can  be  deployed,  first  related  to  the  modelling  and  the 
analytics  to  identify  patterns  and  types  of  behaviour  for  ships,  second  to  signal  anomalies  and  suspicious 
cases.  A behavioural classification model can be run on the datasets to identify fishing behaviour, based on 
the vessel’s pattern of movement and speed. 

During the past two years, two initiatives have worked to address the data sourcing, linkage, processing and 
visualisation challenges related to the proposed automated early-warning system: 

  The ‘Eyes on the Seas’ project, developed by the Pew Charitable Trusts together with the UK Satellite 

Applications Catapult, 

  The ‘Global Fishing Watch’ prototype by Oceana, Google and Skytruth.   

The prototype systems link positional data from ships, collected mainly from the AIS system, with geographic 
and  administrative  data.  AIS  systems  are  codified  with  unique  identifiers  for  each  vessel,  which  allows 
identification  of  the  country  of  origin.  Additional  information  includes  the  type  of  vessel  -  fishing,  cargo, 
tanker etc – although this is not always accurate. The data on AIS is already being collected by coastal stations 
and  distributed  by  commercial  providers.  For  the  rest  of  the  ocean,  the  data  can  be  bought  from  satellite 
providers or intermediaries.  For example: 

  The  ‘Eyes  on  the  Seas’  system  combines  data  sets  from  VMS  (via  GPS  and  satellite),  Synthetic 
Aperture Radar (SAR) satellite, AIS and Optical Satellite Sensors for oceanographic and atmospheric 
data.  These  are  further  linked  to  other  specialist  databases  that  include  international  fishing  and 
marine  reserve  boundaries,  and  oceanic  data  comprising  depth  and  temperature  readings 
(bathymetry).  All  the  information  is  centralised  in  a  ‘Virtual  Watchroom’  where  analysts  can  query 
the data and interesting activity patterns are investigated. 

  The  ‘Global  Fishing  Watch’  system  builds  dynamic  visualisations  of  ships  based  on  AIS  signals,  and 
runs behavioural models. These behavioural models can distinguish between fishing and non-fishing 
ships.  In  a  following  step,  they  are  assessed  by  analysts  and  crosslinked  with  other  datasets  that 
delineate  restricted  fishing  zones.  The  process  started  with  a  dataset  of  3.7  billion  data  points, 
comprising two years of satellite collection and covering 111,374 vessels. This dataset is one terabyte 
in size. 

In addition, the European Marine Observation and Data Network (EMODnet) is an initiative run by the EC’s DG 
MARE.  It  is  a  platform  that  consists  of  more  than  100  organisations  providing  marine  data,  providing 
complementary background information on ocean conditions such as temperature, currents, sediments and 
marine  life.  The  data  and  meta-data  undergo  a  standardisation  and  quality  assurance  process  and  are 
available  without  restrictions  of  use  (free  and  open  access).  This  contextual  information  can  be  used  to 
enhance the information on ship movements. EMODnet is currently starting its third development phase and 
it  is  expected  that  it  will  be  completed  in  2020.  The  rest  of  the  data  is  available  either  from  commercial 
providers or the different governments and institutions.   

Step 5: Produce a working prototype 

The two initiatives  – ‘Eyes on the Seas’ and ‘Global Fishing Watch’  – have both built  prototypes, which  are 
currently  testing  the  feasibility  and  potential  implementation  of  the  idea  launched  in  this  use  case.  In 
addition, the prototypes allow the  set-up and training of  the analysis  models  for this  particular  application. 
This is a necessary step before these solutions can be validated and have sufficient legitimacy to be used in 
policy-making. As at early 2016, the two prototypes still had some limitations. For example: 

  The project ‘Eyes on the Seas’ launched its monitoring activities by focusing on the Chilean territory 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

24 

surrounding the Easter Island, and the Pacific island nation of Palau. The system initially focuses on 
helping  the  governments  of  these  territories  to  enforce  these  protected  marine  reserves,  and  the 
idea is that it will be able to scale to other larger areas. The ‘Project Eyes on the Sea’ documentation 
discusses  some  of  the  behaviours  that  can  trigger  an  alarm.  The  use  of  vessel  data  is  linked  with 
marine reserve boundary data. Patterns can be identified that include activity in closed fishing areas 
or unauthorised switching-off of AIS transceivers. While switching AIS transceivers off can be used to 
try  to  circumvent  detection,  global  coverage  allows  detecting  when  a  vessel’s  position  suddenly 
disappears  from  the  dataset  and  reappears  later  elsewhere  is  flagged  (or  when  vessels  are  using 
duplicated AIS identifiers). Also, the system can detect close proximity between vessels, which could 
signal  that  some  of  the  cargo  is  being  moved  between  ships  at  sea.  When  an  alarm  is  triggered,  a 
team of trained analysts then investigates it and relevant government enforcement organisations are 
notified. A supporting evidence package is transferred to the authorities, who then proceed with an 
appropriate response in case the rules were breached.  

  By  contrast,  the  ‘Global  Fishing  Watch’  had  a  global  reach  from  the  outset,  but  the  prototype 
operates  using  a  snapshot  of  static  data  from  2012  and  2013.  As  at  2016,  it  covered  over  25,000 
unique  vessels  that  were  identified  as  carrying  out  fishing  activities,  and  the  final  visualisation 
comprises  300  million  AIS  data  points.  The  fishing  activity  map  in  the  prototype  shows  data  from 
3,125 vessels that were independently verified. The idea is that when the system is completed, it will 
switch  to  using  near  real-time  data  streams,  analysing  current  conditions  while  still  having  the 
capacity to examine historical behaviour. Oceana stated in late 2014 that the system would be fully 
completed by 2016, depending on funding conditions.   

However,  the  main  difference  is  that  ‘Eyes  on  the  Seas’  primarily  aims  to  facilitate  direct  action  by 
governments and authorities, while ‘Global Fishing Watch’  puts more emphasis on the fact that the system 
will  be  free  and  open  from  the  outset  (like  another  layer  on  top  of  Google  Maps),  so  that  pressure  for 
authorities  to  act  will  come  from  external  pressure  groups,  citizen  activists,  NGOs,  media,  etc.  The  role  of 
NGOs for this particular case is interesting and can have an effect in increasing public awareness and holding 
governments  to  account.  So  far,  governmental  authorities  have  been  reluctant  to  provide  information  on 
fishing, citing confidentiality as an issue.  

As  well  as  the  provision  of  near  real-time  information,  AIS  allows  the  scaling  up  of  the  current  anomalies 
detection system to the global scale. It would equally add an automatic process for identifying misreporting to 
the established  framework. For example, one  can identify cases  where  vessels have incorrect or duplicated 
identifiers.  Regarding  the  governance  process,  it  would  add  a  global  dimension  and  foster  collaboration 
amongst several institutions at EU level, at least amongst EU agencies EMSA, CFCA and Frontex. If the analysis 
were offered openly to the public, this would add elements of open policy-making and allow pressure groups 
and other activist organisations to also carry out monitoring activities.   

The technical scalability of the demonstrators is possible by moving from the use of snapshots to (quasi) real-
time data and by expanding the focus from specific regions to global coverage. As computing power improves 
(or  more  resources  are  dedicated  to  such  initiatives)  the  pilot  projects  can  move  from  dedicated  control 
systems (the concept of a ‘watch room’) to a service that is widely available on the Internet (i.e. the Google 
approach). Additionally, new sources of information can be added in the future, including additional satellite 
imagery,  various  types  of  optical 
imagery  from  unmanned  aerial  vehicles,  crowd-sourced 
photographs  and  sightings,  electronic  signals  such  as  radar  on  ships,  and  possibly  radio  broadcasts.  To 
undertake  this  scaling  up  process,  three  things  would  be  needed:  additional  resources;  validation  of  the 
models that involves experts from the relevant authorities; and more countries and organisations across the 
fish and seafood value chain providing their own data to reduce the false positives. 

imagery, 

Step 6: Use of the data in policymaking (forthcoming) 

The  November  2016  Communication  “International  ocean  governance:  actions  for  safe,  secure,  clean  and 
sustainably managed oceans” proposed that: 

On the basis of technological developments in satellite communications and data analysis, and existing 
in  association  with  the  High 
systems  for  monitoring  maritime  activities,  the  Commission, 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

25 

Representative,  will  launch  a  pilot  project  to  monitor  illegal  fishing  worldwide,  working  to  broaden 
maritime situational awareness, and explore the possibilities for expanding monitoring to other sectors.   

The prototype systems allow government officials and other independent analysts and groups of experts to 
identify  and  monitor  activities  at  sea.  Activities  that  present  specific  activity  patterns  will  trigger  alarms, 
signalling potentially illegal, unreported or unregulated activities. These systems are early alert systems and 
can  prompt  the  relevant  authorities  to  take  a  closer  look  into  specific  cases.  They  are  more  efficient  than 
current  established  procedures  and  reduce  the  amount  of  resources  that  are  needed  to  monitor  vast  sea 
extensions and fleets. The Commission Communication does not indicate how the system will develop from 
pilot  to  operational  phase,  who  will  operate  it  and  who  will  have  access  to  the  results  but  the  intention  is 
clearly there. 

However, it can be assumed that the evidence from these systems will always need to be complemented by 
other proof of wrongdoing. Suspicious patterns will be packaged and sent to the appropriate teams within the 
relevant  authorities  who  will  have  the  remit  of  investigating  particular  cases.  This  is  analogous  to,  for 
example, systems in the banking domain that flag suspicious credit card charges or bank transfers. When an 
alarm  for  suspicious  activity  is  triggered,  the  case  is  passed  on  to  a  relevant  expert  team  that  verifies  and 
checks the situation with the involved parties. The systems will also enable authorities to share information 
on  specific  suspicious  vessels  and  patterns  of  activity,  and  thus  increase  international  collaboration  in 
information gathering and enforcing activities.  

Potential synergies with other initiatives 
Eurostat and ESMA also use AIS as a source data for the collection of transport  statistics and CO2 emissions 
analysis, but limited to the European space. Multipurpose use of this data is possible, but should be extended 
beyond Europe. This broader use of AIS data could increase the relevance and efficiency of the data collection 
processes.  An  element  that  requires  further  consideration  regards  the  political  and  operational  governance 
structure  that  would  support  and  sustain  this  data  collection  and  analysis  initiative:  who  would  be 
responsible, who would be the caretaker and who would fund it. 

Source: www.data4policy.eu 
For further information: Global Fishing Watch, http://globalfishingwatch.org; Project Eyes on the Seas, 
http://www.pewtrusts.org/en/multimedia/video/2015/project-eyes-on-the-seas 

Regarding data availability, the sources might be public or private18: 

  The Statistical Offices remain among the most important players within the public sector, as 
data  collection  and  assembly  is  their  raison  d’être,  but  other  public  bodies  also  generate 
data as a by-product of their daily activities.  

  The  largest  source  of  digital  data  is  privately-held,  including  all  economic  transactions, 
mobile  phone  metadata  (e.g.  location  data  -  relevant  to  transport,  healthcare,  safety  and 
security),  and  search  engine  data  and  the  content  of  social  media,  both  of  which  indicate 
people’s  interest  and  preferences.  Geo-data  from  satellites,  whether  government  or 
privately-owned,  can  contribute  to  mapping  climate  change  (environment  and  agriculture 
policy), people (migration) and freight movements (transport), fish stocks (aquaculture), and 
urban spread (physical planning).   

 

Increasingly,  data  generated  by  citizens  online  through  privately-held  devices,  not  just 
government or businesses. Examples include micro sensors to monitor air and water quality 
or  their  own  health  signs  (blood  pressure,  heart  rate,  etc.).  Some  of  the  most  powerful 

18 This is where the concept of open data is important (see also topic 1.1.4, topic 2.3 and topic 5.4). 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

26 

applications of big data arise from combining different data sources, although this relies on 
access  (if  proprietary),  anonymity  (if  the  data  could  be  used  to  identify  individuals)  and 
interoperability (if the data formats are not immediately compatible).  

  With the advent of the Internet of Things (IoT), data is now also accruing from sensor data in 

equipment and machine-to-machine (M2M) communications. 

Many  data  processing  techniques  (some  of  which  are  summarised  below)  involve  data  analytics, 
which  is  the  process  of  drilling  down  into  raw  and  unstructured  data  to  extract  useful  and  usable 
information, which can be gathered and assembled in a common space, such as a data warehouse. 
These techniques may seem esoteric, but are increasing in usage as the examples show. 

Tool 

Data profiling 

Web scraping  

Text mining 

Description 

involves 

investigating  available  data 

in  an  existing  source  and  collecting 
This 
information about that data (including metadata), to determine if it can be easily used, 
conforms  with  certain  quality  standards  and/or  could  be  integrated  with  other  data 
(interoperability).  The  data  might  be  tagged  (with  keywords  or  descriptions)  or 
categorised, so that it can be drawn upon later. 
Also  known  as  web  harvesting,  this  is  the  automated,  high-speed  and  large  volume 
version of ‘copy-paste’, and involves using software to extract data from websites, and 
save  /  store  it  in  a  database or  spreadsheet,  which  can  then  be  analysed  and  visually 
(re-)presented.  Examples  include  online  price  comparison,  which  is  based  on  trawling 
retailers’ sites for the raw price data on specified products. Eurostat has experimented 
with  online  price  data  obtained  by  web  scraping  as  an  additional  source  for  the 
Consumer Price Index. 
Also known as text analytics, this involves trawling through a set of documents or other 
online  sources  to  extract  relevant,  new  and/or  interesting  information  from  the  text, 
based  on  identifying  keywords,  categorising  and  clustering  the  text,  recognising 
concepts  and  patterns  (e.g.  word  frequency),  and  evaluating  the  output  to  discern 
which  text  is  useful  and  usable.  This  might  be  performed  for  the  purposes  of 
summarising  or  classifying  documents,  looking  for  associations  between  entities, 
conducting  sentiment  analysis,  or  performing  predictive  analytics,  for  example.  The 
European  Commission  has  launched  pilot  projects  which  use  text  mining  of  scientific 
papers to identify research trends, and analyse the ICT job market through information 
available on job posting websites.   

Sentiment analysis   This  technique  seeks  to  extract  and  capture  people’s  inclinations,  opinions  and 
reactions  in  subjective  information  from  the  web  (news  articles,  reviews,  blog  posts, 
tweets,  status  updates,  etc.)  using  natural  language  processing  and  computational 
linguistics (which enable computers to understand human language and are the basis of 
machine  translation,  for  example).  It  can  be  used  for  example  to  gauge  reaction  or 
anticipation  of  policy  announcements,  or  assess  levels  of  public  trust.  Statistics 
Netherlands has used social media sentiment analysis for consumer confidence data 
Using  algorithms,  this  method  enables  computer  to  learn  from  interrogating  data 
iteratively,  recognising  patterns  and  relationships,  and  hence  to  build  its  own  models 
and extract ‘hidden insights’ without being explicitly programmed where to look. 
The  aim  is  to  determine  patterns  within  datasets  and  make  predictions  about  future 
events  or  behaviours  based  on  probability  models  and  ‘what  if’  scenarios.  Predictive 
analytics is used in risk assessment (such as credit ratings), medical diagnostics and the 
identification of tax fraud (by mining tax returns).  

Predictive 
analytics 

Machine learning 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

27 

Social network 
analysis (SNA) 

Agent-based 
modelling (ABM) 

etc. 

SNA  uses  network  theories,  models  and  applications  to  investigate  the  nature  and 
importance  of  relationships  within  social  structures,  namely  the  connections  and 
interactions  between  actors  in  a  network.  These  relations  might  be  social,  economic, 
political, 
The 
biological, 
analysis  focuses  entirely  on  the 
actors’  associations,  rather  than 
their individual attributes. Network 
models  are  built  around  nodes 
(members  of  the  group)  and  the 
links  /  ties  between  them,  which 
can  be  presented  graphically  (see 
diagram  right,  taken  from  World 
Bank blog on cross-border business 
ties).  From  its  origins  in  sociology, 
SNA has become a  valuable tool in 
medicine  (for  example,  regarding 
the spread of diseases), economics, 
information  science  and  ICT,  organisational  behaviour,  etc.  Concepts  used  in  SNA 
include:  tie  strength;  density  (the  ratio  of  direct  ties  to  all  ties  in  a  networks);  and 
distance (minimum number of ties required to connect two particular actors). Network 
methodologies focus on sub-groups, from 2 or 3 nodes upwards, and in the case of big 
data analytics, larger systems or entire networks.  
ABM involves simulation to study the behaviour and actions  of agents in a network or 
system,  either  as  individuals  or  collective  entities  (organisations  or  groups),  to  assess 
their  impact  -  through  interactions  -  on  the  whole  system.  ABM  draws  on  various 
techniques,  including  game  theory,  complex  systems  theory,  Monte  Carlo  simulations 
and evolutionary thinking, to bring together rules-based approaches (using heuristics), 
rationality, randomness and adaption using algorithms. ABM has been used in the study 
of  ecological  and  economic  systems,  traffic  bottlenecks  and  management,  social 
segregation, stock-market crashes and urban planning. It can help to better understand 
path  dependency  and  the  points  in  which  interventions  can  have  a  disproportionate 
impact.  In  complex  systems,  even  simple  decisions  can  have  volatile  and  extreme 
consequences. 

However, analysts need to be always vigilant regarding data quality and especially potential bias. For 
example: 

  Internet access and mobile phone ownership have reached high levels of penetration but are 
not yet 100%, and there is a danger that certain sections of the community are excluded by 
accident from data due to the ‘digital divide’ (see topic 5.4).  

  The  original  data  source  should  be  open to scrutiny,  to ensure  the  methodology  is  sound. 
The  case  of  Google  Flu  Trends  (GFT)  is  widely  cited.  Set  up  by  Google  in  2008,  GFT  used 
searches around the word ‘flu’, such as describing the symptoms, to predict where it would 
become  prevalent,  in  a  technique  known  as  ‘nowcasting’  (usual  real-time  data  for 
forecasting  purposes).  Compared  with  the  traditional  method  of  collecting  flu  data 
(reporting by doctor’s clinics), which naturally involves a time lag, GFT initially outperformed 
in  both  speed  and  accuracy.  However,  GFT  under-performed  seriously  at  the  peak  of  the 
2013 flu season and was closed in 2014. While Google was lauded for taking the initiative on 
flu  (and  subsequently,  dengue  fever),  the  search  methodology  and  hence  the  dataset 
remained  proprietary  and  hence  the  research  could  not  be  replicated  in  line  with  best 
scientific principles. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

28 

Practitioners can draw upon an array of analytical and visualisation tools. At the simplest end of the 
spectrum, desk-based analysis can be performed using standard spreadsheets (e.g. Excel) to produce 
and  present  descriptive  statistics  and  conduct  and  trend  analysis,  while  general-purpose 
visualisation  tools  (e.g.  Google  maps)  can  be  used  to  show  the  results  attractively.    At  the  most 
advanced  end  are  sophisticated  systems  for  statistical  and  predictive  modelling.  Realistically,  the 
sheer  volume,  frequency  and  complexity  means  big  data  analytics  is  often  unsuited  to  the  more 
traditional  systems.  Hence,  new  high-performance  storage  and  processing  solutions,  suitable  for 
huge  and  distributed  data  sets,  have  come  to  the  market,  often  using  open  source  software  (e.g. 
Apache Hadoop). 

For  Member  States,  statistical  offices  and  public 
institutes  (e.g.  economic,  environmental 
meteorological),  traditionally  tasked  with  collecting,  processing  and  holding  high  quality  data,  are 
often  well  placed  to  manage  the  process  of  accessing  new  sources,  such  as  social  media,  mobile 
phone,  sensor  and  satellite  data,  and  linking  them  to  conventional  sources  for  processing  and 
visualisation. They have the foundation of expertise, experience and equipment, which can be built 
upon. For example, the Dutch institute, Statistics Netherlands, has invested in pioneering work using 
three data sources - traffic loops, mobile phones and social media  – to investigate the possibilities 
for informing policy.  

Emerging data-driven projects in public bodies point to process and organisational challenges. First, 
data-driven  policies  cannot  be  built  without  a solid data  literacy  amongst  policymakers.  Second,  a 
strong data capability rests on agile  methodologies and strong data analytics competencies within 
the  organisation.  However,  recruitment  in  this  space  is  generally  a  challenge  for  public  bodies, 
where expertise is expensive and the private sector job market is highly competitive. Third, the skill 
challenge involves multi-disciplinary teams, and the process challenges that come with them. Finally, 
data  capabilities  necessarily  need  to  be  accompanied  by  robust  policy  and  decision-making 
processes that are able and open to absorb data-driven insights. 

Inspiring example: Test cases for big data (The Netherlands) 

In our modern world, more and more data are generated on the web and produced by sensors in the ever-
growing number of electronic devices surrounding us. The amount of data and the frequency at which they 
are produced have led to the concept of 'big data', which is often largely unstructured, meaning that it has 
no pre-defined data model and/or does not fit well into conventional relational databases.  

Apart from generating new commercial opportunities in the private sector, big data is also potentially very 
interesting  as  an  input  for  official  statistics;  either  for  use  on  its  own,  or  in  combination  with  more 
traditional  data  sources  such  as  sample  surveys  and  administrative  registers.  However,  harvesting  the 
information from big data and incorporating it into a statistical production process is not easy. 

At Statistics Netherlands, several big data case studies were performed. Data sources studied as potential 
input for statistics were: a) traffic loop detection data, b) mobile phone data, and c) social media messages.  

The findings are briefly described overleaf. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
   
 
 
Quality of Public Administration – A Toolbox for Practitioners 

29 

Traffic loop detection data   

In the Netherlands, approximately 80 million traffic loop detection records are generated a day.19 This data 
can be used as a source of information for traffic and transport statistics and potentially also for statistics on 
other economic phenomena. The data is provided at a very detailed level. More specifically, for more than 
12,000 detection loops on Dutch roads, the number of passing cars in various length classes is available on a 
minute-by-minute  basis.  The  downside  of  this  source  is  that  it  seriously  suffers  from  under-coverage  and 
selectivity. The number of vehicles detected is not available for every minute and not all (important) Dutch 
roads  have  detection  loops  yet.  Fortunately,  the  first  can  be  corrected  by  imputing  the  absent  data  with 
data  that  is  reported  by  the  same  location  during  a  5-minute  interval  before  or  after  that  minute20. 
Coverage  is  improving  over  time.  Gradually  more  and  more  roads  have  detection  loops,  enabling  a  more 
complete coverage of the most important Dutch roads. In a year, more than 2000 loops were added.   

A considerable part of the loops can discern vehicles in various length classes, enabling the differentiation 
between cars and trucks in three length categories: small (<= 5.6 meter), medium-sized (>5.6 and <= 12.2 
meter), and large (> 12.2 meter). The results after correction for missing data were used. The profiles clearly 
reveal differences in the driving behaviour of the vehicle classes. The small vehicles have clear morning and 
evening  rush-hour  peaks  at  8  am  and  5  pm  respectively.  The  medium-sized  vehicles  have  both  an  earlier 
morning and evening rush hour peak, at 7 am and 4 pm respectively. The large vehicle category has a clear 
morning  rush  hour  peak  around  7  am  and  displays  a  more  distributed  driving  behaviour  during  the 
remainder of the day. After 3 pm the number of large vehicles gradually declines. Most remarkable is the 
decrease in the relative number of medium-sized and large vehicles detected at 8 am, during the morning 
rush  hour  peak  of  the  small  vehicles.  This  may  be  caused  by  a  deliberate  action  of  the  drivers  of  the 
medium-sized and large vehicles to avoid the morning rush hour peak of the small vehicles.   

At the most detailed level (individual loops), the number of vehicles detected demonstrates (highly) volatile 
behaviour, indicating the need for a more statistical approach21. Harvesting the vast amount of information 
from the data is a major challenge for statistics. Making full use of this information would result in speedier 
and more robust statistics on traffic and more detailed information on the traffic of large vehicles, which is 
very likely indicative of changes in economic development.    

Mobile phone location data   

The  use  of  mobile  phones  nowadays  is  ubiquitous.  People  often  carry  phones  with  them  and  use  their 
phones throughout the day. Instrumental for the infrastructure enabling the coverage for mobile phones, 
are  mobile  phone  masts/towers,  called  ‘sites’  in  the  industry.  Those  sites  are  located  at  strategic  points, 
covering as wide an area as possible.    

Much of the activity that is associated with handling the phone traffic, 
i.e. handling the localisation of mobile phones, optimising the capacity 
of  a  site  is  stored  by  the  mobile  phone  company.  So  mobile  phone 
companies record data that are very closely associated with behaviour 
of people; behaviour that is  of interest to statistical agencies. Obvious 
examples  are  behaviour  regarding  tourism,  mobility,  commuting  and 
transport.  The  destinations  and  residences  of  people  during  day-time 
are topics of various surveys. Using data from mobile phone companies 
we  should  be  able  to  provide  additional  and  more  detailed  insight  on 
the whereabouts and the activity of mobile phone users.    

For our research, we obtained a dataset from a mobile telecommunication provider containing records of all 

19 P.J.H Daas, M.J. Puts, B. Buelens, and P.A.M. van den Hurk (2013), Big Data and Official Statistics, Paper for the 2013 
NTTS conference, Brussels.    
20 Daas et al (op. cit.) 
21 Daas et al (op. cit.) 

Topic 1.1: Qualities of good policy-making 

 
 
                                                           
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

30 

call-events (speech-calls and text messages) on their network in the Netherlands for a time period of two 
weeks.  Each  record  contains  information  about  the  time  and  serving  antenna  of  a  call-event  and  a 
scrambled  version  of  the  identification  number  of  the  phone.  This  study  revealed  several  uses  for  official 
statistics, such as economic activity, tourism, population density to mobility and road use.22    

Social media messages 

Around 1 million public social media messages are produced daily in the Netherlands. These messages are 
available to anyone with Internet  access. Social media has the potential of being a  data  source as people 
voluntarily  share  information,  discuss  topics  of  interest,  and  contact  family  and  friends.  To  respond  to 
whether social media is an interesting data source for statistics, Dutch social media messages were studied 
by Statistics Netherland from two perspectives: content and sentiment.    

Studies  of  the  content  of  Dutch  Twitter  messages  (the  predominant  public  social  media  message  in  the 
Netherlands at the time of the study) revealed that nearly 50% of messages were composed of 'pointless 
babble'.  The  remainder  predominantly  discussed  spare  time  activities  (10%),  work  (7%),  media  (TV  and 
radio; 5%) and politics (3%). Use of these more serious messages was hampered by the less serious 'babble' 
messages. The latter also negatively affected text mining approaches.    

Determination of the sentiment in social media messages revealed a  very interesting potential use of this 
data source for statistics. The sentiment in Dutch social media messages was found to be highly correlated 
with Dutch consumer confidence; especially with the sentiment towards the economic situation. The latter 
relation  was  stable  on  a  monthly  and  on  a  weekly  basis.  Daily  figures,  however,  displayed  highly  volatile 
behaviour.23  This  highlights  that  it  is  possible  to  produce  weekly  indicators  for  consumer  confidence  and 
could be produced on the first working day following the week studied, demonstrating the ability to deliver 
quick results. 

Challenges identified   

Our studies and the High-Level Working Group paper24 revealed several challenges/issues that need to be 
addressed. These fall into the following categories:   

  Legislative  -  with  respect  to  the  access  and  use  of  data.  The  right  to  access  administrative  data, 
established in principle by the law, is not adequately supported by specific obligations for big data. 
Many  potential  big  data  sources  are  collected  by  non-governmental  organisations  or  are  ‘freely’ 
available on the web; situations that may not be covered by existing legislation.   

  Privacy - managing public trust and acceptance of data re-use and its link to other sources. Privacy 
is generally defined as the right of individuals to control or influence what information related to 
them  may  be  disclosed.  The  problem  with  big  data  is  that  the  users  of  services  and  devices 
generating the data are most likely unaware that they are doing so, and/or what it can be used for. 
The data would become even bigger if they are pooled, as would the privacy concerns.  

  Financial - potential costs of sourcing data  vs. benefits. There is  likely to be a  cost to acquire big 
data,  especially  big  data  held  by  the  private  sector,  and  especially  if  legislation  is  silent  on  the 
financial modalities surrounding acquisition of external data.  

  Management - policies and directives about the management and protection of the data. Big data 
for official statistics means more information coming to national statistical institutes (NSIs) that is 
subject to policies and directives on the management and protection of the information  that NSIs 
must adhere to. Long-term stability may be a problem when using big data. Typically, statistics for 

22 E. De Jonge, M. van Pelt and M. Roos (2012), Time patterns, geospatial clustering and mobility statistics based on mobile 
phone network data, Discussion Paper 201214, Statistics Netherlands.  
23 Daas et al (op. cit.)  
24 High level working group (2013), Big Data and its Potential Use by the Statistical Community, Working paper for the High-
level group for the modernisation of statistical production and services meeting, d.d. March 10.  

Topic 1.1: Qualities of good policy-making 

 
 
                                                           
 
  
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

31 

policy  making  and  evaluation  are  required  for  extended  periods  of  time,  often  covering  many 
years. Many big data sources have only recently been ‘established’.   

  Methodological - data quality and suitability of statistical methods. When more and more data are 
being  analysed,  traditional  statistical  methods,  which  were  developed  for  the  very  thorough 
analysis of small samples, run into trouble; in the simplest case, they are just not fast enough. Since 
text  is  an  essential  part  of  many  big  data  sources,  the  need  to  extract  information  from  text 
increases.  Also,  the  subpopulations  covered  by  big  data  sources  studied  are  not  the  target 
populations for official statistics. Therefore, such data are likely to be selective, not representative 
of a relevant target population. Assessing representativeness of big data may prove problematic, as 
often there are no characteristics readily available to conduct such comparison. Next, including the 
information content of big data sources in the statistical production process (often without unique 
statistical ID keys) makes integration challenging.   

  Technological  -  issues  related  to  information  technology.  Dedicated  and  specialised  computing 
infrastructures are required to cope  with big data  to  enable processing and speed up analysis of 
large amounts of data. Certainly, for the exploratory phase, during which the content and structure 
of  big  datasets  must  be  understood,  fast  technology  certainly  speeds  up  this  process  and  more 
quickly enable the revelation of their use for statistics.    

Overall,  it  can  be  stated  that  the  work  described  above  revealed  that  there  is  a  need  for  new  legislation 
(enabling  access  to  big  data),  persons  with  new  skills  (statistical  aware  ‘data  scientists’25),  new  methods 
(specifically tailored to large data files and fast) and computational facilities that enable the speedy analysis 
of large data files while ensuring privacy26 . 

Vision   

The official statistics community is only scratching the surface when it comes to exploring the opportunities 
offered by big data. Moreover, at this moment, research activities related to big data are limited to isolated 
initiatives at some NSIs. In our opinion, the methodological and technological challenges mentioned above 
should be met in a big data research programme. Such a research programme should provide guidance and 
financial instruments for the following research, and should include the following topics:   

  Experimentation  with  big  data  sources  by  setting  up  a  number  of  pilot  projects  in  selected 
statistical areas. These pilots will provide guidelines for the effective use of big data for purposes of 
official  statistics.  Important  research  areas  include:  combining  big  data  with  traditional  data 
sources  (survey,  administrative);  replacing  traditional  data  sources,  i.e.  decreasing  administrative 
burden;  opportunities  for  new  output;  and  opportunities  for  faster  or  real-time  statistics 
production.   

  Development of new exploration and analysis methods, specific for the study of huge volumes of 

data, in the context of official statistics.  

  Further  experimentation  with  high  performance  computing  technologies  which  are  essential  for 

the processing of huge volumes of data.  

  Collaboration with third parties such as universities or IT/consulting companies with experience in 

the statistical analysis of large data sources.   

Big data is a highly multidisciplinary field requiring subject matter knowledge, strong maths skills as well as 
strong  programming  skills.  To  ensure  a  speedy  progress,  research  sub-projects  should  be  performed  by 
small,  highly  skilled  and  dedicated  teams  covering  such  expertise.  Moreover,  because  of  the 
multidisciplinary  character  of  the  research  programme,  guidance  could  be  provided  by  a  steering 
committee composed of experts in various statistical fields.   

25 M. Loukides (2011), What is Data Science? O’Reilly Radar report, O’Reilly Media Inc.   
26 P. Daas (2012), Big Data and official statistics, Sharing Advisory Board, Software Sharing Newsletter 7, 23.  

Topic 1.1: Qualities of good policy-making 

 
 
                                                           
 
 
 
   
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

32 

The  strategic  contributions  of  the  above  research  programme  consist  of  the  knowledge  and  experience 
gained  in  applications  of  big  data  for  official  statistics,  as  well  as  breeding  ‘data  scientists’  with  a  strong 
knowledge of official statistics, who will be an indispensable part of NSIs’ human capital in the future. 

Source: P.J.H Daas and M. van der Loo (2013), “Big Data (and official statistics)”, prepared by Statistics 
Netherlands, Meeting on the Management of Statistical Information Systems, 23-25 April, Eurostat, OECD 
Statistical Directorate, UN ECE Conference of European Statisticians and UN ESCAP  

The vision and challenges set out identified by Statistics Netherlands tally with key issues for policy-
makers identified by the data4policy research:   

  Understanding  the  scope:  Policy-makers  need  first  to  become  fully  cognisant  with  the 
potential  and  techniques  of  a  data-driven  approach,  to  use  it  appropriately,  to  select  the 
most  appropriate  sources  and  analytical  tools,  and  to  be  able  to  explain  the  process  and 
outputs  to  other  stakeholders.  They  also  need  to  understand  the  limitations,  as  data 
collection and analysis is not a panacea or a replacement for other research techniques, and 
can be just a trigger for further, more in-depth investigation. 

  Ethics:  The  use  of  data  sources  can  bring  up  questions  of  integrity  regarding  commercial 
confidentiality and privacy protection, especially in the context of sensitive sources, such as 
social media and mobile  phone  data, and can be  accentuated when two or more datasets 
are linked together with the effect of exposing individuals’ personal details and patterns of 
behaviour. These concerns should be handled within the context of Data Protection Law, but 
may also require higher levels of transparency, communication and consent-seeking in data 
collection and/or in data use. 

  Data skills: To embark on using big data for policy, you need to develop the in-house skills to 
collect  data,  analyse  it  using  the  various  tools,  and  interpret  the  results.  Even  the  initial 
brainstorming requires a level of data literacy. This issue is allied to the deficit of digital skills 
in  many  public  administrations.  To  use  the  simpler  analytical  and  visualisation  tools,  the 
necessary  expertise  (low  and  medium  level  skills)  can  be  obtained  relatively  quickly  and 
cheaply, including through self-learning. The more advanced ‘cutting edge’ systems demand 
high-level skills – computing degrees and several years of professional experience. Whatever 
investment  requires  management  commitment  to 
competence 
recruitment, training and development.  

level,  the  necessary 

  Data  access:  It  is  natural  to  lean  towards  using  readily  available  data,  but  policy-makers 
should always contemplate what would be the most relevant for the policy challenge under 
consideration, even if this takes more time and money to gather. Collaborations with other 
interested parties, including in other countries, can help to increase the dataset, spread the 
cost and increase the usefulness.  This may rely on partnerships with or procurement from 
private enterprises, including privately-held data and specialist data brokers and exchanges. 
Statistics  Netherlands,  for example,  receives  scanner  data  from  supermarkets  and  product 
prices  web-scraped  from  the  Internet,  which  provide  most  of  the  input  data  for  the 
Consumer  Price  Index.  It  also  purchased  social  media  sentiment  analysis  from  the  data 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

33 

broker  Coosto,  which  it  used  to  track  consumer  confidence27,  and  accessed  the  mobile 
phone location data from Vodafone. 

Big data is also being used to increase transparency and accountability, and to encourage societal 
pressure for change, by engaging directly with affected citizens. This is illustrated by the EU-funded 
DIGIWHIST initiative, which is supported by the EU’s Horizon 2020 and brings together six European 
research institutes with the aim of empowering society to combat public sector corruption. Led by 
the  University  of  Cambridge  (UK),  the  other  institutes  are:  DATLAB  (Czech  Republic);  Government 
Transparency  Institute  (Hungary);  Hertie  School  of  Governance  (Germany);  Open  Knowledge 
Foundation  Deutschland  (Germany);  and  Transcrime,  Università  Cattolica  del  Sacro  Cuore  (Italy). 
DIGIWHIST will run for three years (March 2015 – February 2018). 

DIGIWHIST – The Digital Whistleblower 

DIGIWHIST’s  goal  is  simultaneously  to  increase  trust  in  governments  and  improve  the  efficiency  of  public 
spending  across  Europe.  It  will  do  this  through  the  systematic  collection,  structuring,  analysis,  and  broad 
dissemination of information on public procurement and on mechanisms 
that  increase  accountability  of  public  officials  across  the  EU  plus  some 
neighbouring  countries  (Armenia,  Georgia,  Iceland,  Norway,  Serbia  and     
Switzerland). 

The project will compile and evaluate micro-level data using information from individual public procurement 
transactions and winning firms’ finance and ownership structures. This data will be linked to information on 
aggregate asset and income declarations data, to detect potential conflicts of interest in the system of public 
procurement, and more specifically, to identify systemic vulnerabilities in the respective legislations and their 
implementation. The ‘whistleblower’ dimension comes from citizen participation:  

“Imagine driving on a road which is full of potholes, in spite of having been repaired only a few months ago. 
DIGIWHIST  would  allow  you  to  instantly  identify  the  corresponding  government  contract  (e.g.  using 
geolocation), the construction company and the amount of public money involved. In addition, you would be 
able to take a photo of the potholes and attach them to a contract or organisation profile, thus contributing to 
the evidence of poor quality.” 

By  using  DIGIWHIST’s  mobile  app  and  web  portal  (www.digiwhist.eu),  citizens  can  file  reports  linked  to  the 
corresponding  procurement  information  concerning  contracts,  winning  companies  and  contracting  bodies 
directly to the respective public authority.  (DIGIWHIST is not intended, however, for general whistleblowing 
or uploading of documents). DIGIWHIST will collect micro-level public procurement data, and combine them 
with  company  and  other  datasets  in  an  open,  structured,  and  standardised  format.  By  marrying  big  data 
analytics  with  the  rich  local  knowledge  of  citizens,  the  aim  is  to  drastically  increase  the  capacity  to  hold 
governments  and  their  contractors  to  account.  In  addition,  DIGIWHIST  is  developing  an  observatory  of 
European  transparency 
legislation,  similar  to  national  procurement  portals,  called  European  Public 
Accountability  Mechanisms  (www.EuroPAM.eu).  This  comprehensive  and  structured  database  on  legal  and 
regulatory norms will cover: public procurement; conflict of interest; income and asset disclosure; and access 
to  information  and  open  data.  EuroPAM  is  an  extension  of  the  Public  Accountability  Mechanisms  Initiative 
(PAM) of the World Bank, which is a primary data collection effort that produces assessments of in-law and in-
practice efforts to enhance the transparency of public administration and the accountability of public officials. 
To ensure the reliability of in-law data, a rigorous and systematic approach is applied to data collection and 
analysis. Researchers produce summaries of the legal provisions collected from primary source documents, in 
the original language where possible. Following the preliminary analysis performed by researchers, the data is 
sent to technical in-country experts for feedback on accuracy and relevance. Country experts are intended to 

27 P.J.H Daas and M.J.H. Puts (2014), Social Media Sentiment and Consumer Confidence, European Central Bank, Statistics 
Paper Series No. 5. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

34 

have either in-depth legal knowledge of the mechanism being examined in a specific country or expertise in a 
related  field.  The  final  data  is  released  in  both  quantitative  and  qualitative  form  for  policy  and  research 
purposes, 

DIGIWHIST is also developing a website called www.opentender.eu to make public tenders more transparent, 
and an easy-to-use risk assessment software for public authorities, which will be based on the data collected 
and  the  indicators  developed  by  DIGIWHIST,  to  assess  corruption  risks  in  their  public  procurement 
procedures.  

1.1.3  Forward thinking 

in 

the 

complex 

Policy design 

longer-term 

Forward thinking 

future.  Foresight  uses 

Analysing big data to solve policy problems 

Increasingly,  governments  are  looking  to 
engage 
thinking  over 
horizons  of  typically  up  to  10-20  years  into 
latest 
the 
scientific  evidence  and  futures  analysis  to 
address 
issues  and  provide 
strategic  options  for  policy.  The  need  for 
anticipatory  functions  in  governments  comes  from  an  increased  awareness  of  the  complexity  of 
issues that policy must deal with, which requires a holistic and systemic approach and broader policy 
horizons.  Foresight  methods  and  processes  complement  quantitative  modelling  through 
qualitative and participatory methods involving all relevant stakeholders. They facilitate thinking 
out-of-the-box.  The  objective  is  to  engage  with  different  possible  futures  (e.g.  providing 
alternative futures) and challenge present assumptions. 

Consultation and co-responsibility 

Within  mainstream  public  administration,  foresight  units  came  to  prominence  at  the  end  of  the 
1990s,  usually  focusing  on  scientific  and  technological  development  and  their  implications  for 
research and innovation policies.  

Foresight contributing to EU policy-making 

The European Commission's Joint Research Centre (JRC)’s foresight activities within the EU Policy Lab28 are co-
designed  with  policy  DGs  to  deliver  insights  that  can  contribute  effectively  to  specific  policy  initiatives.  The 
activities explore the future of societal challenges and their potential implications on policy and society on a 
wide  range  of  issues  (e.g.  visions  for  European  manufacturing  sectors,  scenarios  on  the  future  of  the 
collaborative economy).  This  is in line with the new  emphasis on foresight and forward-looking tools that is 
outlined in the Better Regulation toolbox. All foresight processes have several mandatory characteristics:  

  They are highly participatory, engaging experts from different backgrounds as well as stakeholders from 
the  European  Commission's  policy  Directorates-General  (DGs),  industry,  industrial  associations,  research 
organisations,  universities  and  NGOs.  Foresight  can  play  different  functions  in  support  to  the  policy 
making process.  

  They  should  apply  foresight  tools  and  methods  (e.g.  scenarios,  vision  building,  trends  analysis,  Delphi 
surveys) to enable the analysis of a given issue within a systematic and systemic approach, facilitate inter-
service collaboration; consider emerging challenges and trends in technology and society, which could be 
otherwise overlooked. 

28 For an overview of the EU Policy Lab, please see also topic 1.1.1. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

35 

In  terms  of  the  main  functions  and  benefits  of  applying  foresight  to  policy  making,  four  ones  are  to  be 
highlighted (as illustrated in the table below):  

  Foresight  can  inform  policy  by  generating  insights  regarding  the  dynamics  of  change,  future  challenges 

and options that can be used as an input to policy conceptualisation and design.   

  A  second  function  is  to  facilitate  policy  implementation  by  enhancing  the  capacity  for  change  within  a 
given  policy  field,  by  building  a  common  awareness  on  future  challenges,  as  well  as  facilitating  new 
networks and visions amongst stakeholders.  

  A  third  function  is  related  to  embedding  participation  in  the  policy-making  process  by  facilitating  the 

participation of civil society.  

  Finally, foresight can support policy definition as it translates outcomes from the collective process into 
specific options for policy definition and implementation. All these functions contribute to reconfiguring 
the policy system in a way that makes it more apt to address long-term challenges. 

Function 
1. 

Informing policy 

Outcome 
Understanding of change 
Visions of change  

2. 

Facilitating policy 
implementation 

Networks, shared visions 

3.  Embedding participation in 

policy-making 
Supporting policy definition 

4. 

Transparency of policy-making 
process 
Generation of strategic options 
together with policy makers 

Benefit for policy 
Long-term orientation 
Additional source for information (based on a 
broad variety of views) 
Awareness of future challenges 
Better receptivity of actors for policy objectives 
due to ownership of results, therefore easier 
implementation 
Better identification of citizens with policy 
(legitimacy) 
Direct support in strategy development and 
implementation 

Rather than establish permanent units, some Member States conduct futures research that is time-
limited,  but  wide-ranging  and  far-reaching  in  scope,  such  as  Finland’s  futures  reports,  which  have 
been an integral element of the Parliamentary cycle for over 20 years. The latest Finnish analysis to 
2030 is pan-Governmental and connected to wider networks and expert sources. During preparation 
of the latest report, the Prime Minister’s Office announced it was contemplating a more permanent 
arrangement  to  establish  a  foresight  model,  “to  provide  Finnish  decision-makers  with  the  best 
possible perspectives into the future”. This would include: appointing a Foresight Group comprising 
permanent  and  non-permanent  members,  tasked  with  coordination  and  innovation  relating  to 
Finnish  foresight  activities;  creating  a  national  foresight  network;  inviting  ‘foresight  actors’  to 
convene at regular foresight forums; commissioning an international foresight report to complement 
the  national  one;  providing  training  on  foresight  expertise;  and  the  possibility  of  an  online  portal 
known  as Tulevaisuuskartasto.fi  (“the  future  atlas”), for  the  distribution  of  foresight  data,  analysis 
and discussion. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

36 

Inspiring example: Government Report on the Future (Finland) 

Once in  each electoral period, the Government  of Finland  submits  its Foresight Report  to  Parliament on the 
long-term perspectives and options faced  by society relating to policy decisions to be taken in a  15-20 year’ 
period, with the aim to encourage a broad debate in society. The Prime Minister's Office is responsible for the 
Government Foresight Report and promoting the implementation of policies within the given time frame. The 
last ‘Government Report on the Future’ was adopted by the Government in October 2013, focusing on well-
being  and  sustainable  growth  to  2013.  The  report  was  not  an  action  programme,  but  it  instead  sought  to 
highlight factors and development paths that would facilitate sustainable growth in the future. 

Preparation  of  the  report  was  led  by  a  Government-appointed  ministerial  working  group  representing  all 
parties in Government and chaired by the Minister of Economic Affairs. For the first time, a separate foresight 
phase formed part of the report’s preparation, with the purpose of seeking new directions for Finland in a new 
way. The foresight phase was carried out as a collaborative exercise between the Prime Minister’s Office, the 
Finnish Innovation Fund Sitra, the Academy of Finland, and Tekes, the Finnish Funding Agency for Technology 
and Innovation, alongside a host of independent specialists and experts from research institutions, enterprises 
and  NGOs.  Extensive  analysis  material  was  produced  for  the  range  of  themes  subjected  to  foresight  work: 
participating organisations’ material on trends and drivers, an extensive analysis and a summary of global and 
domestic research and analysis reports, plus a questionnaire making use of social media.  

Discussions were held on the report website at www.2030.fi, and regional discussion events, led by ministers, 
were  organised  in  seven  cities  in  the  autumn  of  2012,  in  which  citizens  were  urged  to  come  forward  with 
ideas,  and  to  discuss  and  ponder  Finland’s  future  and  the  possibilities  that  lie  ahead.  The  results  were 
published in February 2013. Use was also made of the preliminary results of the ‘Sustainable Growth Model’, 
an independent international research project that was carried out concurrently. Expert workshops and broad-
based crowd sourcing were utilised in selecting the themes. As well as four horizontal themes (flexibility and 
crisis  resilience,  skills  and  competences,  use  of  ICT,  and  global  perspective),  the  end-result  comprised  six 
content themes: 

  Public administration as an enabler;  
  Citizens' well-being and inclusion;  
  Working life in the future;  
  Business regeneration;  
  A new geography for the North; and  
  Opportunities in the midst of scarcity. 

The  report’s  key  findings  include  that:  the  present  trend  growth  trajectory  will  not  provide  sustained  well-
being  for  the  ageing  population  in  2030;  a  new  approach  towards  structural  change  and  renewal  is  needed 
throughout Finnish society; the growth sectors or areas of sectors cannot be selected, but an environment that 
is conducive for  sustainable  growth can be established; half of the  value  created  will  be digital in the 2030s 
(the exact figures or  dates are of lesser relevance); and resilience against  shocks and ‘black swans’ will be a 
main condition for wealth creation in 2030; the economy that re-establishes itself first after global or regional 
shock can gain and re-invent itself more smoothly. 

In addition to the Government, Parliament participates in the consideration of the report, which also provides 
issues  for  a  broader-based  debate  within  society.  The  Parliament  has  a  specific  ‘Committee  for  the  Future’, 
established  in  1993,  whose  main  task  is  to  respond  formally  to  the  Foresight  Report.  The  Committee  also 
deliberates  on  parliamentary  documents  and  make  submissions  to  other  committees  on  futures-related 
matters within their spheres of responsibility, as well as conducting research associated with futures studies, 
including  their  methodology.  The  Committee  also  functions  as  a  parliamentary  body  that  conducts 
assessments of technological development and the effects on society of technology.  

The  Parliament  concurred  with  the  main  findings  of  the  Foresight  report  in  its  formal  response  to  the 
Government in autumn 2014. The Resolution that was drafted by the Committee of the Future after hearings 
in six other committees, underlines for example the importance of experimentation in government and further 
development  of  the  participatory  foresight  procedures.  A  Cabinet  meeting  responded  formally  to  the 
Resolution  and  forwarded  it  to  relevant  ministries  for  action,  in  particular  within  the  perspective  of  spring 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

37 

2014  elections  and  subsequent  government  programme  preparations.  The  ministries  also  published  their 
future  reviews  in  autumn  2014,  this  being  the  fourth  time.  The  reviews  are  also  designed  as  background 
documentation for the next government. 

The current national foresight model is an outcome of the implementation phase of the last foresight report 
in 2013. 

The  government  that  was  appointed  in  May  2015  has  started  preparing  the  new  report  on  the  future.  The 
theme  for  the  current  electoral  term  will  be  the  Transformation  and  the  Future  of  Work.  The  political 
preparation  is  done  in  the  government’s  strategy  sessions.  The  Economic  Council,  which  is  chaired  by  the 
Prime  minister  and  includes  key  ministers,  the  Governor  of  Bank  of  Finland  and  representatives  from  the 
central labour market organisations (employers and employees) acts as advisory forum in the process. 

The Government Report on the Future will be completed in two stages: 

  Part 1 (2017): A shared understanding of what does the work and working life look like in 2036 
  Part 2 (2018): Capacity building for the systemic change of work and its impacts in Finnish society 

The  foresight  process  relies  on  the  national  foresight  model,  on  existing  networks  and  on  the  foresight 
cooperation between ministries. The Prime Minister’s Office and Sitra, The Finnish Innovation Fund coordinate 
the national foresight network and support foresight activities and networking in Finland. The network brings 
together Finnish foresight specialists and acts as a discussion and coordination forum for foresight actors. The 
purpose  is  to  foster  public  discussion  and  research  on  the  new  challenges  and  opportunities  facing  Finnish 
society and to promote the use of foresight knowledge and futures perspectives in decision-making. 

The foresight steering group, which is a key component of the of the new foresight model, was appointed  in 
2015 by the Prime Minister’s Office.  The Group consists of Finnish top foresight experts and it supports and 
guides  the  national  foresight  cooperation  and  provides  methodological  and  scientific  guidance  in  the 
government’s foresight work. 

For further information: Dr. Kaisa Oksanen, Senior Specialist, Prime Minister’s Office, kaisa.oksanen@vnk.fi  

Forward planning implies a break with existing patterns of development and hence will most likely 
meet  some  resistance,  as  there  will  be  interested  parties  that  might  lose  out  from  change,  even 
when the cumulative benefits for economy and society exceed individual costs. However, these long 
planning  horizons  have  the  advantage  of  allowing  greater  time  for  adjustment  than  conventional 
policy  timescales,  including  investing  in  research  and  infrastructure,  and  building  capacity  within 
both  the  public  administration  and  business  community.  Europe’s  experience  with  seismic  policy 
changes in the past has shown that industry is able to find the technological solutions, and to adjust 
business models and investment plans accordingly, if the following ingredients are in place. 

Smoothing the path to forward planning 
  A  period  of  consultation  and  reflection,  to  understand  the  implications  for  affected  parties  (usually 

business) and take them on board; 

  An  unambiguous  policy,  based  on  a  clear  statement  of  intent  and  unwavering  commitment  from  the 

public administration, which requires leadership from the top; 

  A ‘level playing field’ to ensure fairness in the policy’s application, including sanctions for non-compliance; 
  Sufficient  time  to  adjust,  for  example  to  find  technological  solutions,  adjust  business  models,  access 

investment finance, develop requisite skills and competences, etc.  

In  the  past,  such  policy  shifts  have  often  emerged  from  environmental  risks  and  dangers  (for 
example, banning CFCs and reducing toxic engine emissions). It can be easier to create a consensus 
around forward plans, including internationally, when faced with a clear prospective crisis, such as 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

38 

droughts  or  flooding  caused  by  climate  change,  ageing  populations,  financial  instability,  etc.  The 
focus of foresight should not be forecasting the future, but shaping it - a process of experimentation, 
not simply extrapolation. It is about having a vision for where the country would like to be in 10, 20, 
30 years’ time, setting out on the journey, and finding the incremental steps and sometimes huge 
leaps that are needed to get there. This means that the public administration must be willing to stop 
and  check  position  regularly,  and  change  direction  if  necessary,  in  a  series  of moves  to  get  to  the 
ultimate destination. If events on the way means the end-point is no longer attainable or desirable, 
then the plan itself must be reconsidered. Whatever happens, the journey will only be successful, or 
indeed gain any momentum at all, if citizens and businesses are brought along too. In the journey to 
achieve  the  high-level  objective  (such  as,  for  example,  fossil  free  road  transport)  –  the  public 
administration is the Sherpa, in service to the public. 

In  converting  plans  into  action,  strategy  documents  can  guide  all  interested  parties,  inside  and 
outside the administration, to deal with deep-rooted challenges that require medium-to-long term 
planning horizons. The word strategy comes from the Greek for ‘general-ship’ and is about how best 
to organise resources and direct operations to achieve the desired outcome, originally to a military 
objective.  

This rationalisation of resources is undermined when a country has a plethora of strategy documents 
that  are  overlapping  in  coverage  and  timescales,  and  inconsistent  with  each  other.  One  medium-
sized Member State with a population below 10 million has over 200 national strategy documents 
alone,  including  multiple  strategies  within  the  same  sector  (health,  education,  environment, 
emergencies,  etc.),  which  argues  for  streamlining  to  provide  a  coherent  framework  for  follow-up 
actions.  The  following  checklist  provides  seven  criteria  for  assessing  the  quality  and  internal 
consistency of individual strategies. 

Criteria 

Scope 

Analysis 

Key questions 

  Does the strategy set out its boundaries, and is explicit about its coverage (what falls 

inside and outside its scope)? 

  Are the meanings of key terms clearly defined, avoiding any ambiguity, and consistent 

with other documents from the public administration? 

  Does the strategy describe links to any other national, regional or local strategies that are 

relevant to its performance? 

  Does the document refer to existing laws, treaties or other international obligations, 
institutions and stakeholders that provide the context for the strategy, or might be 
affected by it? 

  Does the strategy set out the evidence base clearly and comprehensively, present a 
rounded picture of the challenges facing the sector and any contextual factors?  

  Are any statistics used the most recent available (as relevance deteriorates with time) and 

qualified by definitions, sources and interpretation? Do they include data series, to 
discount any one-off blips or irregularities, and projections (where this is both feasible and 
credible) with all underpinning assumptions and caveats?  

  Are trends and patterns assessed and placed in the context of wider socio-economic and 
contingent factors, including international comparisons where they are relevant and 
provide useful benchmarks? 

  Does qualitative information include stakeholder consultations and the views of 

independent commentators, if available? 

Vision  

  Does the strategy set out an achievable vision of the desired future state at the end of the 

period, in the form of the ultimate outcomes for beneficiaries (rather than inputs, 
processes or intermediate steps)? 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

39 

Criteria 

Key questions 
Is this vision articulated as a set of complementary objectives which are unambiguous, 
follow logically from the analysis, can be achieved with the available resources? 

  Do the objectives form a balanced and cohesive whole (the sum of their effects should 

contribute jointly to accomplishing the vision)? 

Measures 

  Are the objectives translated into shorter-term operational solutions, in the form of 

measures, each with their distinct rationale? 

  Does the choice of measures reflect lessons learned from past practice, including 

interventions to be built upon and mistakes to be learned from?  

  Does the strategy consider all appropriate public policy instruments in designing 

measures? Have the pros and cons of different options been assessed for their likely costs 
and consequences, especially impact and sustainability? (see topic 1.2).  

  Does the strategy describe the underlying assumptions, pre-conditions and risks affecting 

the prospects for its measures? 
If the strategy is a ‘road map’, is it clear about the direction of travel, the ultimate 
destination, and the milestones that can be used to measure progress? 

  Do monitoring indicators avoid being captured by ‘quantification’ (counting what can most 

easily be counted)? 
Is the strategy sufficiently flexible to adapt to evolving circumstances which cannot 
reasonably be anticipated? 

  Does the strategy demonstrate that it is widely accepted by affected parties (public 
bodies, citizens, businesses, socio-economic partners and civil society), including 
summarising the consultation process (possibly as an annex)? 

  As it might outlast one electoral cycle, is there a political consensus around the systemic 
problems being addressed and the selected solutions, which crosses party boundaries?  
Is the strategy as succinct as possible, clear in its use of language, and easy to read? 
  Does the document flow logically from analysis to vision / objectives to measures to 

implementation?  

Adaptability 

Ownership 

Presentation 

1.1.4  User-centred consultation and co-responsibility (co-creation) 

and 

citizens, 

businesses 

Policy design 

Policy-makers increasingly recognise the role 
that 
other 
interested  parties  can  and  should  play  in 
designing  policy.  It  is  well  accepted  that 
designing  user-centred  policies  and  services 
increase 
All 
stakeholders  have  a  say  in  the  success  of 
public sector governance, insights that are not available to the administration, and a potential role in 
implementation. Forward-thinking administrations look to capture these perspectives in their policy 
development.  

Analysing big data to solve policy problems 

Consultation and co-responsibility 

Forward thinking 

ownership 

trust. 

and 

As the innovation charity NESTA puts it, one of the priorities for “modern government is to ensure 
that  all  the  actors  in  the  system  are  aware  of  the  available  evidence  about  what  works,  both  for 
policy  and  practice”.  It  is  incumbent  on  public  administrations  to  identify  and  present  the  best 
knowledge they have available, to ensure decisions are well-informed and inclusive. Citizens are no 
longer  just  occasional voters,  or  regular  but  passive service  recipients,  they  are  increasingly  active 
co-creators of public policy. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

40 

“Citizens wish to be addressed and involved in an open way by the public administration, from policy 
making  to  public  service  delivery.  They  do  not  want  a  traditional  civil  service  that  devises  and 
implements  goals  and  solutions  from  their  offices:  instead  they  want  a  civil  service  that  takes 
citizens seriously and works together with them and shares information and data”, EUPAN, Strategy 
Paper, op. cit.  

Public service providers and their clients often see 
more  clearly  than  policy  officials  the  situation  ‘on 
the  ground’,  what  is  needed,  what  has  worked  in 
the  past  or  not,  and why.  They can  spot  potential 
obstacles and pitfalls, and steer officials away from 
expensive  and  embarrassing  errors 
in  policy 
implementation  at  a  later  stage.  The  consultation 
of  the ultimate  beneficiaries  of public  policy,  both 
citizens  and  businesses,  should  provide  crucial 
inputs throughout the policy process. The interests 
of  good  governance  are  served  by  the  intended 
beneficiary  being  integral  to  all  steps  in  policy-
making, not just as an end recipient of government 
programmes, funds or services.  

As  an  example,  the  Small  Business  Act  (SBA)  has  made  SMEs  and  their  representatives  pivotal  to 
policy-making  at  the  European  level.  The  SBA  commits  the  European  Commission,  and  invites 
Member States, to consult stakeholders, including SME organisations, for at least 12 weeks prior to 
making any legislative or administrative proposal that has an impact on businesses (see also theme 
6). The preparation of the SBA itself was subject to a public hearing and online consultation. 

European Commission consultation with small businesses 

The Small Business Act has  established  strong governance mechanisms based on the close cooperation  with 
Member States and SME stakeholders. The implementation of the SBA is now supported by the SME Envoys, a 
network of high-level representatives from Member States.  The nomination of a single point of contact for all 
issues  related  to  the  SBA  in  the  Member  States  has  reinforced  the  application  of  its  principles  and  allows 
Member  States  to  exchange  best  practices.  To  involve  stakeholders  directly,  representative  SME  business 
organisations at European level participate as observers in the meetings of the network. These activities aim to 
ensure  that  regulatory  burden  reduction  becomes  a  priority  in  the  Member  States  through  an  enhanced 
sharing of best practices. For example, the Network has been instrumental in reducing the time to start-up a 
business  in  Europe  (see  theme  6).  Furthermore,  the  Commission  has  proposed  that  the  appointment  of  an 
SME Envoy and the implementation of the SME Test by Member States are introduced as criteria for Member 
States to receive SME-related support from the European Regional Development Fund (see theme 8). 

Regular  annual  meetings  between  SME  associations  and  the  Commission  are  also  now  held  to  identify  and 
monitor  SME  relevant  priority  initiatives  in  the  Commission  Work  Programme  for  SME  impacts.  The 
Commission  is  using  the  Enterprise  Europe  Network  (EEN)  to  consult  SMEs,  including  micro  enterprises, 
directly on forthcoming legislation ('SME Panel' consultation) and to collect their feed-back on the existing EU 
legislation  ('SME  feed-back'  database).  Business  organisations  and  Member  States  have  welcomed  such 
developments as important for SME policy.   

In  addition,  the  Commission  has  organised  conferences  with  SMEs  from  Germany,  Italy,  the  Netherlands, 
Poland, Sweden, and the UK. These conferences allowed entrepreneurs from SMEs to raise their concerns, in 

Topic 1.1: Qualities of good policy-making 

 
 
  
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

41 

different  areas  like  labour  law,  the  regulation  of  the  marketing  of  products  and  the  related  process  of  the 
setting of European product standards confirming the compliance of products with regulatory requirements, 
health  and  safety,  environment,  VAT  and  food  hygiene  and  labelling.  The  conferences  also  allowed  face-to-
face discussion and the exchange of detailed information and positions.  

The  Commission  is  also  consulting  SME  employers’  organisations  regularly  through  EU  social  partner 
consultations  and  through  the  work  of  European  social  dialogue  committees.  SME  associations  have  been 
contributing  actively  to  the  definition  and  implementation  of  the  work  programme  of  the  European  social 
partners.  

Finally,  the  SME  dimension  became  a  focus  of  the  High-Level  Group  of  Independent  Stakeholders  on 
Administrative  Burdens  (HLGAB),  an  expert  group  created  in  2007  and  chaired  by  Dr  Edmund  Stoiber,  the 
former Prime Minister of Bavaria, to advise the Commission on reducing administrative burdens resulting from 
EU  legislation.  The  Group  advises  on  EU  regulatory  measures  adopted  by  Parliament  and  Council  under  the 
Administrative  Burden  Reduction  Programme  and  look  into  how  Member  States  have  implemented  these 
measures.  SMEs will be consulted on the extent to which the measures taken have made a real difference for 
them.  This  work  facilitates  the  exchange  of  information  between  Member  States  on  different  ways  of 
implementing EU legislation and contributes to better understanding of the final impact of measures adopted. 
The HLGAB issued two landmark studies in 2011 and 2014. 

Consultation  is  appreciated  by  citizens  and  businesses,  so  long  as  it  involves  genuine  engagement 
with sufficient time and input to make a productive contribution, rather than as a tokenistic exercise 
or where consultees are presented with a highly restrictive set of options and feel ‘railroaded’ into a 
decision.  A  mix  of  methodologies  can  be  deployed,  such  as  meetings,  written  submissions,  focus 
groups and surveys (see also topic 5.1) to achieve a good mix of breadth and depth of coverage. It is 
important to ensure the consultees are broadly representative, not just the most vocal or the most 
inclined  towards  one  channel  or  another,  and  to  avoid  the  ‘tyranny  of  the  majority’,  where  the 
dominant view overrides the legitimate concerns of the minority. Professional judgement will still be 
required to interpret the findings and feedback, to reconcile dilemmas and to balance the best use 
of resources, as with all forms of policy-making.29  

Some  Member  States  have  adopted  national  standards  for  stakeholder  consultation,  such  as 
Austria’s ‘Standards of Public Participation’, and the UK’s ‘Code of Practice on Consultation’, through 
inter-ministerial working groups and the involvement of NGOs, external experts and interest groups.  

The  example  of  public  consultation  over  the  Development  Strategy  of  the  Malopolska  Region  for 
2011-2020  in  Poland  shows  the  value  of  using  multiple  mechanisms,  including  offline  and  online 
media, to draw in the community and connect with as many residents as possible.  

Inspiring example: Malopolska 2020:  unlimited opportunities – unlimited debate (Poland) 

In  parallel  to  the  work  on  the  consecutive  drafts  of  the  Development  Strategy  of  the Malopolska  Region  for 
2011-2020 (DSMR), the Marshall Office of the Malopolska Region carried out a broad regional debate on the 
future of Malopolska. Public consultations included a range of initiatives and events aimed at opening up the 
preparations  for  the  new  strategy  to  the  region’s  inhabitants,  from  October  2009  (following  approval  and 
publishing  of  the  DSMR  update  guidelines)  until  September  2011  (final  adoption  of  the  Strategy  by  the 
Regional Assembly of the Malopolska Region).  

29 For more extensive insights, see E. Loeffler and S. Martin (2016), Citizen Engagement, in ‘Public Management and 
Governance’, edited by T. Bovaird and E. Loeffler, Routledge. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

42 

Using communication tools resulted in DSMR consultations within the framework of a multi-platform regional 
debate about the future of Malopolska, lasting several months. Over the course of the debate, all stakeholders 
representing social, professional and local community groups, as well as the inhabitants of the region, had the 
opportunity to express their opinions about the document and suggest changes.  At each stage of the work on 
the  strategic  document,  they  could  familiarise  themselves  with  the  content  using  the  specific  website 
dedicated  to  the  process.  Due  to  the  range  of  activities  undertaken,  it  was  decided  formally  to  extend  the 
timeframe  beyond  the  35-day  term  designated  by  law.  This  increased  the  fundamental  quality  of  the 
document, but also raised awareness among the intended audience.  

The  events  and  initiatives  organised  during  the  course  of  the  regional  debate  related  to  the  work  on  the 
Development Strategy of the Malopolska Region for 2011-2020 were as follows:  

1.  Public events:  

a)  Public  consultations  through  the  Internet:  enabling  stakeholders  to  acquaint  themselves  with  the 

document and to communicate their opinions, as well the updated the DSMR draft;    

b)  Territorial consultations consisting of five meetings located in the centres of Malopolska sub-regions. 
These  meetings  allowed  guidelines  to  be  drafted  and  agreed  on  individual  conduct  for  the 
development of Malopolska in the sub-regional dimension;    

c)  Thematic  consultations  consisting  of  conferences  and  seminars  on  the  key  areas  of  the  strategy: 

higher education, culture, rural areas, cities, economy and entrepreneurship;  

d)  Promotional campaign involving broad use of the media: regional television, both public channel TVP 
Krakow and Internet television of the Marshal Office, as well as the regional and local press, Internet 
(Facebook  and  an  Internet  competition  ‘My  idea  on  Malopolska’,  where  any  Malopolska  resident 
could submit an initiative aimed at improving the life of local or regional community or development 
of the region).  

2. Experts’ activities, which included discussion on the DSMR drafts within the framework of the DSMR 2020 
update  team;  Malopolska  Council  for  Observatories  of  Regional  Development  and  Evaluation;  Malopolska 
Innovation Council: Malopolska Council for the Public Benefit; Regional Urban and Architectural Commission.  

3. Discussion within formal bodies: Regional Assembly’s Committees and Regional Board; Malopolska Mayor’s 
Forum  and  Convent  of  the  Poviat  Starostes  of  Malopolska  Region;  Joint  Commission  of  the  Central 
Government and Territorial Self-Governments.  

Around 1500 of the region’s residents took part in the public participation on the updating of the DSMR for the 
years 2011-2020. 

For further information: Jeremiasz Salamon, Manager, Unit for Strategic Planning, Department of Regional 
Policy, Marshal Office of the Małopolska Region, Jeremiasz.Salamon@umwm.pl  

Public administrations are increasingly taking e-Participation on board, as citizens use governmental 
websites  and  social  media  to  convey  their  expectations  to  policy-makers.  ICT  offers  new  tools  to 
better  engage  with  citizens  and  businesses,  and  gather  evidence  to  improve  the  impact  of  policy. 
This is a worldwide phenomenon. At the EU level, the Commission gathers all its public consultations 
at a single access point. 

Participatory governance through online platforms 

Over the last few years, a significant number of countries have been adopting citizen inclusion as part of their 
eGovernment  agenda,  leveraging  multiple  technology  channels  to  enable  e-Participation  e.g.  through  online 
surveys or feedback forms, chat rooms, listservs, newsgroups and social media such as Twitter and Facebook.  

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
  
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

43 

Some of these initiatives include:  

  Have Your Say section (National Portal), Australia – citizens can send their inputs on draft regulations 

to the respective ministry by email (http://www.australia.gov.au); 

  eGovernment Site, Brazil – Forum section allowing users to send comments regarding accessibility 
and integration of services and also contains a public consultation section on draft regulations 
(http://www.governoeletronico.gov.br); 

  e-Democracy Site, Hungary – Government officials/agencies respond to citizens’ comments and 

conduct moderation activities (https://edemokracia.magyarorszag.hu); 

  Ministry of Education and Ministry of Health websites, Mozambique – online discussion forums for 

users’ inputs on policy issues regarding education (http://www.mec.gov.mz) and health 
(http://www.misau.gov.mz); and 

  e-Petition (National Portal), UK – citizens can lodge online petitions for Governments to propose to 

Parliament if enough signatures are acquired (http://www.direct.gov.uk). 

With  an  increasing  number  of  people  using  social  networking  in  their  personal  lives,  online  platforms  are 
becoming powerful tools for engagement between governments and their constituents. 

Source: PwC (2013), “Future of Government: Tomorrow’s leading public body”  

As  a  form  of  civic  activism,  these  web-based  platforms  are  often  initiated  by  civil  society 
organisations (CSOs), but implemented with the agreement and typically active involvement of the 
public  administration  in  shaping  the  agenda.  Their  purpose  is  to  enable  (self-selecting)  citizens  to 
feed their views on pressing matters of public policy and services, usually to sub-national authorities 
at  the  city  or  municipal  level.  Examples  from  the  US  include:  Open  Town  Hall,  which  is  a  “cloud-
based online civic engagement  platform  that  augments and diversifies public participation in ways 
that also enable government leaders to increase public trust in their governance”; and online Citizen 
Report  Cards  (CRCs),  which  were  originally  pioneered  in  paper  form  in  1999  to  facilitate  citizen 
participation in raising service standards in Bangalore India, but have since been taken up in e-form 
in the highly prosperous community of the State of California. 

Co-design 

Increasingly, administrations are looking to move 
from  consultation  to  co-responsibility,  giving 
citizens  and  businesses  a  much  greater  stake  in 
policy-making,  and  sharing  ownership  of  policy 
decisions  with  the  community  that 
is  most 
affected  by  them.  The  traditional  model  of 
policy-making 
is  administration-centric  and 
hierarchical:  political  and  administrative  leaders 
determine what programmes and services are to 
be  provided,  on  what  terms  and  to  whom,  and 
officials and professionals subsequently organise 
and  deliver  them.  The  role  of  the  intended 
beneficiary  is  largely  passive.  Recent  years  have 
seen a paradigm shift, however, with a growing range of actors involved institutionally or on an ad 
hoc basis in the design, production, delivery and evaluation of public policy, and the role of citizens 

Co-
responsibility 

Co-
production 

Co-
evaluation 

Co-decision 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

44 

and businesses has become more and more active.30 This implies that public agencies evolve from 
closed,  self-centred  service  providers  to  open  networking  organisations  that  the  public  can  trust. 
This occurs through transparent processes  and accountability, developing the  democratic dialogue 
from  an  internal  focus  (resources  and  activities)  to  an  external  one  (outputs  and  outcomes), 
involving  stakeholders  in  every  step  of  the  policy  process.  Citizens  and  businesses  become  co-
designers, co-deciders, co-producers and co-evaluators.31  

In  the  spirit  of  co-decision32,  policy-makers  are  specific  seeking  to  engage  with  the  citizens  and 
businesses that will be affected by legislation, inviting their inputs in the shaping of new laws and 
regulations. This includes e-Participation tools at the EU level, such as  European Citizens’ Initiative 
(ECI) which allows EU citizens to participate directly in the development of EU policies, by calling on 
the European Commission to make a legislative proposal. 

Co-design  is  also  known  as  ‘co-creation’  and  is  well  illustrated  by  the  Basque  Government’s 
approach to housing policy, in which citizens were invited to participate in the housing master plan, 
shaping the housing law and influencing policy management.  

The  Basque  approach  to  citizen  participation  has  been  recognised  by  the  United  Nations  Public 
Service Award in 2012 and by the  International Observatory on Participatory Democracy (IOPD)  in 
2016.  The  IOPD  is  a  worldwide  network  of  more  than  600  cities,  associations,  organisations  and 
research centres  interested in learning about participatory democracy on a local scale, exchanging 
impressions  and  know-how,  and  applying  experiences,  with  the  aim  of  deepening  the  roots  of 
democracy  in  government  and  enriching  public  policies.  This  network  was  created  within  the 
framework of the Decentralised Cooperation Projects of the European Union’s URB-AL Programme 
(547  cities  and  52  countries  networked,  310  local  governments,  237  universities,  research  centres 
and associations). 

Inspiring example: Creating housing together (Spain) 

One of the Basque Government´s priorities for the 2009-2013 legislature was to apply the principles of good 
governance  to  housing  policy.  The  principles  of  consensus,  transparency  and  participation  are  particularly 
noteworthy. Housing is one of the main concerns of Basque citizens, which is only exceeded by unemployment 
and the current economic situation.  

In  the  context  of  the  deep  financial  and  economic  crisis  with  high  unemployment  rates,  the  contraction  in 
credit  availability  and  restrictions  on  public  resources,  the  Basque  Government  considers  it  more  necessary 
than ever to join forces and to search for  a wide consensus to ensure all citizens have the right of access to 
adequate housing. Due to this public concern, and the need to define a new housing policy and make essential 
legislative  changes  to  apply  it,  the  Department  of  Housing,  Public  Works  and  Transport  of  the  Basque 
Government designed and implemented a participative process structured into three public participation sub-
processes, through which it aimed to foster the full participation of the Basque Society in drawing up housing 
strategy, policy lines and legislation: 

  Housing  Social  Pact:  Basque  Housing  Strategy  2010-2025.  The  target  population  was  institutional, 

social and political agents (December 2009 - June 2010). 

30 N. Thijs and P. Staes (2009), European Primer in Customer Satisfaction Management, EIPA, Maastricht. 
31 N. Thijs (2011), Measure to improve. Improving public sector performance by using citizen-user satisfaction information, 
EIPA, p.104 
32 Co-production is explored further in topic 1.2, and co-evaluation in topic 1.3. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

45 

  Housing and Urban Regeneration Master Plan 2010 – 2013, “On Housing, your opinion counts”: The 

target population was citizens and experts’ groups (April 2010 - July 2010). 

  Basque Housing Law: The target population was citizens (January 2011- May 2011). 

The  overall  objective  was  to  promote  the  participation  of  the  Basque  population,  to  enhance  the  strategy, 
policy  lines  and  legislation  that  may  help  to  facilitate  access  to  housing  for  Basque  citizens.  Other  specific 
objectives were: to find out the opinion of the Basque people in relation to housing strategy, policy lines and 
legislation;  to  receive  proposals  that  may  enhance  the  strategy,  policy  lines  and  legislation  proposed  by  the 
Housing  Department;  to  establish  long-term  cooperation  relations  between  the  institutional,  economic  and 
social agents involved in the housing market in order to help fulfil the desired objectives; to take advantage of 
the  potential  of  the  new  technologies  to  promote  the  participation  processes;  to  achieve  a  high  level  of 
participation in the process; and to achieve a high level of satisfaction with the process.  

The  initiative  was  a  social  innovation  process,  because  it  generated  value  (social  benefit)  for  the  Basque 
society in a field which is of prime concern for citizens, in addition to being one of the fundamental rights of a 
modern and united society, and an open innovation process of transparency, plurality and client orientation, 
because  it  takes  advantage  of  joint  intelligence  to  develop  innovative  solutions  in  relation  to  housing. 
Furthermore, in this case, the Public Administration opens its doors to ensure that the groups directly affected 
by the actions developed are involved in them). 

The implementation had three parts:  

  Design:  Definition  of  the  aim,  participation  channels  and  duration  of  the  participative  process, 
evaluation  and  decision  whether  to  foster  processes  with  specific  groups  in  relation  to  certain 
questions on which the group’s opinion is important, and explanation of the rules of the game.  

  Launch activation and monitoring: Pre-testing of the participation tools and communication of the 
initiation of the process. Liaising with the media is extremely important, so that information on the 
process  reaches  its  target  population,  particularly  when  this  is  the  public.  Cooperation  with 
associations or other types of representative  social  groups is  recommended, to raise awareness of 
the  process  among  specific  groups.  During  the  open  participation,  on-going  monitoring  and 
introduction  of  the  planned  participative  elements  and  new  elements  that  have  arisen  from  the 
process, performing the necessary communication actions to foster participation. 

  Termination: Evaluation of the proposals received for their possible inclusion in the strategy, policies 
or  regulations  put  forward  for  social  debate,  preparation  and  presentation  of  a  report  on  the 
participative  process.  Evaluation  of  the  process  from  three  points  of  view:  level  of  participation, 
satisfaction  with  the  participative  process  and  impact  of  the  participation.  Satisfaction  with  the 
participative process is measured by means of a survey that is carried out at the end of the process. 
The impact of the participation is measured in accordance with the number of proposals put forward 
and the number of which are adopted, to improve the Department´s initial proposal. 

As a result of the initiative: 

  78  social  and  economic  agents  signed  the  Housing  Social  Pact:  agents,  social  organisations  and 
citizens (12); professional agents (14); sectoral agents (11); municipal urban development companies 
(13); public and partly-owned  companies (6); universities  (3); financial  entities (18); and the  Basque 
Government. 

  45  citizen  proposals  were  evaluated  for  their  potential  inclusion  in  the  Housing  and  Urban 
Regeneration Master Plan, of which 30 (67%) were included in the Master Plan. Of the remaining 15 
proposals,  8  were  not  included  because  they  were  already  implemented  and  the  other  7  were 
dismissed because the  Department  lacked the capabilities for their development  or were proposals 
that needed to be analysed in the debate on the future Housing Law.  In practice, therefore, 85% of 
the proposals were included in the Housing Master Plan.  In total, 15,748 people participated, 5,230 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

46 

surveys were answered, 569 opinions were expressed through the forums, and 120 suggestions were 
made through proposals. 

  Concerning  the  Basque  Housing  Law,  there  were  17,187  web  visits,  2,223  opinions  received,  188 

citizen proposals received and 312 social networks followers.  

The process focused on the establishment of more democratic forms of governance at regional level, following 
the  guidelines  of  the  White  Paper  on  European  Governance  (European  Commission,  2001),  where  the 
Commission  argued  that  good  governance  must  build  on  the  core  principles  of  openness,  participation, 
accountability, effectiveness and coherence. Therefore, it was a valuable experience at three different levels: 

1.  Department  of  Housing:  the  internal  dynamics  of  the  process  can  be  implemented  in  any  other 

projects developed 

2.  Basque Government: it is an innovative process in the framework of new legislation process within 

the Basque Administration 

3.  Good practice:  it constitutes  an example for other public  administrations, at local, regional or even 

national level. 

In  2012,  this  public  participation  project  won  1st  place  in  the  United  Nations  Public  Service  Award  in  the 
category of “fostering participation in policy-making decisions through innovative mechanisms”. 

In  a  further  international  recognition,  the  International  Observatory  on  Participatory  Democracy  (IOPD) 
awarded ‘Your Home, Our Commitment’, carried out by the Department of Employment and Social Policy of 
the Basque Government through the Basque Observatory of Housing, with a special mention in the IOPD 2016 
Award. The participatory process concerning the Housing Master Plan 2013-2016 was recognised by the judges 
in  the  ‘Special  Distinction’  category  among  36  nominations  from  15  countries.  The  IODP  promotes  its  "Best 
Practice in Citizen Participation" annually, with the aim of recognising those innovative experiences at the local 
level that promote the involvement of citizens in the development and implementation of public policy. With 
this participatory process, the Basque Observatory of Housing had the following two objectives: 

  To  check,  with  citizens  and  expert  agents,  the  strategic  principles  and  the  main  action  lines  of  the 
housing  policies  that  the  Basque  Government  was  to  implement  through  the  new  Housing  Master 
Plan 2013-2016; 

  To identify new ideas and citizen proposals to enrich and complement the already predefined actions 

in the draft Master Plan 2013-2016. 

The result was a Director Plan with more than 17,000 elements of participation including surveys, proposals, 
comments  etc.  More  than  16,000  visits  to  the  websites  specifically  created  for  this  purpose,  and  the 
installation of information tents and specific workshops on urban rehabilitation, have allowed the collection of 
citizens’ and professionals’ needs and suggestions first hand. For the Basque Observatory of Housing, the value 
lies in creating a new system of public-private partnership which can integrate conflicts in the design of public 
policy and maintain long-term relations with stakeholders, experts, academics and citizenship. 

These  initiatives’ outstanding achievements have demonstrated excellence in  serving the public interest  and 
have  made  a  significant  contribution  to  the  improvement  of  public  administration,  serving  as  an  inspiration 
and encouragement for others working for public service. 

For further information, Mario Yoldi, Mario-Yoldi@euskadi.eus, Elena Sánchez: 
observatoriovivienda@euskadi.eus  

The principle of co-design has been integrated with the concept of forward planning in the example 
of  ‘MijnBorne2030’  in  which  citizen  and  business  representatives  shaped  the  20-year  vision  and 
development programme for this Dutch municipality. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

47 

Inspiring example: Mijn Borne 2030 (The Netherlands) 

living 

Borne  is  a  municipality  in  the  province  of  Overijssel  in  the  eastern  part  of  the  Netherlands.  It  has  21,500 
in  three  population  centres:  Borne,  Zenderen  and  Hertme.  With  the  project 
inhabitants, 
‘MijnBorne2030’,  civil  society  organisations  (CSOs)  and  citizens  came  together  to  determine  the  future 
development  of  the  community  of  Borne.  The  project’s  objective  was  to  create  a  widely-shared  vision  as  a 
starting point for joint action.  

The  city  council  and  assembly  decided  to  delegate  this  responsibility  to  CSOs:  20  took  on  this  challenge, 
representing  entrepreneurs,  housing  corporations,  district  representatives,  health  care,  sports,  education, 
youth and elderly people.  Together they formed a  steering committee  ‘regiegroep’.  The steering committee 
was responsible for organising an interactive process with the community of Borne, leading to a new vision for, 
by, and of the community. Unique to the Netherlands, with this process, the project ‘MijnBorne2030’ reached 
the  highest  rung  on  the  participation  ladder:  delegated  power.  It  was  the  first  time  that  this  level  of 
participation has been reached while creating a vision for a whole community. 

The process consisted of six steps: (i) a trend report; (ii) identity study; (iii) formulating ambitions by citizens; 
(iv)  formulating  scenarios;  (v)  election  of  the  preferred  scenario  by  citizens;  and  (vii)  drawing  up  and 
determining the new vision ‘MijnBorne2030’. 

The trend report (i) was written with the help of local and regional experts on health, well-being, community 
building,  spatial  planning,  housing,  economics,  entrepreneurship,  governance  and  sustainable  development. 
To determine the identity of the community (ii), research was carried out by the University of Twente. Over 
200 inhabitants returned the questionnaire. Citizens were invited to take part in one of the 27 workshops to 
determine  the  most  important  ambitions  for  the  community  in  2030  (iii);  470  people  contributed  to  these 
workshops and over 400 people completed the (online) questionnaire. With these three building blocks, the 
steering  committee  created  four  scenarios  for  the  community  (iv).  In  April  2011,  elections  were  organised 
during  which  all  the  municipality’s  inhabitants  (aged  15  and  above)  had  the  right  to  vote  on  their  preferred 
scenario (v). This resulted in a majority vote for the scenario ‘dynamische dorpen’ – dynamic villages. Based on 
this scenario a new vision was drawn up (vi) and formally ratified by the city council (September 2011). 

Following the ratification of ‘MijnBorne2030: dynamic villages’, both the municipal government and four CSOs 
committed themselves to the realisation of this vision, working to ensure the continued effects of the process.  
These partners come together on a yearly basis to hold each other accountable for actions – centred around 
yearly  themes  such  as  safety,  sustainability,  social  activation  –  and  to  set  new  goals.  The  partners  are  also 
challenged  to  find  new  ways  to  embed  the  vision  in  future  activities.  As  during  the  initial  process,  this 
realization  strategy  makes  use  of  joint  action  and  social  media.  As  a  result,  MijnBorne2030  informs  many 
government  policies,  projects  and  goals,  while  CSOs  use  the  knowledge,  energy,  and  goodwill  of  the 
empowered community to reach shared goals. 

The  process  itself  has  been  evaluated  by  the  University  of  Twente,  to  learn  from  this  unique  form  of 
participation and make the lessons learned transferable, while the project has been lauded both nationally and 
internationally. 

For further information: Roeland Ambting, Project Manager, r.ambting@borne.nl  

Greater  connectivity  is  playing  a  helping  hand  in  co-responsibility.  The  eGovernment  Action  Plan 
2016-2020  envisages  future  initiatives  -  facilitated  by  the  digital  environment  -  that  contribute  to 
“engaging citizens, businesses and civil society in the collaborative design, production and delivery of 
public  services”.  Stakeholder  participation  in  public  policy  is  accentuated  by  administrations  that 
subscribe  to the principle of open government  (see also  principles and values of good governance 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

48 

and topic 2.2), and open up their public service information (PSI) to citizens and businesses, in line 
with the PSI Directive33 and the concept of open data. 

Open  data  is  data  that  can  be  freely  used,  re-used  and  distributed  by  anyone.  The  only 
requirements  are  that  users  attribute  the  data  and  make  their  work  available  to  be  shared  by 
others. 

In July 2013, the G8 leaders signed the G8 Open Data Charter. The Charter recognises that “access 
to  data  allows  individuals  and  organisations  to  develop  new  insights  and  innovations  that  can 
improve  the  lives  of  others  and  help  to  improve  the  flow  of  information  within  and  between 
countries”,  but  that  while  “governments  and  businesses  collect  a  wide  range  of  data,  they  do  not 
always  share  these  data  in  ways  that  are  easily  discoverable,  useable,  or  understandable  by  the 
public”.  As  well  as  improving  transparency  about  what  government  and  business  are  doing,  open 
data  can  be  a  catalyst  for  innovation.  Hence,  the  Charter  committed  its  signatories  to  making 
government  data  available  to  all  based  on  five  core  principles:  open  data  by  default;  quality  and 
quantity; useable by all; releasing data for improved governance, and releasing data for innovation.  

Following the launch of the G8 Charter, representatives of governments and CSOs across the world 
came together under the auspices of the Open Government Partnership (OGP) to “identify and share 
good  practices  to  help  OGP  governments  to  implement  their  commitments  and  develop  more 
ambitious  and  innovative  action  plans  related  to  open  data”,  through  the  Open  Data  Working 
Group. 

By providing managed access to PSI34, public administrations can help stakeholders become better 
informed  about  what  their  governments  are  doing  on  their  behalf,  and  better  equipped  to 
participate  and  collaborate  in  the  policy  process.35  This  includes  information  on  public  services 
through  online  channels  (see  theme  5),  and  the  facilitation  of  open  government  through  online 
communities,  such  as  the  Commission’s  Joinup  webpage.  In  addition,  the  opening-up  of  data, 
through  developments  such  as  the  European  Cloud  Initiative,  creates  opportunities  for  public 
administrations to realise the  potential of ‘big data’ analytics to generate  new policy thinking  (see 
topic  1.1.2).  In  view  of  its  own  digital  transformation,  the  Commission  has  committed  under  the 
eGovernment Action Plan 2016-2020 to publish its data on the EU Open Data Portal, which will feed 
into the European Data Portal (see also topic 5.4).  

33 PSI is all the data produced, collected or financed by public bodies in the European Union. 
34 See European Commission (2014), Delivering on the European Advantage? How European Governments can and should 
benefit from innovative public services. https://ec.europa.eu/digital-agenda/en/news/eu-egovernment-report-2014-
shows-usability-online-public-services-improving-not-fast 
35 For example, geo-information is very useful for location-based public services, within the framework of the INSPIRE 
Directive. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

49 

Open, modular and collaborative government 

The EU has been funding several eGovernment studies and related case studies, the most relevant of which 
are: 

  "Open eGovernment Services and how the public sector can become an agent of innovation through 

ICT", which identified examples of open government cases inter alia on an ongoing basis;36 and 

  “Towards Faster Implementation and Take-Up of Open Government”. 

In addition, the European Commission has been supporting European projects in the area of ICT-enabled open 
government under the Horizon 2020 Research and Innovation Framework Programme, Societal  Challenge 6 
“Europe in a changing world - Inclusive, innovative and reflective societies”, featuring the following topics: 

  Applied co-creation to deliver public services; 
  Co-creation between public administrations: once-only principle; 
  Policy development in the age of big data: data-driven policy-making, policy-modelling and policy-

implementation; 

  Understanding the transformation of European public administrations; 
  New business models for inclusive, innovative and reflective societies. 

Some examples of Horizon 2020 projects supported by the EU include DIGIWHIST (see topic 1.1.2 on big data 
analytics),  along  with  WeLive,  YourDataStories,  ROUTE-TO-PA,  OpenBudgets,  EUth  and  STEP.  A  full  list  of 
funded projects can be found here. 

New technologies combined with emerging innovative models create the possibility of new ways of 
co-creation  (see Public Sector Innovation below in  1.3.3 Fostering innovation) 

The  OECD  Council’s  2014  Recommendation  proposes  that  digital  government  strategies  can  bring 
public  administrations  closer  to  citizens  and  businesses  through  more  open,  transparent  and 
trustworthy  government,  and  enable  a  fundamental  shift  from  citizen-centric  approaches 
(government  anticipating  the  needs  of  citizens  and  businesses)  to  citizen-driven  approaches 
(citizens and businesses formulating and determining their needs in partnership with governments). 
Both  administrations  and  CSOs  are  employing  a  variety  of  new  tools,  especially  digital  ones,  to 
empower people in the service of the public good. In recent years, the United States has been in the 
vanguard, and hence most of the illustrations come from the other side of the Atlantic Ocean, but 
the techniques are potentially applicable across the EU too.  

Tool 

Description and examples 

Crowd storming  This online tool takes the concept of brainstorming from the few to the very many - scaling 
up  from  organising  an  internal  discussion  in  the  administration  to  seeking  the  ideas  and 
insights of potentially up to thousands of people, to develop innovative solutions to often 
complex policy problems. The highest profile example historically is the Habitat Jam, which 
was  a  massive  online  event  on  1-4  December  2005,  organised  by  the  UN’s  Human 
Settlements  Programme  (HABITAT),  the  Government  of  Canada  and  IBM,  to  help  solve 
urgent problems of the world's cities, in a  collaborative  Internet-based environment  that 
was able to accommodate up to 100 000 participants worldwide. 
Building on the principle behind crowd storming, crowdsourcing takes it a step further by 
inviting  citizens  to  contribute  more  than  ideas,  but  also  information  and  other  inputs  to 
public services. In some cases, crowd-sourcing involves prize-funding, such as NYC BigApps, 

Crowdsourcing 

36 See also other eGovernment studies on the Digital Single Market webpage, including ‘Towards faster implementation 
and uptake of open government’ (SMART 2015/0041). 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

50 

Tool 

Hackathons 

Civic hacking 

Living labs (LLs) 

Prototyping 

Description and examples 
which is an annual competition sponsored by the City  of New York that, over a period of 
five  months,  invites  “developers,  designers  and  entrepreneurs  to  create  functioning, 
marketable  tech  tools  that  help  solve  pressing  civic  challenges”.  Some  contest  winners 
have gone on to become viable companies. 
Combining  two  familiar  concepts,  hackathons  comprise  a  ‘marathon’  of  ‘hacking’  (in  the 
original  sense  of  exploratory  programming,  rather  than  criminal  activity).  These  events 
typically last  between 1 day  and 1  week, and involve an  open invitation to software and 
subject  specialists  to  convene  at  a  physical  location  and  jointly  develop  IT  solutions  to 
support selected service challenges. Hackathons have been organised for public authorities 
and NGOs, such as the Hackathon in London’s Hackney in March 2016 to support free legal 
services  for  low  income  citizens.  Hackathons  have  been  run  for  education,  health, 
transport and disaster management, inter alia. 
According  to  the  pioneering  CSO,  Code  for  America,  civic  hacking  is  “the  act  of  quickly 
improving  the  processes  and  systems  of  local  government  with  new  tools  or  approaches, 
conducted with cities, by citizens, as an act of citizenship” in the belief that “government 
can work for the people, by the people in the 21st century”. Code for America runs several 
initiatives  to  bring  the  benefit  of  volunteers’  skills  and  open  source  software  to  public 
services.  These  include:  ‘brigades’,  which  involve  groups  of  local  citizens  partnering  with 
local  governments  on  an  ongoing  basis;  and  ‘fellowships’  which  are  small  teams  of  paid 
developers and designers who work with the public administration (at state, county or city 
level)  for  a  year,  building  open  source  apps  and  helping  spread  awareness  of  how 
contemporary technology works among the government workforce and leadership. 
Living Labs are “user-driven innovation environments where users and producers co-create 
innovation  in  a  trusted,  open  ecosystem  that  enables  business  and  societal  innovation”, 
according to the European Network of Living Labs (ENoLL). An initiative under the Finnish 
EU  Presidency,  ENoLL  was  established  as  an  international,  non-profit,  independent 
association in 2010, reaching over 300 members across Europe by 2016. The World Bank 
and  ENoLL  have  jointly  produced  a  guidebook  on  citizen-driven  innovation  for  city  and 
municipal administrations,  which  explains that the  concept  was originally  developed as a 
way of more effectively carrying out research and development in ICT for public services, 
to  leverage  the  mass  knowledge  of  citizens  and  other  users.  LLs  take  R&D  out  of  the 
laboratory  and  into  the  real  world,  engaging  citizens  and  other  stakeholders  in  the 
collaborative design of new services. 
Rarely, some CSOs are engaged in piloting service innovations that can be rolled-out, such 
as Participle, which was active for 10 years in the health and welfare field (ageing, families, 
youth and employability), but took the decision to close in September 2015, leaving behind 
a legacy website. 

Topic 1.1: Qualities of good policy-making 

 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

51 

1.2 

Instruments of policy implementation 

considers 

policy-making 

Laws & the regulatory environment 

Achieving outcomes by changing behaviour 

Good 
the 
implications for implementation during policy 
design: translating the desired state-of-affairs 
(the high-level objective) into practical steps, 
weighing up the pros and cons of all available 
instruments, and choosing the most effective 
options  to achieve the  policy  goal  (including the ‘do nothing’ scenario of non-intervention). This  is 
easy  to  recognise  but  harder  to  realise,  as  the  instinct  of  different  units  and  competences  within 
public administrations is often to opt for the most readily available policy tool. If you are a ministry 
or municipal department with a budget allocation, the first response to policy problems tends to be 
spending.  If  you  are  the  centre  of  government  or  cabinet  with  responsibility  for  organisation,  it  is 
natural to look towards institutional change for solutions, such as restructuring or outsourcing. If you 
are a legislator, you tend to see the answer in more regulation or possibly de-regulation.  

Co-production 

For  any  government  at  any  level,  the  purpose  of  public  policy  could  be  described  in  the  broadest 
terms  as  determining  whether  and  how  to  intervene  to  ensure  the  security  and  well-being  of  its 
citizens, now and in the future. When considering their policy options, Governments can choose to 
intervene  directly  through  public  spending  (including  public  service  delivery),  or  indirectly  by 
influencing behaviour through regulation, information, taxation and other instruments, such as fees 
and charges. 

Public spending can have a direct impact on essential services and infrastructure where the market 
does  not  operate effectively  or  at  all  (e.g. education,  health, environment),  or should  not  operate 
(e.g.  defence,  police)  and  can  intervene  positively  to  stimulate  enterprise,  investment  and 
innovation.  Expenditure  can  have  a  ‘multiplier’  or  ripple  effect,  by  invigorating  local  economies, 
energising communities, securing the environment and local cultures and traditions,  and providing 
the  risk  capital  and  leverage  for  long-term  changes.  But  it  can  also  have  a  distortionary  effect  on 
private behaviour (favouring some interests over another), and always comes with a price tag, given 
that  public  expenditure  is  financed  through  taxes,  duties,  fees,  charges  and  borrowing.  Public 
finance  management  (PFM)  is  explored  further  in  theme  8.  As  an  instrument,  ‘public  spending’ 
covers: 

  Fiscal transfers, including central government financing of regional and local authorities (see 

theme 3) and other bodies, welfare benefits, grants and subsidies;  

  Expenditure on public service delivery (see theme 5);  
  Procurement of supplies, services and/or works (see topic 8.2), and  
  Use of EU funds including European Structural and Investment Funds (see topic 8.3).  

Moreover,  the  concept  also  includes  what  is known  in  PFM  as  ‘negative  expenditure’,  namely  the 
use of taxes, duties, fees and charges to influence behaviour. 

Laws  and  regulations  are  essential  in  many  policy  fields,  to  ensure  public  safety  and  security,  set 
standards  and  protect  the  public  interest.  They  can  have  beneficial  incentive  effects,  shaping 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

52 

personal  and  private  behaviour  by  permitting  some  activities  and  proscribing others. Regulating  is 
often seen as a more attractive option for administrations than spending, especially in times of tight 
finances, as it can appear ‘cost-free’. The reality, of course, is that there are always costs that must 
be  taken  into  account.  The  most  immediate  and  visible  ones  to  the  administration  are  the 
institutional implications of executing and enforcing the regulation. But it is the public, and more so 
the private sector, which usually faces the much greater burden from regulatory compliance - from 
‘hidden’ costs (e.g. person time, extra spending, use of space, lost opportunities) in the home, office, 
factory,  site  or  transit.    As  an  instrument,  ‘laws  and  regulations’  also  includes  de-regulation:  the 
decision to remove or revise regulations to reduce their impact. 

Soft Policy Instruments are deployed when the subsidiarity and proportionality options to address a 
given problem demonstrate that traditional legal instruments (regulations, directives, and decisions) 
are  not  desired.  Policy  makers  may  resort  to  "soft",  more  flexible  approaches  instead.  Soft  policy 
tools may include: recommendations, technical standards, communications, self-regulation, and so 
on.    The  drawback  of  soft  instruments  is  that  they  are  non-binding.  Therefore  it  is  advisable    to 
complement    such  policies  with  strong  ownership  incentives,  communication  and  engagement 
mechanisms.    See http://ec.europa.eu/smart-regulation/guidelines/docs/br_toolbox_en.pdf 

Reform of government structures can have a positive impact in finding better ways to achieve policy 
goals,  whether  it  involves:  creating,  abolishing  or  merging  public  bodies;  allocating  functions 
differently across the administration; centralising or decentralising powers; pooling resources across 
authorities; outsourcing, privatising, bringing under public ownership or control, or creating public-
private  partnerships.  As  with  other  instruments,  each  scenario  has  its  merits  and  its  drawbacks. 
Institutional reforms are disruptive and have short-term costs as a minimum, which must be justified 
by the longer-term benefits. Responsibilities rely on resources, so reallocating functions should have 
budgetary  implications,  may  affect  revenue  collection,  and  often  the  administration  of  regulatory 
authority  too.  Government  structures,  organisation,  cooperation  and  coordination  are  explored 
further in theme 3. 

Public administrations also affect outcomes by the ways in which they present information, an issue 
that the advertising industry understands well. In recent years, the art and science of ‘nudging’ has 
attracted  much  attention  –  how  to  influence  people’s  decision-making  in  an  unforced  way  by 
understanding  their  motivations,  incentives  and  behaviours,  and  steering  them  (through  ‘nudges’) 
towards the desired outcome. The application of these behavioural insights (see topic 1.1) to policy 
execution  goes  much  wider  than  just  information  provision  for  citizens,  businesses  and  other 
administration, for example in the context of service delivery (see theme 5), it also has implications 
for  how  laws  and  regulations  are  drafted  and  executed37,  for  how  grants,  subsidies,  taxes  and 
charges are targeted, and for how institutions can become more effective in achieving their goals.  

Administrations  are  also  increasingly  looking  to  co-production:  involving  citizens  and  businesses 
directly in the implementation of public goods and services (see also theme 5). 

37 See, for example, this published JRC paper on the application of behavioural insights to competition policy.  

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

53 

Every  instrument  has  its  place  -  its  potential  to  incentivise  behaviour,  influence  performance,  and 
achieve certain results. These outcomes can often be anticipated with confidence, but many times 
the full consequences cannot be foreseen, leading to unintended effects both good and bad. Each 
policy  tool  has  its  values  and  virtues,  but  each  also  brings  its  risks.  In  reaching  a  balanced  and 
conscious decision about which policy tool is best deployed in any situation, the costs as well as the 
benefits must be carefully weighed, and all instruments put in the mix.  

1.2.1  Laws and the regulatory environment 

water, 

Laws and the regulatory environment 

Legislation  makes  a  vital  contribution  to 
cohesive societies and prosperous economies 
in  many  ways.  Through  laws  and  standards, 
the regulatory framework aims to ensure that 
food, 
buildings, 
infrastructure, transportation and workplaces 
are safe, air is protected from pollution and land from contamination,  borders are secured against 
illegal activity, and the rights of consumers, employees, innovators and investors are respected. By 
creating  a  level  playing  field  for  enterprises  and  ensuring  fair  competition,  laws  and  regulations 
stimulate  productivity,  job  creation  and  economic  growth,  both  nationally  and  across  the  EU’s 
internal market.  

Achieving outcomes by changing behaviour 

Co-production 

products, 

While legislation plays an important and indispensable role, every regulation comes with a ‘price tag’ 
for  businesses,  citizens  and  administrations  too.  For  businesses,  the  cost  of  regulatory  compliance 
takes many forms: 

  Taking  time  for  registrations,  applications,  permissions,  providing  statistics,  completing 

reports and other ‘paperwork’ to meet information obligations; 

  Dedicating staff to act as compliance or information officers (full or part-time);   

  Incurring expenses on equipment and contracted-in services (e.g. legal, financial and other 

advice) etc.;  

  Reserving  space  for  keeping  records,  goods  and  materials  required  for  regulatory 

compliance; and 

  Changing  products  or  processes,  for  example,  due  to  changes  in  minimum  standards  or 

specifications, or the introduction of prohibitions. 

These represent opportunity costs for both the enterprise and the economy - time, staff, expenses 
and space that could be used in productive business activities, and hence must be well justified. The 
actual and potential costs for compliance tend to be disproportionately much greater for small and 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

54 

medium-sized  enterprises  (SMEs).38  On  average,  where  a  large  enterprise  spends  one  euro  per 
employee to comply with a regulatory requirement, a medium-sized enterprise might have to spend 
around four euros, and a small business up to 10 euros.39  

Reducing regulatory burdens 

Given actual and potential regulatory impact, public administrations have an implicit duty to justify 
both  new  and  existing  regulations,  to  check  that  the  compliance  costs  are  more  than  offset  by 
benefits to the economy, society, and environment, and to seek out the least burdensome solutions 
that are compatible with delivering policy objectives and priorities (see theme 5 on service delivery 
and theme 6 on the business environment). 

Laws  and  regulations can prove  problematic  if  their  preparation  is  performed  without  proper  and 
full  consideration  of  their  consequences,  including  how  they  will  be  put  into  operation,  and  the 
implications of secondary legislation. This can happen if the law or regulation is a rapid response to 
an  emergent  situation  that  could  not  reasonably  have  been  foreseen.  It  can  also  occur  if  the 
responsible authority is facing a tight timeline due to parliamentary timetables, the imminent lapsing 
of  an  earlier  law,  or  an  upcoming  deadline  for  transposing  a  directive.  Weaknesses  in  legal 
provisions,  especially  regarding  the  practicalities  of  implementation,  can  be  the  result  of  allowing 
insufficient time to consult with affected parties. Sometimes, legal flaws and anomalies arise from 
unclear or poorly formulated language, errors in scope or coverage creating gaps, or conflicts that 
emerge  with  other  legislation.  The  positive  effects  of  beneficial  legislation  can  be  undermined  by 
either failure to follow it up by passing the necessary by-laws to put it into practice, or alternatively 
by creating badly-worded by-laws that unravel the primary law’s intentions. 

Some  governments  have  sought  to  anticipate  potential  problems  by  setting  out  guidance  for  the 
public administration, including the EU institutions themselves through a joint practical guide of the 
Parliament,  Council and  the  Commission.  One  example  is  Finland’s  bill-drafting instructions,  which 
lay down principles and standards, which should be observed “so that the Parliament will receive the 
information  it  needs  on  the  legislative  proposals  it  is  to  consider,  and  so  that  the  Bills  will  be  of 
sufficiently  uniform  quality.  No  derogations  should  be  made  without  a  good  reason.  These 
instructions should be brought to the attention of every official involved in drafting work, as well as 
of the various drafting organs.”  

Inspiring example: Instructions to officials on drafting laws (Finland) 

These  instructions  provide  general  guidance  for  the  drafting  of  legislative  Bills.  The  instructions  cover  the 
specifics  of  Bill  structure  and  style,  such  as  the  various  parts  of  the  reasons,  their  purpose,  extent  and 
interrelationship. In addition, the instructions contain a brief description of the stages of a legislative project 
and of project scheduling. That said, the internal structure and the technical aspects of the proposed Act itself 
are covered only in outline, because more detailed guidance on these issues is available in the Legal  Writer’s 
Manual (2013, “Lainkirjoittajan opas”) and in other similar manuals. To make it easier to make sense of Bills, 
they must be drafted to the same basic structure, using the same standard headings.  Derogations from these 

38 The definition of an SME covers all enterprises with less than 250 employees, and equal to or less than either €50 million 
turnover  or  €43  million  balance  sheet  total.  Micro-enterprises  are  the  smallest  category  of  SME,  with  less  than  ten 
employees and a turnover or balance sheet total equal to or less than €2 million. 
39 Report from the Expert Group on “Models to Reduce the Disproportionate Regulatory burden on SMEs”, May 2007 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

55 

instructions should not be made unless there is a special reason for the same. 
The instructions are based on  

  A good explanation must be given for why the proposed legislation is necessary.  
  The Bill must be brief and concise.  
  Proper, plain language must be used.  
  The factual basis of the Bill must be correct.  
  The impact and the alternatives must be assessed and explained.  
  The constitutional issues must be settled.  
  The proposed legislation must be linguistically and technically complete and legally flawless. 

A Bill constitutes a proposal for a decision to be  made by the Parliament. The Bill must be drafted so that it 
supports parliamentary decision-making. The Bill must explain, concisely and to the point, what the proposed 
Act  or  legislative  package  is  all  about,  concentrating  especially  on  the  issues  that  are  relevant  as  to  the 
background, objectives and regulatory choices in the Bill. The current situation and the problems inherent in it 
must be described and reasons supplied why, precisely, the proposed legislation is the correct solution to the 
problems. The minimum requirements for appropriate law drafting are that the proposed legislation is indeed 
necessary,  that  it  achieves  the  objectives  set  to  it,  and  that  it  is  the  best  possible  way  of  achieving  those 
objectives. Bills must be drafted in proper, plain language. It is very important that a coherent and clear overall 
picture  is  provided  in  the  Bill  of  all  the  essential  impacts  of  the  proposed  legislation.  Reasoned  justification 
must  be  supplied  about  how  the  stated  objectives  can  be  achieved  by  the  proposed  legislation.  Open 
discussion  is  required  about  the  pros  and  cons  and  about  the  anticipated  costs,  not  only  of  the  proposed 
legislation,  but  also  of  any  alternative  legislative  or  regulatory  arrangements.  A  Bill  must  also  contain 
information about its relationship to solutions reached in other countries. If there is a link to EU legislation or 
other EU decisions, an appropriate account of these circumstances is also necessary. Moreover, the Bill must 
contain  an  account  of  how  the  proposed  legislation  is  intended  to  be  implemented  and  how  the  follow-up 
regarding the achievement of its objectives is to be arranged. 

For further information: parempisaantely@om.fi. Please also see http://lainvalmistelu.finlex.fi/en, which 
contains information on the legislative drafting process in English 

in  the  European 

The Finnish instructions place particular emphasis on performing an impact assessment (IA) during 
the  drafting  of  the  law,  and  presenting  a  summary  of  the  findings  with  the  bill  itself.  IA  is  an 
increasingly  well-established  technique 
institutions  and  across  European 
administrations  for  ensuring  that  the  consequences  of  a  law  or  regulation  are  fully  taken  into 
is  reached.  IAs  typically  cover  the  economic,  social  and 
consideration  before  a  decision 
environmental  impact,  positive  and  negative,  both  direct  and  indirect,  short-term  and  long-term. 
They are a mechanism for testing whether there is a need for a public intervention at all, whether 
the objective  of  the  law or  regulation  is  precisely  and  clearly  formulated,  and whether  alternative 
courses  of  action  have  been  fully  explored,  including  the  ‘do  nothing’  option.  The  Commission’s 
impact  assessment  guidelines  are  a  valuable  reference  tool  in  this  respect,  as  they  set  quality 
standards and include both general guidance on conducting IAs and thematic guidance for assessing 
specific impacts. The OECD also published guidance in 2008 on performing regulatory IAs specifically, 
which might also be a useful source, while some Member States, such as Poland, have also prepared 
their own national guidelines. 

It is important for the completeness and credibility of IAs that they are planned well, publicised, and 
subject to consultation and scrutiny. The Commission has laid down a series of steps for conducting 
IAs, which are transferable to other administrations: 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

56 

  After  the  need  for  an  IA  has  been  established,  draft  and  publish  a  ‘roadmap’  to  inform 
stakeholders about the upcoming work, feed in comments at an early stage, and plan ahead 
their participation in the public consultation; 

  Set up an IA steering group, which involves all relevant public authorities with an interest in 

the IA’s preparation (in the Commission’s case, all relevant Commission services); 

  Consult interested parties, collecting expertise and all available data; 
  Carry out the IA analysis, in accordance with the guidelines and best practice in CBA; 
  Present the findings in the draft IA report, including recommendations (opinion), and open 

its main elements to consultation and scrutiny; 

  Finalise  the  IA  report,  taking  on  board  comments,  prepare  an  executive  summary  and 

disseminate the report in accordance with the guidelines.40 

The  roadmap  is an information tool for all planned  initiatives  that may have  significant  economic, 
social  or  environmental  impacts,  not  just  legislative  proposals,  but  also  white  papers,  spending 
programmes,  implementing  measures,  etc.  If  there  is  no  expectation  of  a  significant  impact,  then 
neither  roadmap  nor  IA  is  required.  Roadmaps  describe  the  problem  that  the  initiative  aims  to 
address and possible policy options. They provide an overview of the different planned stages in the 
development of the initiative, including consultation of stakeholders and impact assessment work. If 
an impact assessment will not be carried out, the roadmap explains why. 

imposed  by  government  on  business  from 

There  are  various  methods  for  measuring  and  assessing  the  costs  and  benefits  of  regulation.  An 
example is the Standard Cost Model (SCM), which was originally developed in the Netherlands, and 
information 
measures  the  administrative  costs 
obligations. The Dutch SCM was adapted by the European Commission to become the EU Standard 
Cost Model (described in section 10 of the impact assessment guidelines), with additional elements 
including one-off costs, and is applicable also to citizens, public administrations, and the voluntary 
sector.  The  Netherlands  has  also  evolved  the  original  model  into  SCM  2.0.  Many  Member  States 
have  adopted  the  SCM  for  measuring  administrative  burdens,  and  in  some  cases,  extended  their 
cost-benefit  analysis  (CBA)  further,  by  encompassing  other  categories  of  cost  beyond  information 
obligations,  to  capture  more  fully  compliance  or  regulatory  costs.  For  example,  France  has 
completed the use of the SCM model with an evaluation of the compliance costs and/or the costs 
related to reducing burden and the assessment of the time lost or gained by a company in terms of 
business development.  

A review of techniques, including the merits of the SCM, is set out in a 2013 study on cost-benefit 
analysis  for  the  Commission.  Other  mechanisms  which  are  used  for  administrative  burden 
evaluation, include an indexed system in Belgium called “score board” that maps the administrative 
burden landscape.  

40 In the Commission’s case, draft IA reports are subject to quality control by the Regulatory Scrutiny Board (RSB), which 
comprises senior officials appointed by the Commission President that are independent from the IA preparation itself. The 
revised IA report, together with the IAB opinion, goes into inter-service consultation along with the draft proposal, and 
then submitted to College of Commissioners, with the findings published on Europa. 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

57 

The Standard Cost Model: core methodology 

The  Standard  Cost  Model  (SCM)  measures  the  administrative  costs  imposed  on  business  by  central 
government regulation. The costs are primarily determined through business interviews where it is possible to 
specify in detail the time companies use to fulfil the government regulation.  

The SCM breaks down regulation into three manageable components that can be measured: 

Information obligations are obligations to provide information and data to the public sector or third 
parties (e.g. reports about labour conditions, labelling provisions). 

  A  data  requirement  is  each  element  of  information  that  must  be  provided  in  complying  with  an 
information obligation. Each information obligation consists of one or more data requirements (e.g. 
VAT number, identity of business).  

  To provide information for each data requirement a number of specific administrative activities have 
to  be  carried  out.  These  may  be  done  internally  or  be  outsourced.  They  can  be  measured  (e.g. 
description, calculation, archiving information). 

The SCM then estimates the costs of completing each activity on the basis of cost parameters: 

  Price consists of a tariff, wage costs plus overhead for administrative activities done internally or 

hourly costs for external services.  

  The amount of time required to complete the administrative activity.  

  Quantity comprises of the size of the population of businesses affected and the frequency that the 

activity must be carried out each year.  

The combination of these elements gives the basic SCM formula:  

Cost per administrative activity = price × time × quantity. 

Source: OECD (2006), Cutting Red Tape: National Strategies for Administrative Simplification, OECD Publishing, 
Paris.  

In  the  context  of  the  European  Semester  and  Europe  2020,  the  priority  for  policy-makers  is 
identifying regulatory and other initiatives that stimulate smart, sustainable and inclusive economic 
competitiveness:  an  economy’s  ability  to  provide  its  population  with  high  and  rising  standards  of 
living and high rates of employment on a sustainable basis.41 The key ingredient of competitiveness, 
whether of  the whole economy or  a single enterprise,  is productivity  -  the value  that each unit of 
input  adds  to  output.  Where  policy  proposals  have  a  potential economic  impact,  competitiveness 
proofing within the framework of IAs is about paying special attention to the factors that are widely 
recognised as important to productivity, namely: 

  Costs  of  doing  business  (cost  competitiveness):  this  includes  the  costs  of  regulatory 
compliance,  but  also  changes  in  the  prices  of  inputs  (including  energy)  and  factors  of 
production (labour and capital);  

41 http://ec.europa.eu/smart-regulation/impact/key_docs/docs/sec_2012_0091_en.pdf  

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

58 

  Capacity  to  innovate:  the  potential  for  businesses  to  produce  more  and/or  higher  quality 
in  their  design, 

products  and  services  that  better  meet  customers'  preferences 
specifications, functionality, efficiency, etc.; 

  International competitiveness: the likely impact on the comparative advantage of European 

industries and their share of global markets; and 

  Better  allocation  of  resources:  reducing  barriers  to  the  reallocation  of  capital  and  labour 
within  sectors  helps  ensuring  that  the  most  productive  firms  can  achieve  their  growth 
potential and that less efficient ones leave the industry (or get restructured). 

The objective of competitiveness proofing is to identify these potential impacts and, where they are 
present, and measuring them would involve proportionate effort, to quantify them. Competitiveness 
proofing  is  not  a  'make  or  break'  test  of  new  policy  proposals.  Its  purpose  is  to  deepen  the  IA 
analysis  and  provide  evidence  to  policy-makers  that  may  lead  to  the  ranking  of  policy  options 
(favouring the most positive net impact) and/or the introduction of mitigating measures to alleviate 
any  negative  consequences  for  growth  and  jobs.  Quantifying  the  potential  impact  of  reforms  and 
provide evidence-based policy is a necessary task, but it is often a challenge, as data availability can 
be a serious constraint, it is difficult to infer causality, and the indicators of reforms’ effects can be 
hard to construct. Therefore, any effort from Member States to generate data, in particular at the 
micro level, will help to improve such assessments. 

The  Commission  provides  guidance  in  12  steps,  as  a  simple,  effective  and  flexible  tool  to  test  the 
impacts on sectoral competitiveness within IAs. 

Competitiveness proofing in 12 steps 

Getting started 

1.  Does your IA require a specific analysis of impacts on sectoral competitiveness in the first place?  
2. 

If the answer to step 1 is yes, what is the proportionate level of this analysis? 

Qualitative screening 

3.  Which are the affected sectors? 
4.  What is the effect on SME competitiveness? 
5.  What is the effect on cost and price competitiveness? 
6.  What is the effect on the enterprises’ capacity to innovate? 
7.  What might be the effect on the sector’s international competitiveness? 

Quantifying the impacts: data sources 

8.  Provide evidence on the structure and performance of the directly affected sector(s) 
9.  Provide data evidence on indirectly affected sectors 
10.  Quantify additional compliance and/or operational costs related to the assessed initiative 
11.  Quantify the expected impacts on the capacity of affected enterprises to innovate 
12.  Quantify the expected impacts on affected sectors’ international competitiveness 

Source: Better Regulation Toolbox, Chapter 3, as complement to Commission Staff Working Document, “Better 
Regulation Guidelines" SWD(2017) 350 final of 7 July 2017 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

59 

As well as being step 4 of the competitiveness proofing process, the SME test has been an integral 
element  of  the  Commission’s  IA  methodology  since  2009.  This  evaluates  the  economic  impact  of 
policy  proposals  on  SMEs,  which  can  be  disproportionate  for  two  reasons.  First,  regulatory 
compliance costs tend to be higher on average for SMEs as a proportion of total revenues, and they 
face  greater  difficulties  and  higher  costs  in  accessing  finance.  Without  the  scale  economies  that 
larger firms enjoy, they lack the information, staff and time to find out about regulations and deal 
with administrative rules. Second, SMEs account for the majority of innovative firms, and on average 
are  more  reliant  on  innovation  to  compete  and  prosper  than  larger  firms.  The  four  steps  of  the 
Commission’s  SME  Test  are  broadly  applicable  in  Member  States’  practices,  subject  to  data 
availability. 

Four steps to the SME Test 

(1)  Identify  affected businesses 
During this stage, you should establish whether  and which SMEs are among the likely affected population. If 
that is not clear, you will need to identify the characteristics of the businesses / sector(s) likely to be affected. 
Relevant  sources  of  information  should  be  explored  including  SME  representatives.  A  non-exhaustive  list  of 
elements to consider includes, when applicable: 

  Proportion of the employment concerned in the different categories of enterprises affected;  ; 
  Weight of the different kind of SMEs in the sector(s) - micro, small and medium-sized ones; 
  Links with other sectors and possible effect on subcontracting.   

If  the  preliminary  assessment  leads  to  the  conclusion  that  SMEs  are  amongst  the  affected  parties,  further 
analysis should be carried out. 

(2)  Consult with SMEs representatives  
Care should be taken to ensure SME dimension is a central element of the consultation strategy. In addition to 
an open public consultation, techniques can include: roundtable discussions with stakeholders; focus groups, 
hearings, SME panels, etc. 

(3)  Measure the impact on SMEs  
The analysis of the costs and benefits of the policy or legislative proposal should be performed with respect to 
the business size, differentiating between micro, small, medium and large enterprises, both qualitatively and, 
if possible and proportionate, quantitatively. It is important to establish to which extent the proposal affects 
SMEs' competitiveness or the business environment in which it will affect their operations. It is likely that an 
EU  measure  would  have  direct  and  indirect  beneficial  effects  on  SMEs.  It  is  equally  important  to  assess  the 
impacts  of  SME  specific  or  mitigating  measures,  where  they  already  exist.  The  direct  benefits,  such  as 
improved working conditions, legal certainty, increased competition, and new market opportunities, should (at 
some stage) be reflected in reduced costs to SMEs. Yet, these benefits may be offset by various costs, some of 
which may be disproportionately felt by SMEs, notably:  

  Compliance costs: created by the obligation to pay fees or duties and by the obligation to adapt the 

nature of the product/service and/or production/service delivery process to meet economic, social or 
environmental standards (e.g. the purchase of new equipment, training of staff, additional 
investments to be made) 

  Administrative costs: created by the obligation to provide information on the activities or products of 

the company, including one-off and recurring administrative costs (e.g. resources to acquire or 
provide information).  

The economic impact should be compared between SMEs and large enterprises, using overall costs identified  
to the number of persons employed to obtain the average cost per employee, or costs identified to the total 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

60 

overhead or turnover of the company. In addition, it would be useful to consider the following additional 
elements:  

  Possible impacts on barriers to entry, competition in the market and market structure, for example in 

terms of possibilities for SMEs to enter markets  
Possible impact on innovation 

(4)  Assess alternative options and mitigating measures  
If the chosen option creates disproportionate burden for SMEs relative to large enterprises,  and the existing 
measures do not address SME needs sufficiently, then consider the use or the revision of specific measures to 
ensure  a  level  playing  field  and  the  respect  of  the  proportionality  principle.  This  can  include  changes  to  the 
legislation to apply permanent or temporary exemptions, extended transition periods, or simplified reporting 
obligations.  It  could  also  include  fiscal  measures,  such  as  tax  relief,  direct  financial  aid,  or  lower  fees  and 
charges. The cost of the application of mitigating measures should also be fully considered and included in the 
final assessment. Mitigating measures are explored further under theme 6 

For further information:  
https://ec.europa.eu/info/files/better-regulation-toolbox-22_en 

While  IAs,  competitiveness  proofing  and  SME  tests  are  most  commonly  applied  to  proposed  new 
regulations in ex ante evaluation, public administrations need to take care of the  existing stock of 
legislation  through  ex  post  evaluation,  including  both  primary  and  secondary  legislation.  If  the 
existing  legal  base  is  not  addressed,  then  the  flow  of  new  laws  and  regulations  automatically 
increases the stock, and hence the burden on business, with the potential for legal anomalies.  The 
following  checklist  sets  out  a  series  of  questions  that  Member  States  can  apply  to  systematically 
evaluate each policy domain: 

Stock-take checklist of existing laws and regulations 

Question 
1.  Does the law or 

regulation create an 
excessive 
administrative burden 
on businesses? 

2. 

Is there an overlap 
between one or more 
laws or regulations? 

3.  Where there is more 
than one law / 

Clarification 

Suggestion 

It is assumed that laws/regulations are 
always justified, before they are adopted. 
But they should also always take account 
of the impact on business of their 
implementation. Many existing 
laws/regulations pre-date the 
introduction of IAs, or the IA was 
performed but there were unforeseen 
effects, possibly due to subsequent 
revisions or the impact of by-laws. 
The drafting of new laws and regulations 
should take account of existing ones, but 
discrepancies may remain. For example, 
new EU regulations may overlap with 
existing national legislation, in which the 
case the EU law has primacy. In addition, 
there is a need to eliminate duplications 
and improve the coordination between 
layers of the administration (local, 
regional and central government) and 
within layers (e.g. a business might 
receive the same administrative 
requirement from different ministries). 
Whether there is overlap or not, the 
existence of several laws or regulations in 

IAs could be performed on the 
stock of existing laws. One option 
would be to conduct stock-takes 
per sector, so that the combined 
effect of a body of 
laws/regulations can be analysed 
and corrective measures applied 
collectively. 

During the stock-take of existing 
laws and regulations, 
administrations could perform a 
comprehensive mapping exercise, 
to ensure that provisions are not 
duplicated, which may lead to 
inconsistences (see Q3) or 
obsolete laws / regulations (see 
Q5). 

The mapping exercise (under Q2) 
should also cover the consistency 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

61 

Question 
regulation covering a 
policy area, are there 
inconsistencies 
between them? 
4.  Where there is more 
than one law / 
regulation covering a 
policy area, are there 
gaps in provision, 
which create legal 
‘blind spots’? 

Clarification 

the same field can produce legal 
anomalies, such as inconsistent 
terminology, conflicting definitions, or 
contradictory rules.  
The flipside of overlap in laws and 
regulations in a specific field is gaps, 
where practice has shown that the legal 
base did not anticipate an eventuality, and 
hence there is no legal provision to either 
permit or prevent it from happening. 

5.  Are any laws or 
regulations now 
obsolete, but remain in 
place? 

Obsolete measures may be the reason 
why there is more than one 
law/regulation in a specific policy field 
that is creating overlaps, inconsistencies 
or excessive burdens. 

Suggestion 
of laws and regulations. 

The administration should 
propose amendment to existing 
laws or regulations, or the 
creation of a new legal base to 
replace outdated laws if 
appropriate. This process may 
lead to more rules, but they 
should be appropriate rules – 
carefully designed and consulted 
with businesses, as any new law 
or regulation would be. 
Following the ex post evaluation 
(stock-take), legislation should be 
adopted to ‘tidy up the statute 
book’ by repealing obsolete laws 
& regulations and/or codifying or 
re-casting amended laws into one 
consolidated law.42 To prevent 
obsolescence being repeated, and 
force future legislatures to decide 
consciously whether a law or 
regulation should continue, the 
administration should consider 
introducing ‘sunset clauses’ into 
new laws, at which time the 
legislation is automatically 
repealed. 

The European Commission has led the way on reducing the administrative burdens on business from 
EU legislation by developing the Better Regulation agenda, a concerted campaign to intervene only 
where  necessary, 
involve  stakeholders  through  consultation,  and  keep  burdens  on  public 
authorities, businesses and citizens to the minimum necessary to achieve societal goals. This is fully 
in  line  with  the  Small  Business  Act,  which  extols  policy-makers  to  ‘think  small  first’  in  designing 
legislation to reduce the burden on micro, small and medium-sized enterprises (see theme 6). The 
Commission has been particularly active in pushing forward this agenda in recent years with a series 
of decisive actions, in partnership with Member States and the European Parliament.  

42 Codification is the process of bringing together a legislative act and all its amendments in a single new act. Recasting is 
like codification in that is brings together in a single new act, a legislative act and all the amendments made to it, but unlike 
codification, recasting involves new substantive changes, as amendments are made to the original act during preparation 
of the recast text. In both cases, the new act passes through the full legislative process and replaces the acts being codified 
(Source: REFIT Progress Report, 2013) 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

62 

Better EU legislation: leading by example 

In  January  2007,  the  Commission  presented  an  ambitious  ABR  Action  Programme  to  eliminate  unnecessary 
administrative  burdens  on  businesses  in  the  EU.  The  European  Council  endorsed  the  programme  in  March 
2007  and  agreed  that  administrative  burdens  arising  from  EU  legislation,  including  national  measures 
implementing  or  transposing  this  legislation,  should  be  reduced  over  5  years  by  a  target  of  25%  (2012). 
According to the Commission, this could add €150 billion to the EU’s GDP in the medium term. The European 
Council also invited Member States to “set national targets of comparable ambition”. 

In pursuit of the ABR Action Programme target, EU legislation is estimated to generate administrative burdens 
of €124 billion, equivalent to around one-third to one-half of the total burdens on businesses.43 In 2009, the 
Commission  made  a  commitment  to  present  Sectoral  Reduction  Plans  for  13  priority  areas:  agriculture  and 
agricultural  subsidies;  annual  accounts/company  law;  cohesion  policy;  environment;  financial  services; 
fisheries; food safety; pharmaceutical legislation; public procurement; statistics; taxation / customs; transport; 
and working environment / employment relations. 

Since 2010, the Commission has developed Fitness Checks to assess the overall regulatory framework in the 13 
policy areas. These checks are designed to evaluate entire sectors, identify excessive administrative burdens, 
examine  regulatory  overlaps,  gaps,  inconsistencies  and/or  obsolete  measures  and  assess  the  cumulative 
impact  of  legislation.  Their  findings  serve  as  a  basis  for  policy  decisions  on  the  future  of  the  regulatory 
framework and to improve the quality of new legislation.  

The  Commission’s  November  2011  report  “Minimising  regulatory  burden  for  SMEs”  outlined  ways  of  taking 
the  concept  of  "Think  Small  First"  a  step  further  to  deliver  rapid  results.  It  set  out  how  the  Commission 
planned to strengthen the use of exemptions for micro enterprises and lighter legislative regimes for SMEs. It 
also explains how this will be followed up through the legislative process and implementation. 

Between  2007  and  2012,  the  ABR  programme  achieved  its  ABR  Action  Programme  target,  covering  72  EU 
legal acts in the 13 domains, or around 80% of the main sources of administrative burden. Measures equalling 
25%  have  since  been  adopted  by  the  co-legislators.  The  Commission  itself  went  beyond  the  target  by 
presenting proposals to cut the administrative burden by 33%, or the equivalent of close to €41 billion.  

As part of its commitment to stakeholder consultation, the Commission invited SMEs and their representative 
organisations  to  identify  the  top  10  most  burdensome  regulations  through  an  EU-wide,  Internet-based 
consultation  from  October  to  December  2012,  eliciting  1000  responses,  including  600  from  individual  SMEs 
based  in  the  EU,  40%  of  which  were  micro-enterprises.  With  the  support  of  the  High  Level  Group  of 
Independent Stakeholders on Administrative Burdens (HLGAB) the entire stock of EU legislation was reviewed 
and ABR measures were suggested to the Member States (‘ABR Plus’).  

The better regulation policy has since evolved further.  The Commission’s Regulatory Fitness and Performance 
Programme (REFIT), aims to achieve a  simple, clear, stable and predictable regulatory framework. The REFIT 
platform allows national authorities, citizens and other stakeholders get involved in improving EU legislation. 
They can make suggestions on how to reduce the regulatory and administrative burdens of EU laws, which are 
then analysed by the REFIT platform and the Commission. The REFIT scoreboard tracks the progress of each 
individual  initiative  and  the  changes  introduced  by  Parliament  and  Council  during  the  legislative  procedure. 
Each year, the Commission launches a set of simplification initiatives within its REFIT programme – drawing on 
input from individuals, businesses, NGOs, national authorities and other stakeholders The changes to existing 
law can take different forms: 

 

 

codification: all amendments made to a piece of legislation over the years are incorporated into a 
single new act, reducing volume and complexity 
recasting: similar to codification, but in this case the legislation itself is amended at the same time as 

43 Communication from the Commission to the Council and the European Parliament “Action Programme for Reducing 
Administrative Burdens in the EU: Sectoral Reduction Plans and 2009 Actions”, Brussels, 22.10.2009 COM(2009) 544 final,     

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

63 

 
 
 
 

 

 

previous amendments are incorporated to form 1 consolidated text 
repeal: unnecessary and irrelevant laws are removed 
review/sunset clauses: laws are reviewed or automatically removed after a given period 
revision: laws are modified to keep them up-to-date 
directives are replaced with regulations, so that all EU citizens are subject to the same rules and 
national governments can't add extra requirements 
laws still in preparation are withdrawn if they become obsolete due to scientific or technical advances 
or if they are no longer in line with new policy objectives 
legally binding laws are replaced with lighter alternatives such as voluntary agreements (self-
regulation, co-regulation) 

As  a  result  of  its  better  regulation  agenda,  between  2015  and    2017  there  have  been  137  initiatives  for 
regulatory simplification.  Commission has made 109 proposals for withdrawal of legislation, out of which 74 
laws have been repealed.  

In  July  2017,  the  Commission  has  consolidated  its  Better  Regulation  Guidelines  and  also  produced  a 
comprehensive Better Regulation Toolbox.  

For further information:  

Refit – making EU law simpler and less costly:  https://ec.europa.eu/info/law/law-making-process/overview-
law-making-process/evaluating-and-improving-existing-laws/reducing-burdens-and-simplifying-law/refit-
making-eu-law-simpler-and-less-costly_en 

 Better Regulation Factsheet: https://ec.europa.eu/commission/sites/beta-political/files/better-regulation-
factsheet_en.pdf 

Like the Commission, many Member States are especially committed to getting the views of citizens 
and  businesses  concerning  where  they  see  the  biggest  burdens.  A  prime  example  is  Belgium’s 
‘Kafka’  initiative,  which  had  resulted  in  130  laws  being  repealed,  as  well  as  contributing  to 
administrative  simplification of service  delivery (see  themes  5  and 6).  Other well-known initiatives 
include the UK’s Red Tape Challenge. 

Inspiring example: ‘Kafka’ (Belgium) 

With  its  motto  “la  simplification  fait  la  force”  (inspired  by  the  national  motto  “l’union  fait  la  force”),  the 
Belgium government launched the Kafka initiative in 2003 as an innovative way to cut red tape. The website 
www.kafka.be  is  managed  by  the  Administrative  Simplification  Agency  (ASA)  and  allows  a  user-friendly 
approach to collect and consider views and priorities from all stakeholders, citizens and businesses, affected by 
regulations.  The  success  of  Kafka  is  partly  linked  to  the  publicity  around  the  initiative  and  the  continuous 
political  backing,  but  also  the  fact  that  the  national  action  is  mirrored  by  efforts  at  the  regional  level.  The 
website was redesigned in 2014 and is now easily accessible from mobile devices. Many reform projects that 
took place are resulting from this unusually accessible and un-bureaucratic contact point. 

For further information: ASA@premier.fed.be; www.kafka.be  

More ambitiously, the Danish ‘Burden Hunter’ initiative does not wait for businesses to come to the 
administration to complain – the civil servants go out to enterprises to see the impact of regulations 
for themselves, especially those which are “irritating”, as much as time-consuming or costly. 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

64 

Inspiring example: ‘Burden hunter’ - methods for reducing administrative burdens and cutting red 
tape by involving end users and their perspectives (Denmark) 

Burden hunters are civil servants who involve end users - businesses - in developing better regulation that can 
remove red tape. Focus on the end user is an integral part of the Danish better regulation effort, and is a mind-
set (always keep the end users in mind), as well as a method (systematically involve end-users). The burden 
hunter  approach  allows  authorities  to  focus  on  removing  the  administrative  requirements  that  businesses 
perceive  as  the  most  burdensome.  By  using  qualitative  methods  -  observation  studies,  process  mapping, 
expert  interviews,  focus  groups,  co-production,  ‘nudging’  (see  1.3.3),  service  design  and  user-centred 
innovation - the authorities can gain insight into how businesses perceive the regulation and how they actually 
perform the administrative task that derives from the rules. These insights allow the public administration to 
efficiently change and adjust the regulation and its  administration with the best possible outcome and effect 
for businesses, as well as for regulatory compliance.   

in-depth  study. 

The process in a burden hunter project is structured as shown in the model (below). Depending on the scope 
and topic, the involvement of end users can be designed as a quick, cheap and simple process, or as a greater, 
Important  for 
more 
using the model is the circular way that 
concepts are developed: sometimes the 
testing  shows  that  you  will  have  to  go 
back  to  the  users  to  gather  more 
information, 
using 
sometimes 
different methods than the ones used in 
the first round.  

by 

When  you  engage  in  burden  hunting, 
you  need  to  go  to  the  person  (‘end-user’)  in  the  company  who  is  actually  facing  the  red  tape,  who  must 
perform  the  administrative  tasks  required  by  the  authorities,  and  can  help  identify  the  real  problem  and 
develop an appropriate solution.  

at 

aims 

By using the approach of involving end users, authorities can target their innovation effort towards increasing 
the  effect  and  outcome  on  a  number  of  different  ‘bottom  lines’  at  the  same  time:  productivity;  service 
delivery/experience; results/outcomes; and legitimacy/rule of law. Productivity refers to the effectiveness or 
productivity  increase  of  the  public  administration  –  getting  more  (compliance)  with  the  same  or  fewer 
resources  because  of  innovation.  The  service 
delivery/experience 
delivering 
regulation  and  public  service  to  businesses 
with  the  least  possible  compliance  costs  and 
burdens.  The  result/outcome  is  the  effect  of 
innovation  on  the  policy  target  of  regulation 
and  compliance.  What  has  happened  in  the 
real  world  in  the  final  analysis?  For  example, 
has compliance improved or do you meet the 
objective  of  the  policy  better  in  other  ways? 
Finally,  the  legitimacy/rule  of  law  refers  to  a 
justice  and 
perception  of  due  process, 
increased transparency.  

The  burden  hunter  approach  to  better 
regulation makes it possible to find solutions that create an impact on several bottom lines simultaneously and 
can actually work for both businesses and the relevant authority. This means that burden hunting basically can 
become  a  win-win-win-win  situation.  What  is  important  with  the  four  bottom  lines  is,  however,  that  not  all 
projects necessarily will give positive results on all four and that a project should not give unintended negative 
results on any of the four bottom lines. The objective of a burden hunter project must always be to make the 
area in the spider web bigger, not smaller (right). 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

65 

Burden  hunting  and  systematic  involvement  of  end  users  require  a  different  skillset  than  traditional  “public 
governance”. Preferably, an anthropological or design background is required, as well as experience in working 
with  a  qualitative  approach.  Projects  should  always  be  handled  as  a  collaboration  between  people  with 
different skills 

Burden hunting methodologies can be used for both broad and narrow projects.  

  Broad projects aim at uncovering the burdens in a specific area or from a specific law. This approach 
can be used when you reduce the burdens but do not know where to begin. It can give you a better 
understanding of the real challenges businesses face.  This approach has, for instance, been used for 
discovering  growth  barriers  for  start-up  companies  in  Denmark.  In  this  project,  the  burden  hunters 
visited  several  companies  with  an  open  questionnaire  to  let  them  guide  us  to  where  the  barriers 
could be found.  

  Narrow projects give you concrete solutions to specific problems. This type of project can create the 
groundwork  for  decisions  about  concrete  workflows,  guidelines  or  input  for  digital  solutions.  The 
narrow  approach  has,  for  instance,  been  used  for  making  it  easier  for  companies  to  do  the 
administrative tasks in connection to getting environmental permits in Denmark. The project with a 
narrow scope gave us specific input to things that could be changed. 

For further information: Helle Venzo, Danish Business Authority, HelVen@erst.dk  

Administrative  burden  reduction  is  a  joint  endeavour  and  a  shared  responsibility  of  the  European 
institutions  and  Member  States.  Commission  initiatives  -  such  as  mapping  and  screening  the  EU 
acquis,  performing  fitness  checks,  and  codifying  or  recasting  laws  -  tackle  business  burdens  at 
source,  but  the  regulatory  process  does  not  end  with  better  crafted  legislation.  The  High-Level 
Group’s authoritative 2011 study found that almost a third of the administrative burden arising from 
EU  legislation  was  caused  by  how  EU  laws  are  interpreted  and  implemented  by  Member  States. 
Action  to  introduce  EU  legislation  more  efficiently  at  the  national,  regional  and  local  levels  could 
reduce the burden on businesses by nearly €40 billion.  

All Member States are obliged to transpose EU directives by the deadline in the legislation itself, or 
risk  infringement  proceedings,  but  how  they  are  transposed  is  matter of  national  discretion.44  For 
example, some Member States pass one over-arching law during the Parliamentary session to put all 
outstanding EU directives on the national statute, which ‘mops up’ their obligations across all policy 
fields in one sweeping motion, without tailoring them to domestic circumstances or tidying up loose 
ends  from  existing  laws,  creating  an  overall  increase  in  the  regulatory  base  and  potential  legal 
minefields. When transposing EU directives, it is important that Member States do not  ‘gold-plate’ 
them, inserting additional provision that go beyond what was agreed by the co-legislators, which is 
estimated to account for 4% of the ABR arising from EU regulations and is entirely avoidable. They 
should also use this opportunity to reconcile new and existing  laws in a harmonised legal base. To 
ensure that the acquis is transposed on time, and without gold-plating which will put businesses at a 
disadvantage,  the  UK  has  published  transposition  guidelines  for  all  authorities  engaged  in  law-
making.  

44 Member States’ performances on transposition and infringements are considered in the EU Single Market Scoreboard. 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

66 

Inspiring example: Guidelines on transposing EU directives (United Kingdom) 

The Guiding Principles are aimed at ensuring the UK systematically transposes so that burdens are minimised 
and UK businesses are not put at a disadvantage relative to their European competitors. The Principles state 
that, when transposing EU law, the Government will:    

  Ensure  that  (save  in  exceptional  circumstances)  the  UK  does  not  go  beyond  the  minimum 

requirements of the measure which is being transposed; 

  Wherever possible, seek to implement EU policy and legal obligations through the use of alternatives 

to regulation; 

  Endeavour  to  ensure  that  UK  businesses  are  not  put  at  a  competitive  disadvantage  compared  with 

their European counterparts;   

  Always  use  copy-out  for  transposition  where  it  is  available,  except  where  doing  so  would  adversely 
affect UK interests e.g. by putting UK businesses at a competitive disadvantage compared with their 
European  counterparts  or  going  beyond  the  minimum  requirements  of  the  measure  that  is  being 
transposed. If departments do not use copy-out, they will need to explain to the Reducing Regulation 
Committee (RRC) the reasons for their choice;   

  Ensure  the  necessary  implementing  measures  come  into  force  on  (rather  than  before)  the 
transposition  deadline  specified  in  a  Directive,  unless  there  are  compelling  reasons  for  earlier 
implementation; and   

Include a statutory duty for ministerial review every five years. 

Source: “Transposition Guidance: How to implement European Directives effectively” (April 2013) 

In  Denmark,  the  Government  has  introduced  an  ‘Implementation  Council’  to  safeguard  against 
creating  unnecessary  burdens  on  business  when  implementing  EU  regulations  and  transposing  EU 
directives, including gold-plating.  

Inspiring example: Implementation Council (Denmark) 

The Implementation Council is an advisory body to the Danish Government and was established in  2015. The 
aim is that the Council works, along with the Government, to ensure that Danish companies are not subject to 
stricter requirements than companies in other EU countries.  Additionally, the aim of the Council's work is to 
contribute to the Government's ambition to ease the economic burdens on Danish companies by 3 billion DKK 
by the end of 2020, compared to 2015.   

The role of the Implementation Council is to ensure efficient implementation of EU regulation and to prevent 
Danish special rules and gold-plating (whereby EU directives are given extra strength when being incorporated 
into national law) by 1) identifying future EU regulations, where there is a need for early and proactive Danish 
action  to  avoid  unnecessary  burdens,  2)  preventing  unnecessary,  potentially  burdensome  issues  to  be 
introduced  when  EU  legislation  is  transposed  into  Danish  legislation,  and  3)  pointing  out  areas  in  existing 
Danish  regulation  where  EU  regulation  has  been  gold-plated  or  implemented  inefficiently,  and  where 
simplification or deregulation is therefore needed.  

The  Danish  Government  has  adopted  five  principles  for  implementing  EU  business  regulation  that  aim  to 
prevent Danish special rules and gold-plating, unless the overriding public interest requires doing so. The five 
principles are: 

1.  National regulation should not go beyond the minimum requirements of the EU regulation. 
2.  Danish companies should not be disadvantaged in international competition, so the implementation 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

67 

should not be more burdensome than the expected implementation in comparable EU countries. 

3.  Flexibility and derogations in the EU regulation should be utilised. 
4.  EU  regulation  should  be  implemented  through  alternatives  to  regulation  when  appropriate  and 

possible. 

5.  Burdensome  EU  regulation  should  be  implemented  as  late  as  possible,  taking  into  account  the 
common commencement dates (which means that business regulation can only enter into force two 
times a year). 

The  15  members  of  the  Implementation  Council  are  primarily  representatives  of  business  organisations,  as 
well as consumer, employer and employee organisations. Three members are experts with special knowledge 
of implementation of EU regulation and are appointed by the Minister of Business and Growth. 

The Implementation Council meets four times a year. Prior to the meetings, members of the Council outline 
proposals to be discussed at the meeting. The proposals agreed upon by the Council are subsequently sent to 
the Government. The Danish Business Authority provides the secretariat and supports the Council in making 
qualified recommendations.  

inter-ministerial 

The Implementation Council is part of the Government’s action to prevent gold-plating, which is managed by 
an 
the 
recommendations  put  forward  by  the  council.  The  Government’s  responses  to  the  recommendations  are 
made  publicly  available  at  www.implementeringsraadet.dk.  As  of  September  2016,  the  Implementation 
Council had sent 66 recommendations to the Government.  

implementation  committee  of  eight  ministers.  The  committee  receives 

For further information: Secretariat of the Implementation Council, IR@erst.dk  

All Member States have adopted their own national targets for business ABR, some on a gross basis 
(counting just burdens that have been removed) and some on a net basis (counting also the negative 
effects of new burdens introduced over the same period), ranging from 15% (Luxemburg and Malta) 
to 30%  reductions  (Lithuania  and  Spain).  In  some  cases,  these  targets  include  all  EU  legislation,  in 
others  they  are  nationally  derived  only.  Some  Member  States  have  met  initial  targets  and 
subsequently set new ones. Others have  also set targets for administrative burdens on citizens, or 
focused on specific metrics (i.e. compliance costs).45 The basis of measurement and timescales mean 
they  cannot  be  directly  compared  across  the  EU-28,  but  they  show  the  priority  given  to  tackling 
regulatory burdens and compliance costs that now exists across Europe.  

The 2014 HLGAB report and the dissenting opinion from four of its members suggest that the role of 
targets is open for debate. Regulation remains an essential instrument in the public administration’s 
armoury, especially in the fields of safety, consumer, environmental and employee protection, and 
where  it  creates  a  fairer  and  more  efficient  marketplace,  but  there  is  no  justification  for  badly 
conceived and poorly implemented rules that exceed what is necessary to achieve policy objectives. 
There  is  a  consensus  that  if  an  existing  rule  is  unnecessary,  out-dated  or  ineffective,  it  should  be 
removed or replaced, which requires targeted and systematic action. 

Portugal’s ‘Simplegis’ programme is among the most successful initiatives in recent years to cut the 
regulatory stock through repealing obsolete laws and avoiding unnecessary law-making, but also 
to  ensure  what  is  left  is  more  easily  accessed  by  citizens  and  businesses,  and  more  effectively 
enforced. This initiative is one facet of the national ‘Simplex’ programme of administrative burden 
reduction,  which  has  been  driven  from  the  very  top  of  Government,  and  started  off  aimed  at 

45 HLGAB study (2011), op. cit. 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

68 

ministries,  but  has  since  devolved  to  the  municipal  level,  including  ‘Simplis’  within  Lisbon  City 
Council (see theme 5). 

Inspiring example: ‘Simplegis’ programme (Portugal) 

The  goals  of  ‘Simplegis’  can  be  summed  up  in  three  ideas:  fewer  laws,  more  access,  and  improved 
enforcement. The following achievements can be highlighted: 

In 2010, more than 300 obsolete legislative acts were repealed;  

  During the same year, the number of new decree-laws enacted by the Government was the lowest of 
the last 10 years as they were approved solely when necessary and after thoughtful consideration of 
the regulatory needs, to avoid unnecessary law-making;  

  Also in 2010, the rate of flawless legislative acts - therefore, with no need of correction by means of 

amending statement - was above 95% (a record in the last decade). 

Since October 2010, the decree-laws and implementing decrees that are published at the online version of the 
official journal are accompanied by a plain language summary, both in Portuguese and English. Portugal was 
the  first  EU  Member  State  to  offer  such  service  to  citizens  and  businesses  totally  free  of  charge.  A  new 
legislative and legal information web portal  was due to be launched online by the second semester of 2011, 
and  would not  only feature the online  version of the official journal, but  also added-value legal information 
(free  of  charge),  consolidated  versions  of  relevant  pieces  of  legislation  and  online  public  consultations  with 
direct and straightforward questions to make participation accessible and simple for everyone. 

In September 2010, a  wide  variety of acts were  eliminated from the official gazette  (hunting ground-related 
acts,  for  instance)  and  were  assigned  to  specialised  websites,  thus  making  it  easier  to  search  through  the 
gazette for the most relevant acts. From January 2011, a new method of thorough ex ante impact assessment 
was put in place and applied to all government acts, with good results. Moreover, impact assessment teams in 
all ministries that will perform ex post assessments have been trained. 

Source: 
http://ec.europa.eu/smart-regulation/refit/admin_burden/best_practice_report/report_example_en.htm 

Some  Member  States,  such  as  the  Netherlands  and  the  UK,  have 
introduced  common 
commencement  dates  (CCDs)  to improve  communication and  predictability of necessary laws and 
regulations for businesses. The government sets fixed dates on which new business legislation will 
come  into  force.  By  publishing  and  providing  notice  in  advance  of  forthcoming  legislation,  public 
administrations can give businesses time to plan, prepare for compliance, and budget for the costs 
of  new  rules  that  might  affect  them.  There  are  exceptions  and  exemptions,  inevitably,  but  these 
should be kept to a minimum.  

Once  adopted,  Member  States  have  a  range  of  options  to  ease  the  regulatory  burden  during  the 
implementation  and  enforcement  of  EU-derived  and  national  legislation,  which  are  considered 
further elsewhere in this toolbox. 

Question 
Has the national administration ensured that legislation is accompanied 
by appropriate institutional arrangements to implement and enforce it, 
so that businesses are clear which organisation is tasked with providing 
information and responsible for transactions and can respond to queries 
and concerns? Is this accountable body sufficiently equipped and staffed? 

Refer next 

  Government structures – 

organisation, cooperation and 
coordination (theme 3) 
  Organisations – managing 
performance, quality and 
people (theme 4) 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

69 

Question 
Has the national administration ensured that the implementation of the 
regulation is kept as simple, fast, easy and user-friendly as possible, with 
respect to deadlines, documents, steps, costs, etc., and made best use of 
service delivery channels, including one-stop shops and eGovernment? 

Refer next 

  Service delivery and 

digitalisation (theme 5)  
  Enhancing the business 
environment (theme 6). 

1.2.2  Achieving outcomes by changing behaviour 

responses 

Laws and the regulatory environment 

In  the  pursuit  of  causality  (linking  action  to 
outcome),  policy-makers  can  be  swayed  by 
the  illusion  of  ‘rationality’,  particularly  in 
people’s 
government 
interventions.  The  reality  is  that  human 
cognition  follows  its  own  path:  in  being 
emotional and social creatures, the human brain is hard-wired to act intuitively in many situations, 
prone to pre-conceptions and biases, susceptible to group behaviour and struggling to cope with the 
overload of information in 21st century life. 

Achieving outcomes by changing behaviour 

Co-production 

to 

In many cases, public policy is not simply about governments ‘doing things’ to safeguard security and 
improve  prosperity,  or  in  the  case  of  some  legislation  ‘stopping  things’,  it  is  concerned  with 
individual,  collective  or  corporate.  Whether  spending,  taxing, 
influencing  behaviour,  either 
legislating,  regulating,  informing  or  delivering  services,  this  goal  can  be  achieved  directly  or 
indirectly46: 

  In  some  cases,  behaviour  change  is  the  primary  objective  of  the  policy.  Examples  include 
campaigns  to  encourage  people  to  lead  healthier  lives,  protect  the  environment  (e.g. 
recycling, installing solar panels or insulation), plan for  retirement, enter higher education, 
wear seatbelts in cars, engage in lifelong learning, etc.  

  In other cases, the behavioural consequences are secondary effects that policy-makers need 
to take account of. For example, fuel duty will raise a certain amount of predicted revenue, 
but the level will also affect vehicle usage by citizens and businesses (which could equally be 
the primary objective), which will in turn determine the total revenue take; increasing excise 
on tobacco might raise revenue, but also lead to higher levels of cross-border smuggling, etc.  

  In  still  other  cases,  government  actions  are  targeted  at  intermediaries  (e.g.  businesses) 
whose activities affect or are affected by the behaviour of citizens as third parties. Examples 
include  consumer  protection,  such  as  introducing  laws  and  standards  on  misleading 
advertising, food labelling, cigarette packaging, etc. 

In  each  case,  the  policy’s  effectiveness  is  determined  by  people’s  reactions  and  responses.  These 
behavioural effects  need to be  factored into the  policy design  and implementation.  How  much do 
we know about people’s behavioural traits? Thanks to a growing body of research by psychologists, 

46  As  succinctly  described  in  the  European  Commission  (2013),  Applying  Behavioural  Sciences  to  EU  Policy-Making,  JRC 
Scientific and Policy Reports, René van Bavel, Benedikt Herrmann, Gabriele Esposito and Antonios Proestakis. 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

70 

sociologists, economists and neuroscientists, under the banner of behavioural science, the answer is 
quite  a  lot.  And  the  main  conclusion  is  that  the  conventional  model  of  ‘rationality’,  which  often 
underpins policy-making as an implicit assumption, is a highly idealised and often unrealistic model. 

The old orthodoxies of rational behaviour are increasingly being challenged. This is particularly the 
case in classical economic theory, which assumes that autonomous individuals make optimal choices 
in their own interest, independent from others, to maximise their ‘utility’ (essentially, satisfaction), 
having  reviewed  and  rejected  the  alternative  scenarios.  But  economics  is  not  alone.  The  default 
setting for many decision-makers is to assume the intended beneficiaries will respond rationally to 
policy levers and triggers. 

None  of us is a fully-formed ‘homo economicus’ as described above. We are fallible  homo sapiens 
instead, each with our own  flaws and foibles. We  can be  rational and reflective, but  we often act 
first and think later47. We struggle to deal with the vast volume of available information out there, 
ever-growing  thanks  to  the  Internet.  We  are  prone  to  using  heuristics  (‘rules  of  thumb’)  to  make 
decisions,  rather  than  investigating  the  options.  We  are  creatures  of  habit,  engaging  in  repetitive 
actions. We want to eat our cake and have it too.48  

Some of the most common cognitive and emotional biases relevant to public policy-making are well 
studied and summarised briefly here: 

Characteristic 

Anchoring and 
adjustment 

Framing 

Availability 
bias 

Brief explanation 
People tend to rely disproportionately on one piece of information when making decisions 
as  a  reference  point  or  ‘anchor’,  which  is  typically  the  first  piece  of  information  they 
receive.  Once  the  anchor  is  in  place,  subsequent  decisions  are  made  by  adjusting  away 
from it, rather than challenging the premise of the anchor itself. This trait can reveal itself 
in negotiations, when the first number (e.g. price, budget, compensation claim) that is put 
on the table becomes the anchor, and success is judged in relation to that original number. 
People tend to respond differently to information, depending on how the case is presented 
(‘framed’).  For  example,  they  respond  to  risk  more  positively  if  a  question  is  framed  to 
achieve a positive result, compared with avoiding a negative result, even if the probability 
of the actual outcome is the same (e.g. people are more likely to agree to an operation that 
has a  90%  chance of  success than the same operation that  is presented as having  a  10% 
chance of failure).49  
People tend to assess the likelihood or prevalence of something as higher, the more readily 
they can bring instances of it to mind. Hence, for example, people are more likely to think 
the crime rate is higher than it actually is, if they have recently read or heard stories about 
robberies  or  shootings.  If  the  person  has  recent  personal  experience  of  the  event  itself, 
these  instances  stand  out  (‘salience’)  and  accentuate  the  effect.  This  tendency  reveals 
itself,  for  example,  in  surveys  that  show  people  believe  immigrants  constitute  a  much 
higher % of the population than is the case, often by a factor of 10. It also affects people’s 
perspectives  on  specific  health,  crime,  financial  and  disaster  risks,  either  much  higher  or 
lower  than  the  real  probabilities,  and  hence  the  actions  they  take  (e.g.  lifestyle  changes, 
insurance) accordingly.  

47  The  characterisation  by  Nobel  Prize  winning  psychologist,  Daniel  Kahneman,  of  ‘system  1’  and  ‘system  2’  thinking  is 
neatly summarised in the JRC’s paper and can be read in depth in Thinking, Fast and Slow, Penguin Books. 
48 See also D. Ariely (2008), Predictably Irrational: The Hidden Forces that Shape our Decisions, Harper Collins. 
49 The Nutrition and Health Claims Regulation (EC 1924/2006) lays down harmonised rules for the use of health or 
nutritional claims (such as “low fat”, “high fibre” and “helps lower cholesterol”) on foodstuffs based on nutrient profiles. 
This is intrinsically related to the question of framing, as in the past consumers were often misled by changes of the 
reference point (a 20% fat cheese was often packaged as 80% fat-free). 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

71 

Characteristic 

Confirmation 
bias 

Optimism and 
over-
confidence bias 

Endowment 
effect and loss 
aversion 

Status quo bias 
(inertia) 

Brief explanation 
People tend to  seek out and focus on information that confirms their preconceived ideas 
and  ignore  evidence  that  contradicts  them.  This  reveals  itself  in  their  media  choices, 
including news sources and blogs.  
People  tend  to  over-estimate  the  chances  of  a  favourable  outcome  (optimism  bias),  and 
their own individual ability to influence situations (over-confidence bias), which encourages 
risk-taking.  Examples  include  the  chances  of  business  success  (most  fail  within  the  first  3 
years),  winning  the  lotto  (millions-to-one),  or  completing  a  project  within  the  agreed 
timeframe.  Over-confidence  reveals  itself  in  the  classic  example  that  most  people  when 
surveyed assess they are better-than-average drivers, which is statistically impossible.  
People tend to value more highly something they already own than something they do not 
yet own (endowment effect). Similarly, people tend to attach a higher weight to not losing 
€10  than  to  gaining  the  same  amount  (loss  aversion).  Both  tendencies  run  counter  to 
classical  economic  theory,  which  suggests  they  should  be  indifferent  between  the  two 
outcomes. E.g. it suggests that people would be more reluctant to, say, take out a loan that 
is secured against their existing property than take out a loan for a new property. 
People  tend  to  stick  with  the  current  state-of-affairs  and  gravitate  towards  this  ‘default’ 
position. This  reveals itself,  for example, in reluctance to  switch banks or utility  suppliers 
when the market is opened to competition. It is the reason that online retailers send out e-
mails offering ‘free trials’, subject  to providing bank  account  or credit card details,  which 
the customer can cancel at no cost during the 1st month otherwise the direct debit will be 
triggered. They know that most people will not act in the final days of the free subscription. 
This inertia is why governments regulate to ‘opt in’ to deals, rather than ‘opt-out’.50   

There are other effects too, which determine how people process information. For example, we face 
information overload from the Internet, beyond the willingness or ability of people to sift through 
all  the  available  sources.  As  well  as  our  innate  confirmation  bias,  which  leads  us  to  seek  out 
information providers which reinforce our personal beliefs, we also rely on search engines to make 
choices for us – 98% of Google searches stop after the top three sites, with 60% of hits on the first. 
This makes Googles algorithms among the most powerful influencers on human knowledge. 

We are not just guided by our individual traits. We are social animals, with an underlying desire to 
conform,  and  shaped  by  social  norms  and  cultural  factors.  We  are  also  deeply  affected  by  those 
around us; we  are influenced by family, friends and colleagues, but also strangers  on social media 
with whom we have only the most cursory contact. The economist Paul Ormerod refers to network 
effects:  people  often  change  their  preferences  based  purely  on  what  others  do.51  He  argues  that, 
throughout history, people have had a propensity to copy or imitate  the  behaviours, opinions and 
choices  of  those  around  them.  This  can  happen  for  a  variety  of  reasons  –  so  that  we  fit  in  (e.g. 
fashion),  because  it  is  risky  to  stand  out  (e.g.  in  the  workplace),  because  we  assume  others  have 
privileged  information  that  we  do  not  (like  choosing  between  an  empty  restaurant  and  a  packed 
one), or we are simply guided by the most popular choice (like clicking on the biggest ‘hits’ on music 
websites). These traits are characterised as group biases, such as group-think, herd behaviour and 
bandwagon effects. 

50 In 2008, the European Commission recognised the existing scientific evidence on the impact of default options, and 
proposed a Directive on Consumer Rights to the European Parliament and the European Council, including a clause limiting 
the use of default options in consumer contracts. The proposal was adopted in 2011 and the new rules entered into force 
across the EU in June 2014. Specifically, sellers are now obliged to obtain express consent from consumers for any payment 
that is additional to the payment for the main contractual obligation, and could not rely on defaults that require buyers to 
take an active choice to avoid an extra payment. 
51 See P. Ormerod (2012), Positive Linking: How Networks Are Revolutionising Your World, Faber and Faber Ltd. 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

72 

Phenomenon 

Group-think 

Herd behaviour 

Bandwagon 
effect 

Brief description 
Individuals  in  a  decision-making  group  seek  consensus,  minimise  conflicts,  suppress 
dissent,  avoid  controversial  issues,  isolate  themselves  from  external  influences,  and/or 
fail  to  challenge  the  dominant  view  (from  the  majority  opinion,  the  group’s  leader  or 
strongest  personality)  by  making  a  critical  assessment  of  alternative  perspectives  and 
options,  resulting  in  an  irrational  and/or  dysfunctional  outcome.  It  involves  the 
subjugation  of  individual  creativity  and  independent  thinking  to  the  cohesion  of  the 
group.  
This  occurs  when  individuals  in  a  group  act  together  (collectively)  without  any  central 
planning  or  direction.  Human  examples  usually  cited  include  riots  and  stock  market 
bubbles  and  crashes.  More  benign  examples  include  the  renowned  1969  study  of  the 
influence of crowds, which involved employing a group of actors of increasing number to 
stop in a busy street and look up at a building (4% of the passers-by stopped alongside a 
single individual looking up, rising to 40% of passers-by when the stimulus crowd reached 
15).52 
As  the  name  suggests  (‘jumping  on  the  bandwagon’),  this  is  the  tendency  to  take  up 
ideas,  beliefs  or  trends  because  other  people  have  done  the  same,  regardless  of  the 
underlying  qualities  (e.g.  evidence,  attributes).  The  probability  of  adopting  these 
behaviours,  beliefs  or  actions  increases  in  accordance  with  the  proportion  who  have 
already  done  so.  This  can  occur  because  of  a  desire  to  conform,  because  individuals 
derive their information from others, especially where it is limited, or because individuals 
want to be on the ‘winning side’. The most obvious example is fashion. 

We  are  especially  influenced  by  our  social  networks,  which  makes  techniques  like  social  network 
analysis and agent-based modelling (see topic 1.1.2 on big data) particularly pertinent.  

“Network  effects  require  policy  makers,  whether  in  the  public  or  corporate  spheres,  to  change 
radically their view of how the world operates. In part, they make policy much harder to implement 
successfully,  and  they  help  explain  many  of  the  failures  of  policies  based  on  the  assumption  that 
incentives and not network effects are the key drivers of behaviour. But they open up the possibility 
of much more effective and successful policies, ones which harness our knowledge of network effects 
and how they work in practice”.  Professor Paul Ormerod, Positive Linking, 2012. 

In many cases, copying can be a successful strategy (e.g. studies have shown people are more likely 
to stop smoking if a friend gives up), but herd behaviour can also be potentially fatal. For example, in 
the event of a fire or other type  of emergency in a  stadium, people will tend to follow  the  crowd 
even if the exit  is packed  -  and will ignore  unused exits, even when marked as such, for the same 
reason  they  ignore  empty  restaurants.  So,  while  opening  more  evacuation  points  might  seem  the 
rational  solution  to  increasing  safety,  a  behavioural  approach  is  needed  which  focuses  on 
information and flow.  

Behaviour in an emergency 

In a fire emergency, most people go through four behavioural steps before they even begin to evacuate: 

Disbelief: Patrons don’t believe it is happening. If they hear an alarm, they will tend to assume it is a false 
alarm. 

Commitment: Patrons typically finish what they are doing. If they are buying a snack, they will finish doing 

52 S. Milgram, L. Bickman and L. Berkowitz (1969), Note on the drawing power of crowds of different size, Journal of 
Personality and Social Psychology 13, 79–82.  

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

73 

that  before  they  investigate  further.  In  the  King’s  Cross  Underground  fire  in  the  1980s  in  London, people 
were observed stepping over hoses deployed from fire trucks to get to  the escalators to go underground. 
They  were  committed  to  getting  their  train  home.  In  numerous  tests  in  the  retail  environment,  patrons 
observing actual smoke will take their goods to the check-out, rather than move to a fire escape. 

Affiliation: Patrons assemble in friends/family groups and won’t evacuate until then. This is a huge issue in 
stadiums. When patrons are away from their seats, and convinced that there is a fire emergency, they will 
not evacuate — they will return to their seats to re-unite with family members first. 

Seek authority figures: Once they have formed their social and family groups, they will look for authority 
figures to tell them what to do. But they must be perceived as having authority. A teenage usher employed 
on the day of the event may not be perceived as an authority figure. Where do people in a sports stadium 
look if they don’t know what is going on? The scoreboard! 

Source: http://www.aurecongroup.com/en/thinking/archive/stadia-fire-safety.aspx  

As  decision-making  organisations,  public  administrations  themselves  are  also  influenced  by 
individual and group biases. Group-think is an ever-present risk in any hierarchical institution, unless 
members  are  encouraged  by  management  to  speak  up,  offer  fresh  perspectives  and  critical 
viewpoints, and challenge orthodox thinking. As the most extreme example, the Rogers Commission 
Report  found  the  1986  Challenger  space  shuttle  disaster  was  a  consequence  of  flaws  in  the 
organisational culture, as much as technical deficiencies. 

In  many  ways,  our  common  idiosyncrasies  make  people’s  behaviour  just  as  ‘predictable’  as 
rationality.  This  is  something  that  businesses,  especially  advertising  and  marketing  professionals, 
have recognised for decades. These behavioural insights are the basis for the new vogue of ‘nudging’ 
applied to public policy, made famous by Richard Thaler and Cass Sunstein.53 Whenever information 
is  presented  to  individuals  for  decision-making,  it  contains  a  ‘choice  architecture’  in  the  words  of 
behavioural scientists, whether consciously or unconsciously.  

A nudge is “any aspect of the choice architecture that alters people’s behaviour in a predictable way 
without  forbidding  any  options  or  significantly  changing  their  economic  incentives.  To  count  as  a 
mere nudge, the intervention must be easy and cheap to avoid. Nudges are not mandates. Putting 
the  fruit  at  eye  level  counts  as  a  nudge.  Banning  junk  food  does  not.”  Richard  Thaler  and  Cass 
Sunstein, Nudge, 2009. 

In  the  case  of  public  policy,  the  public  choice  architecture  has  often  arisen  by  accident,  and  only 
been  refined  over  time  through  trial  and  error  (as  opposed  to  rigorous  controlled  testing). 
Increasingly,  however,  policy-makers  are  using  behavioural  sciences  to  inform  policy  design  and 
implementation, whenever there is a behavioural element. It can help design new policies, suggest 
improvements  to  established  ones,  or  provide  ex  post  explanations  of  outcomes.  It  can  apply  to 
spending, taxation, regulation and information.  

For example, nudging has been employed by the Danish Business Authority (DBA) to help businesses 
comply  more  easily  with  their  regulatory  obligations.  In  the  example  below,  the  DBA  is  helping 
enterprises  avoid  errors  in  the  submission  of  annual  reports  and  to  check  the  quality  of  their 
business data in a user-friendly manner. 

53 R. H. Thaler and C.R. Sunstein (2009), Nudge: Improving decisions about health, wealth and happiness, Penguin Books. 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

74 

Inspiring example: Nudging (Denmark) 

‘Nudging’ is one of the newest tools used by the Danish Business Authority (DBA) in attempting to create the 
best conditions for growth of business in Europe, and to make it easy and attractive to run a business. By using 
behavioural insights, the DBA develops ways of identifying burdens and creating solutions. Using nudging as a 
tool  combines  using  trials  (preferably  randomised  control  trials)  and  data  analysis  with  knowledge  of 
behaviour. The nudging tool is primarily used in the context of better regulation and is often combined with 
more  qualitative  research  focused  on  identifying  the  behavioural  patterns  that  might  create  inopportune 
outcomes for businesses as well as for the DBA. In particular, the DBA uses nudging to tackle situations where 
businesses  unwittingly  follow  a  behavioural  pattern  that  leads  them  to  make  mistakes,  which  are  not 
intentional.  

The  DBA  has  run  initial  trials  that  have  showed  that  nudging  is  a  relevant  tool  for  tackling  diverse  issues 
ranging from getting businesses to discover services that the authority provides to getting more businesses to 
turn in their annual report correctly. Going forward, the DBA will run even more initiatives that will focus on 
improving businesses take up of growth programmes and on improving registration systems to relieve burdens 
and boost case management.  

When developing nudging initiatives, the DBA addresses a number of different ‘bottom lines’ at the same time. 
These  are:  productivity;  service  delivery/experience;  results/outcomes;  and  legitimacy/rule  of  law  (see  the 
‘Burden Hunter’ case study under topic 1.2.1 
for  definitions  of  terms).  More  often  than 
not,  the  DBA  found  that  new  solutions  can 
create  an  impact  on  several  bottom  lines 
simultaneously  and  can  actually  work  for 
both  businesses  and 
the  relevant 
authority.  What  is  important  with  the  four 
is,  however,  that  not  all 
bottom 
projects necessarily will give positive results 
on  all  four  and  that  an  initiative  should  not 
give  unintended  negative  results  on  any  of 
the  four  bottom  lines.  The  objective  of  a 
nudge initiative must always be to make the 
area in the spider web bigger, not smaller. 

lines 

for 

The following examples show that it is possible to create nudge solutions that benefit both business and the 
DBA - thus, without it, incurring extra expenses for the businesses or the DBA itself. The lessons learned can be 
reused  across  other  areas  where  the  DBA  is  charged  with  quality  control  and  where  newly  digitalised  data 
gives us the opportunity to nudge to avoid specific mistakes: 

  A  marked  improvement  in  annual  reports  submitted  to  the  DBA  via  a  digital  solution:  All  Danish 
businesses  are  required  to  send  an  end-of-year  financial  report.  There  are,  however,  different 
requirements  regarding  the  amount  of  information  required,  based  on  the  type  and  size  of  the 
business. DBA targeted the smaller businesses that have relatively uncomplicated annual reports and 
use  the  so-called  “Regnskab  Basis”  digital  solution,  which  involves:  creating  a  draft  of  the  annual 
report  in  Regnskabs  Basis;  printing  out  the  information;  getting  the  management  and  board  to 
approve and sign off the annual report (if required, the accountant must sign off as well); and getting 
approval of the report from the general meeting of the involved partners. The person charged with 
reporting controls the draft and makes sure the information is identical to what has been approved, 
but  there  is  a  tendency  to  create  the  draft  of  the  annual  report  after  the  general  meeting,  which 
indicates  that  the  submitted  report  is  not  necessarily  identical  to  the  one  approved  by  the 
management  and  board.  In  2013,  20,000  annual  reports  (under  the  responsibility  of  around  5,000 
different  individuals  -  in  many  cases,  company  accountants)  did  not  follow  the  correct  procedure, 
which  represents  about  43%  of  the  annual  reports  submitted  through  the  Regnskabs  Basis  digital 
solution.  To  change  behaviour  and  address  systematic  errors,  5,000  ‘nudging’  mails  were  sent  out, 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
  
 
Quality of Public Administration – A Toolbox for Practitioners 

75 

leading to 50% fewer mistakes over the same three-month period in 2014 compared with last year. 
The trial has reduced the rate of reoffending (committing the same mistake again) from 61% to 36% 
when compared to the same period last year. This action has helped to reduce the risk of penalties if 
the company’s annual report is subject to control, and led to time savings from getting it right in the 
first instance.   

  A marked improvement  in quality of business data: With the introduction of more and more digital 
services and data-based business models, the area of data quality is ever growing in importance. The 
nudge consisted of introducing a pop-up that allows for the businesses to see and control their own 
data  as  part  of  the  normal  log-on  to  the  digital  platform  (www.virk.dk).  The  nudge  worked  on  the 
premise  that  businesses  do  not  know  that  their  data  is  outdated  and  that  they  don’t  know  how  or 
where  to  correct  the  information.  During  the  trial,  15,000  businesses  were  presented  with  their 
currently registered data and were asked to verify them - 53% of the businesses confirmed their data 
by  pressing  the  “verify”  button  and  42%  pressed  the  “correct”  button,  giving  the  DBA  strong 
indications on the current data quality and the need for further measures that can help manage the 
problem.  

For further information: Kristine Poulsen-Hansen, Danish Business Authority, KriPou@erst.dk  

Nudging is only one application. In 2016, the JRC published a state-of-the-art study of the application 
of  behavioural  insights  (BIs)  to  policy  across  the  EU  Member  States  and  four  EFTA  countries.  The 
main report is accompanied by a full set of individual country reports, detailing over 200 behavioural 
policy initiatives, as well as institutional developments regarding the policy application of BIs. 

Behavioural Insights Applied to Policy (BIAP) 

BIAP 2016 is one of the first  publications of the newly-created  EU Policy Lab54 at the Joint  Research Centre 
(JRC) of the European Commission. 

Since 2008, the European Commission has been a front-runner in bringing behavioural insights into legislation 
and  regulatory  intervention.  This  approach  has  been  used  in  a  number  of  cases,  from  the  Directive  on 
Consumer  Rights,  and  other  consumer  protection  interventions,  to  a  competition  policy  decision.  The 
potential  contribution  of  behavioural  sciences  is  also  mentioned  in  the  "Toolbox"  for  Better  Regulation, 
guiding the design of policies and laws that achieve their objectives at minimum cost.  BIAP 2016 focuses on 
developments across Europe and provides a state-of-the-art view of the contribution of behavioural insights 
to policy-making, while also putting forward an analysis of institutional developments. It showcases examples 
of  behavioural  interventions  in  a  range  of  policy  areas,  such  as  employment,  consumer  policy,  health, 
taxation, environment or transport, pointing to their respective outcome whenever this was available.  BIAP 
2016 identifies areas where additional work is needed to improve mutual learning, strengthen the evaluation 
of policy impacts, and encourage a  more systematic use  of the behavioural approach. The report  is also an 
invitation for sharing experiences, and a call for cooperation in making full use of these insights to deliver on 
the EU objectives. 

BIAP  2016  draws  on  information  collected  via  desk  research,  a  survey  and  personal  exchanges,  including 
interviews  with  policy-makers,  academics  and  a  range  of  other  stakeholders  from  32  countries  (28  EU 
Member States and the four EFTA countries).  

Overall,  the  evidence  shows  that  insights  from  behavioural  sciences  are  contributing  to  reshaping  public 
policy  in  a  wide  range  of  domains,  in  particular  employment,  consumer  protection,  health,  taxation, 
environment  and  transport.  Furthermore,  some  successful  behavioural  initiatives  seem  to  be  replicated  or 
adapted  across  countries,  and  this  includes  well-known  examples  (e.g.  receipt-based  tax  lotteries)  but  also 
less obvious ones  (e.g. the penalty points system for driving offences). The current  review covers a  total of 

54 For an overview of the EU Policy Lab, please see topic 1.1 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

76 

more than 200 behavioural policy initiatives, the full range of which can be found in the Country Overviews. 
The insights presented allow for a better understanding of the context and ways through which a given policy 
issue can be tackled, as well as of the behavioural element (i.e. behavioural biases and/or levers) underlying 
given policy initiatives. In some cases, policy-makers explicitly took into account behavioural biases  - such as 
information  overload,  overconfidence,  loss  aversion  -  when  designing  appropriate  policy  solutions.  For 
instance,  in  view  of  reducing  information  overload  and  superfluous  complexity,  initiatives  such  as 
prepopulated tax forms aimed at simplifying administrative procedures and increasing tax compliance.  

BIAP  2016  also  gives  account  of  institutional  developments  regarding  the  policy  application  of  BIs.  The 
interest 
in  harnessing  the  potential  of  BIs  for  policy-making  has  already  triggered  organisational 
developments  in  some  EU  Member  States.  In  the  UK,  The  Netherlands,  Germany,  France  and  Denmark, 
dedicated teams have been created to this end, while similar approaches are being considered in Finland and 
Austria.  The  report  analyses  these  developments  through  the  lenses  of  a  tool,  PRECIS,  which  allows  for  a 
characterisation  of  teams  applying  BIs  to  policy-making  across  six  dimensions:  Political  support,  Resources, 
Expertise, Coverage, Integration and Structure. Looking at these six PRECIS dimensions, it is clear that while 
the  existing  behavioural  teams  present  several  differences,  they  have  all  contributed  to  raising  awareness 
about  the potential of BIs for policy-making and stimulated their effective use in their respective countries. 
Additionally,  while  no  specific  structure  has  been  developed  in  the  public  administration  of  most  European 
countries, the application and impact of BIs on policies is nevertheless increasingly visible.  

At  EU  level,  BIs  have  explicitly  informed  a  number  of  policy  initiatives  since  2009.  BIs  can  inform  more 
targeted and efficient  solutions at all stages of EU policy, from design to implementation of EU regulations. 
The  2014  European  Commission's  Better  Regulation  Agenda  calls  for  evidence-based  policy-making  with  a 
view of delivering more effective policies. By taking an outcome-oriented approach, BIs strengthen the focus 
on evaluation and support impact assessment as recognised in the Better Regulation "Toolbox."  

The  use  of  BIs  for  policy-making  is  debated  wherever  it  develops.  Beyond  the  legitimate  ethical  questions 
raised by the using nudges designed to favour a particular behaviour, a few myths and misconceptions have 
to be dismissed:  

  BIs are not "old stuff;"  
  While they might at times be "so close to intuition," they rely on a scientifically-based methodology 

and evidence;  

  BIs do not only rely on behavioural economics and should not be confused with nudges;  
  BIs do not breach data privacy;  
  Randomised controlled trials are not necessarily too costly to be justified for policy purposes.  

Behavioural  sciences  can  inform  policies  by  providing  an  analytical  framework  for  experimentation  and  ex 
ante testing of policy options to assess their  effectiveness. In this  context,  transparency and the  sharing of 
experiences and outcomes should be two primary concerns for all policy-makers applying BIs. Transparency is 
needed to respond to ethical concerns, while sharing can lead to more robust behavioural policy initiatives, 
built with a greater understanding of "what works," and under which conditions (e.g. cultural, geographic, of 
specific cohorts).  

Behavioural sciences can derive valuable behavioural evidence from existing large datasets or from merging 
relevant datasets and analysing the resulting picture. Some breakthrough academic papers in this field, using 
existing  European  datasets,  could  serve  both  as  a  basis  for  further  stimulating  exchange  between 
policymakers and researchers, as well as for providing inspiration in view of future similar studies.  

Four main conclusions stem from BIAP 2016:   

1. 

In  terms  of  capacity-building,  there  is  significant  dynamism  and  growing  appetite  to  apply  BIs  to 
policymaking.    

2.  There is certainly room for improved exchange and knowledge sharing between the policy-making and 
the  academic  communities.  For  instance,  there  is  great  potential  in  analysing  large  datasets  for 
extrapolating useful insights for policy with the associated challenge of making more publicly-owned data 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
   
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

77 

available for research.  

3.  BIs  should  be  applied  throughout  the  policy  cycle  -  including  in  anticipating  implementation  and 
enforcement issues - to generate useful evidence in the most effective way. There is still little awareness 
of  the  insightful  evidence  that  could  come  from  a  more  systematic  analysis  of  the  impact  of  policy 
solutions.  

4.  There is space to undertake more actions to improve the effectiveness of behavioural policy initiatives, 
shedding  light  on  their  long-term  impact  and  increasing  transparency,  namely  through  more  effective 
communication and evidence sharing with citizens.  

BIAP 2016 constitutes a  starting point  towards a  process that should ideally lead to further evidence-based 
policy, increased use of behavioural approaches and policy experimentation, and mutual learning. 

The JRC review of behavioural policy initiatives is supported by a new classification, which helps to 
structure thinking about evidence-based policy approaches:  

  Behaviourally-tested:  Initiatives  are  based  on  an  ad-hoc  test,  or  scaled  up  after  an  initial 

experiment;  

  Behaviourally-informed:  Initiatives  are  designed  explicitly  based  on  already  existing 

behavioural evidence; or 

  Behaviourally-aligned: Initiatives can be said to be in line with behavioural evidence, at least 

a posteriori. 

Among  the  examples  of  institutional  approaches  cited  in  the  JRC  study  is  France’s  General 
Secretariat  for  the  Modernisation  of  the  Public  Action  (SGMAP),  which  is  leading  the  way  in 
coordinating behavioural insights within the French administration at all levels. SGMAP is applying all 
three  types  of  approach.  In  the  case  of  ‘online  tax  declaration  and  payment’  case  below,  for 
example, the pre-filling of tax declarations is behaviourally informed, while the use of nudges in the 
promotional campaign is behaviourally tested.  

Inspiring example: Applying behavioural sciences to public policies (France) 

Since 2014, the Secrétariat Général pour la Modernisation de l’Action Publique (SGMAP) has been pioneering 
new  approaches  to  policy-making  in  France,  playing  a  coordinating  role,  and  spreading  the  message  about 
behavioural  insights  (BIs)  across  the  French  administration,  including  ministries,  municipalities  and  public 
agencies. 

Sitting at the centre of government in the Prime Minister’s Office, SGMAP has integrated ‘nudging’ and more 
generally  a  behavioural  approach  into  its  innovation  team,  with  a  combined  total  of 7  staff  in  2016,  so  that 
policy design / prototyping takes account of BIs. ‘Agile’ approaches based on field experimentation and using 
interdisciplinary  knowledge  is  rare  among  public  administrations,  so  the  team  works  closely  with  other 
ministries,  research  institutions55  and  local  administrations  to  redefine  public  policies  and  build  capacity 
through to use BIs. The approach to policy development involves five stages: 

55 Experimental Economics Lab (Sorbonne University); French National Centre for Scientific Research (CNRS); French 
National Institute of Agricultural Research (INRA); Groupe d'Analyse et de Théorie Economique (GATE),  
Lyons; INSEAD Social Science Research Centre, Paris; Laboratoire d'expérimentation en sciences sociales et analyse des 
comportements (LESSAC), Dijon; Laboratoire d’économie appliquée de Grenoble (GAEL); Laboratoire d'Economie 
Expérimentale, Toulouse; Laboratoire d'Economie Expérimentale, (LEEM), Montpellier; Laboratoire d'Economie 
Expérimentale, (LEEP), Paris; Laboratoire d'Expérimentation en sciences sociales (LABEX), Rennes; Laboratoire d'Economie 
Expérimentale, (LEES), Strasbourg; School of Public Health (University of Toulouse); University of Strasbourg 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

78 

1.  Understanding by ‘immersion’ – observing what is happening on the ground 
2. 
3.  Co-constructing prototype policies and services with the people affected by them to building on 

Identifying new methods using behavioural economics 

citizens’ real experiences and uses 

4.  Testing the proposed solutions 
5.  Rolling out the new policy 

Immersion is the first  step in the process and arguably a very important one, so SGMAP has been delivering 
training for municipal employees in how to observe behaviour, including fieldwork, so that in-house staff have 
the  skills,  rather  than  outsourcing  to  external  agents.  Implicating  the  in-house  staff  in  the  process  creates  a 
greater engagement on their behalf and allows them to experiment with the approach themselves  to better 
understand it. This process is done with the help of experts (working at SGMAP) that provide the training, give 
general guidelines and that regularly guide the agents along the way.  

As an example of the process, the French Government wishes to make tax compliance easier, and is trying to 
encourage people to declare their income and pay the corresponding taxes online. However, the vast majority 
of citizens still physically visited their tax office, rather than using the Internet – in 2014 only 35% of taxpayers 
declared  their  revenues  online.  To  understand  better  why  this  was  the  case,  the  administration  visited 
taxpayers in their homes to see how they filled in the declaration, and discovered that many were scared of 
missing something and so felt it would be safer to do it with the tax office’s help. It was recognised that the 
automatic pre-filling of declarations with both fiscal and non-fiscal information would simplify the process and 
reduce  the  information  overload  on  citizens.  It  would  also  reduce  the  cost  of  tax  management,  which 
amounted to approximately € 250 million per year. In May-June 2014, the Government launched a campaign 
to promote online tax return. Seven different messages - based on nudge levers such as salience, social norms, 
immediate call to action, loss aversion - were tested and sent to taxpayers who had access to the Internet but 
did not use it for their tax return. The Minister of Finance announced that, as a result of the campaign, online 
tax returns increased by 10 percentage points.  

The elaboration and testing of the ‘nudges’ themselves can be conceived in six phases, shown in the diagram 
below: 

Phase 1: Explore/ study behaviour (observations) 

Phase 2: Analyse cognitive biases (based on a review of the scientific 
literature)  

Phase 3: Convene creative workshop to generate nudges - formulate as many 
as possible 

Phase 4: Select & enrich nudges for deployment 

Phases 5 and 6:  Design & conduct experiment - choose terrain & 
implementation modalities (P5), monitor & analyse results, recommend & 
evaluate (P6)  

SGMAP and its partners in the administration have recently initiated a set of projects applying BIs mainly in the 
policy  fields  of  environment,  health  and  taxation,  but  also  consumer  protection,  social  cohesion,  gender 
stereotypes and drug addiction. Examples include: 

  Reducing  mobile  phone  use  while  driving:  In  2014,  SGMAP  worked  with  the  delegation  for  road 
safety (DGSCR) to reduce mobile phone use while driving. This project lead to the design of a “driving 
mode”  application  (equivalent  to  the  “plane  mode”  setting)  and  a  reframing  of  road-safety  related 
administrative mail usually sent to citizens. 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
 
                                                                                                                                                                                     
 
Quality of Public Administration – A Toolbox for Practitioners 

79 

  Encouraging patients to pay remaining hospitalisation fees: In 2015, SGMAP worked with the French 
health care services (DGOS) to encourage patients to pay the "uncovered remainder" of their hospital 
fees which they forget/omit to pay (this represents the sums still remaining at the patient's cost after 
all  refunds  have  been  received).  This  ongoing  project  is  now  in  the  testing  phase  where  different 
nudges  are  being  trialled  in  three  different  hospitals.  The  proposed  nudges  include:  introducing 
payment-related  information  as  part  of  the  default  welcome  message  in  the  hospital’s  telephone 
voice prompt; launching a campaign with a mnemonic message on the documents to bring with you 
to the hospital, namely the 4Cs: social security card, insurance card, identity card and credit card; and 
adding salience to the ‘remaining fees’ notice on the bill.  

  Promoting physical activity in older adults: SGMAP launched this project with the municipal services 
of Villeurbanne (a commune in the Metropolis of Lyon) in June 2016. Knowing that older adults are at 
a particular risk of leading sedentary lifestyles, and that physical inactivity has been established to be 
an independent risk factor for a range of threatening chronic diseases, SGMAP is working on finding 
effective behaviourally-informed interventions to address this problem.  

  Consumer  protection:  SGMAP  launched  this  project  with  the  Directorate  for  Competition  Policy, 
Consumer Affairs and Fraud Control (DGCCRF) in July 2016. Based on the idea that consumer decision-
making  may  not  be  fully  rational  and  that  firms  can  exploit  such  consumers,  SGMAP  is  working  on 
behaviourally-informed  solutions  to  devise  appropriate  policy  interventions  to  make  markets  work 
better for the consumers they are intended to serve.  

  Encouraging eco-friendly behaviour in the city’s public institutions: SGMAP launched this project in 
September 2016 with the city hall of Paris. The main aim of this project  is to encourage agents and 
citizens to adopt eco-friendly behaviour (such as turning heaters off when leaving a building) in public 
institutions such as schools, town halls, public gymnasiums, etc. 

SGMAP’s experience shows that it is not always plain sailing. It is challenging to implement the solutions and 
test them on a national scale. Robust evaluation techniques are being put into place to control for confounding 
factors  that  can  come  into  play  when  testing  a  nudge’s  impact  and  efficiency.  Ultimately,  this  is  a  reminder 
that behaviourally-based policy is experimental, and does not necessarily lead to easy answers.  

In  2015,  the  SGMAP  and  the  French  market  research  company,  BVA,  joined  forces  and  resources,  together 
with Professors Alemanno and Oullier, to create  NudgeFrance, a  foundation aimed at further promoting the 
use of behavioural insights in policymaking. In the fourth quarter of 2015, NudgeFrance launched a  national 
and international contest  for schools and universities  called “NudgeChallenge”, in the context of COP21, the 
United Nations Climate Change Conference. Participants were invited to submit videos up to four minutes long 
with their ideas for nudges to tackle climate change, to be considered by a jury and a public vote. The response 
was  a  success  -  this  first  NudgeChallenge  drew  92  entrants  with  creative  presentations,  which  led  the 
competition to be repeated in 2016.  

For further information: Françoise Waintrop, SGMAP, Francoise.WAINTROP@modernisation.gouv.fr; Mariam 
Chammat, SGMP, mariam.chammat@modernisation.gouv.fr  

One of the distinctive characteristics of SGMAP’s approach is that it recognises explicitly the ethical 
dimension  of  BIs.  The  use  of  psychology  to  frame  policy-making  has  raised  questions  about 
manipulation - whether people are being pushed into behaving in certain ways - a debate which is 
alive in the academic community too. The response is that, like all policy interventions, the aim of 
behaviourally-based actions is to improve lives and achieve better outcomes, not to worsen them. 
The  way  that  interventions  are  designed  and  information  is  presented  to  enable  people  to  make 
decisions – the ‘choice architecture’, in the psychologist’s parlance – can either be prepared blindly 
(without acknowledging people’s behavioural traits) or developed with a vision.   

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

80 

In the case of SGMAP, their approach is based on four pillars, which act as safeguards: 

1.  BIs  should  preserve  and  enhance  individuals’  autonomy.  By  contrast,  our  cognitive  biases 
leave us less well-placed to make sound decisions, and therefore less autonomous. SGMAP 
uses a framework that allows them to categorise nudge interventions on two scales, namely 
their “avoidability” and their transparency. This categorisation helps them ensure that their 
nudge interventions preserves citizens’ autonomy. 

2.  The way the intervention is designed should be pedagogic - not just to ‘nudge’, but also to 
inform about these biases. In addition, SGMAP is developing a massive open online course 
(MOOC) that promotes critical thinking and the understanding of biases. 

3.  There  should  be  transparency  in  the  use  of  BIs,  so  the  projects  are  described  on  a  public 

webpage. 

4.  The process itself should be built around robust evaluation methods, using tried-and-tested 

scientific techniques so the results are robust and fair. 

The JRC has identified six sets of issues for policy-makers to consider, when thinking about applying 
behavioural insights generally or conducting a specific behavioural study.  

Stage 

Identifying at what 
stage behavioural 
science should be 
applied to the policy-
making process  

Defining the role of 
behaviour in a policy 
initiative 

Comment 

In principle, behavioural science can be applied at any stage: 
  At  the  design  stage,  when  having  a  clear,  evidence-based,  understanding  of 
how people may respond to a policy is a valuable asset. Wrong assumptions 
at  this  stage  may  restrict  the  spectrum  of  available  policy  options  and 
jeopardise the success of the policy.  

  Later in the process, for  example when deciding  how to implement a  policy 
initiative. In the EU policy-making cycle, this would be the impact assessment 
stage.  
It is also relevant once a particular option has been decided. For example, it 
may be sensible to run a  small behavioural pilot study before committing to 
full-scale implementation.  

 

  Finally, applied behavioural science is not only relevant to the design of new 

policy initiatives; it is also relevant to the critical evaluation of existing ones.  

The  earlier  in  the  policy-making  process  behavioural  science  is  incorporated  the 
more  effective  its  contribution  is  likely  to  be.  An  early  application  will  not  only 
ensure the soundness of the policy design from a behavioural perspective, but it 
will  also  allow  for  more  time  to  conduct  behavioural  studies  in  support  of  that 
policy. 
Before  applying  behavioural  science  to  a  policy  issue,  the  relevant  behavioural 
element needs to be defined in as much detail as possible. Imagine a behavioural 
study is being considered for an EU policy initiative on effective labelling of food 
products.  Since  labels  are  intended  for  consumers  to  read,  understand,  and  act 
upon, there is clearly a  behavioural element  here. However, what  should  be  the 
aim of such a study? Discovering how consumers understand the label or finding 
out how their behaviour changes due to the label? The choice will depend on the 
objective of the policy itself. This process of narrowly defining the objective of the 
policy and the role of behaviour is therefore necessary for adequately designing a 
research project and allowing for effective behavioural support to policy. 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
Stage 
Reviewing the available 
evidence 

Estimating the value 
added of a behavioural 
study 

Specifying the unit of 
analysis 

Considering time 
constraints 

Quality of Public Administration – A Toolbox for Practitioners 

81 

Comment 
The attitudes and behaviour of citizens regarding the policy issue need to be well 
understood  before  any  behavioural  study  is  considered.  This  may  imply  carrying 
out  a  thorough  literature  review  or  consultation  with  stakeholders.  Such  a 
preliminary phase is necessary to get a first impression of people’s attitudes and 
behaviour, and of the possible policy options available. This will provide a  sound 
basis on which to begin designing a behavioural study. 
A  sound  preliminary  reflection  of  the  aims  of  a  specific  study  and  its  possible 
results is absolutely essential. Without it, the study could well fail to deliver useful 
information.  Other  kinds  of  studies  (e.g.  a  data-gathering  exercise  on  economic 
indicators)  can  generate  large  amounts  of  data,  which  in  turn  might  lead  to 
analyses that were not anticipated. Other types of study can afford ‘casting a wide 
net’ in the hope of landing valuable information. Not so with a behavioural study. 
An internal foresight exercise must be conducted, and a series of questions need 
to be asked. What do we expect from this study? What if the results are not what 
we  expected?  What  if  we  don’t  get  any  statistically  significant  results  (i.e.  if  the 
various policy options are equivalent in terms of influence on behaviour)? 
Usually  the  unit  of  analysis  for  a  behavioural  study  will  be  the  individual  (not  a 
firm,  not  a  family,  not  a  group  of  people).  It  is  the  individual  who  is  subject  to 
biases  and  heuristics  in  his  or  her  thought  processes,  the  individual  who  feels 
emotions, the  individual who has a  set  of preferences, and the individual who is 
ultimately  responsible  for  his  or  her  actions.  However,  there  is  a  demand  for 
behavioural studies to be applied to firms (and other complex bodies), since their 
behaviour  is  such  a  vital  component  of  economic  activity.  This  is  a  difficult  task. 
Complex bodies, especially large ones, tend to behave ‘rationally’. They rely on the 
expertise  of  many  experienced  and  trained  professionals  and  often  have 
structured and formalised decision-making processes. Complex bodies cannot  be 
nudged, as a nudge works well when individuals take decisions using their ‘system 
1’, the fast and spontaneous way of thinking. Complex bodies do not think fast or 
spontaneously; rather, they slowly and deliberately seek to maximise their utility 
taking  reasoned  decisions.  However,  small  enterprises  (particularly  micro-
enterprises, with fewer than 10 people) are often less structured when it comes to 
formal  decision-making.  Sometimes  owners  of  companies  –  human  beings 
themselves – will make decisions on behalf of the company, without the benefit of 
a support structure, in the same way they make decisions about their lives. Also, 
small  groups  of  people  organised  around  a  common  task  or  objective,  such  as 
research  teams  applying  for  funding,  will  sometimes  respond  to  incentives  in  a 
similar  way  to  individuals.  Therefore,  while  not  common,  there  is  scope  for 
applying  behavioural  insights  to  small  groups  of  people,  especially  if  they  lack 
formalised decision-making mechanisms.  
Behavioural studies, like most studies, require time to be conducted properly. The 
exact problem to be investigated needs to be narrowly defined and translated into 
a research design. If the study is based on an experiment, the proper experimental 
treatments and controls, the appropriate size of subject pool, and the right level 
and form of incentives all need to be established. Moreover, after the collection of 
data, time is needed to analyse and interpret the results. The simplest version of a 
project of this kind, with some lab experiments or on-line surveys, needs at least 6 
months. If a randomised control trial (RCT) is considered, the project can run for a 
year or more. In the context of the usual time horizon of the policy-making cycle, 
this  would  present  a  challenge.  Hence,  studies  of  this  kind  must  be  considered 
early in the game, if they are to be a realistic option. 

The JRC guide also includes advice on commissioning behavioural studies: experiments, randomised 
controlled trials (RCTs), surveys and qualitative research. 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

82 

It  should  be  emphasised,  however,  that  behavioural  insights  are  not  a  panacea  for  all  policy 
problems.  There  are  clearly  many  challenges  facing  policy-makers  that  are  not  concerned  with 
cognitive and emotional biases, such as major spending programmes on defence and infrastructure. 
Nevertheless, behavioural science is an increasingly essential instrument in the policy armoury. 

1.2.3  Co-production 

Laws and the regulatory environment 

Achieving outcomes by changing behaviour 

Public  administrations  are  increasingly  aware 
that  they  can  overcome  their  limitations  in 
policy  delivery  by  working  with  programme 
and  service  users,  empowering  them  to 
develop solutions as equal partners. increasing 
ownership  and  user-centricity.  As  the  OECD56 
has  observed,  public  administrations  do  not  hold the monopoly on  the  delivery of  public  services, 
and  can  benefit  from  interaction  with  other  stakeholders,  including  but  not  only  public-private 
partnerships: “in some cases, organisations from civil society may be better placed in terms of local 
knowledge  and  specialisation  to  deliver  services”.  Contemporary  governments  facing  complex 
problems do not hold all the answers: strength comes through collaboration and co-responsibility. 

Co-production 

Co-production could be described as a form of ‘outsourcing’ which involves citizens and businesses 
directly  in  the  implementation  of  public  policies  from  which  they  benefit.57  Delivery  becomes  co-
owned, more  visible  and  more  understandable  for  the  partners  in  the  process.  In  this  way,  policy 
ceases to be a ‘black box’ to beneficiaries, and where citizens are involved, becomes more legitimate 
in the eyes of the public and potentially more sustainable. Co-production is a complex term, since it 
implies  a  permanent  or  temporary  involvement  of  different  actors  in  different  stages  of  a 
sometimes-complicated  process.  These  actors  can  include  for-profit  businesses  or  non-profit 
associations  in  public-private  partnerships  (PPPs),  and  citizens  who  play  a  role  in  service  delivery, 
which can happen individually (for example, as a parent, as a guide, as a fire service volunteer), or 
collectively (for example, via NGOs for social services or park maintenance, etc.).  

“Co-delivery  of  public  services  is about  citizens  and  the  public  sector  working  TOGETHER  in  new 
creative,  innovative  and  collaborative  ways.  This  joint  working  between  professionals  and  service 
users, building on each other’s assets, experiences and expertise, enables the service to be delivered 
more efficiently.” Elke Löffler, Governance International, at the 7th European Quality Conference in 
Vilnius, 2013.  

To avoid ambiguity, full co-production can be characterised as comprising several features: citizen’s 
involvement is voluntary; they are people with assets (‘capable to contribute’), not just needs; the 

56 M. Daglio, D. Gerson and H. Kitchen, Building Organisational Capacity for Public Sector Innovation, Background Paper 
prepared for the OECD Conference “Innovating the Public Sector: from Ideas to Impact”, Paris, 12-13 November 2014. 
57 A 2015 paper by R. Calleja and U. Marantz for the New Synthesis initiative neatly distinguishes between co-creation and 
co-production of public services. Essentially, co-creation involves public authorities working with other actors "to plan and 
design” the delivery of public services “that produce better results for society", whereas co-production concerns "the 
involvement of citizens in the production/implementation of their own services …  At its core, co-production involves a shift 
in traditional models of service delivery by dividing responsibilities between government and citizens based on which actor 
is best placed to perform specific tasks”.  The authors acknowledge that the two terms are often used interchangeably in 
reality.   

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

83 

working relationship is collaborative (‘doing things with people’), not passive or paternalistic (‘doing 
things to them’); they are involved in the decision-making process; and the aim is better services and 
outcomes. 

When users and communities help to deliver services, it brings immediate and direct benefits: 

  More resources to the service, in terms of the knowledge, expertise, skills, co-operation and 

commitment of service users; 

  Better quality services, focused on the features and outcomes that users value most highly; 
  More innovative ideas for public agencies to try out; and 
  Greater transparency in the way services are delivered, supporting greater community 

involvement and open government. 

Co-production is not a new idea. It has been around for about 30 years at least, as testified by the 
examples  of  Italy’s  social  cooperatives  and  Sweden’s  children’s  day  care  cooperatives.  The 
cooperative is a well-established organisational form, has the advantage of a democratic governance 
structure  (each  member  has  an  equal  stake),  and  as  a  legal  entity,  provides  a  corporate  vehicle 
through which public authorities can contract with citizens, subject to procurement rules. 

Inspiring examples: Social cooperatives (Italy) and children’s day care cooperatives (Sweden) 

After  the  Second  World  War,  Italian  local  authorities  provided  health  and  education  services  but  not  social 
care,  and  over  time  the  traditional  role  of  families  providing  support  started  to  diminish.  As  needs  and 
expectations rose, the social cooperative model grew as a way of providing better care. The cooperatives were 
also seen as a way of improving service efficiency. In 1991, a new law created a specific legal framework for 
social  cooperatives in Italy. As their purpose, they have 'to pursue the general interest of the community in 
promoting  human  concerns  and  in  the  social  integration  of  citizens.'  Today,  there  are  more  than  7,000 
cooperatives and they have  become a  core element  in the delivery of social care for  many local authorities. 
Social cooperatives are permitted to distribute profits, so long as distributed profits are restricted to 80% of 
total profits and the profit per share is no higher than 2% of the rate on bonds issued by the Italian post office. 

Cooperative  provision  of  children’s  day  care  was  developed  in  Sweden  in  the  1980s  as  a  response  to  the 
growing demand for services and the inability of local authorities to provide sufficient capacity. The Swedish 
government made a decision to finance specialist cooperative development agencies. These agencies grew up 
in different ways in different parts of Sweden, and then federated into a national support body, which in 2006 
decided  to  work  under  the  name  of  Coompanion.  The  Coompanion  network  of  support  agencies  is  funded 
centrally  by  the  Swedish  Agency  for  Economic  and  Regional  Growth.  Specialist  advisers  have  played  an 
important  role  in  the  development  of  new  cooperatives  and  the  continuing  success  of  existing  ones.  In 
Sweden,  there  are  around  1,200  cooperatives  providing  pre-school  day  care  for  about  30,000  children, 
representing about 7% of the total. 

Source: European Commission, Directorate-General for Research and Innovation (2013), “Powering European 
Public Sector Innovation, Towards a New Architecture”, Report of the Expert Group on Public Sector Innovation 

Co-production is relevant to many policy areas. Care services have proven a particularly fruitful field. 
An  award-winning  example  is  the  care  of  elderly  residents  in  Denmark’s  Fredericia,  which  is  now 
organised  to  happen  at  home  wherever  possible,  with  multi-disciplinary  support  from  the 
municipality,  so  that  they  continue  to  live  independent  and  fulfilling  lives  within  their  own 
communities. Given the trend towards an ageing population across Europe, the principles are now 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

84 

being  exported  to  other  municipalities  in  Denmark,  and  attracting  attention  in  Norway  and 
elsewhere internationally. 

Inspiring example: “Life Long Living” in Fredericia and beyond (Denmark) 

“Life Long Living” is a new model of interaction between the elderly citizens of Fredericia who request practical 
or  personal  care  and  assistance,  and  the  municipality  -  providing  everyday  rehabilitation  and  prevention, 
rather than just  offering traditional and expensive compensatory care. The objective  is to maintain physical, 
social  and  cognitive  abilities,  to  postpone  age-related  weakening  and  dependence.  “Life  Long  Living”  is  an 
innovative initiative  whose purpose is to  maintain independent  living  for  as long as possible - to change the 
conditions of future care by focusing on the resources of each individual, and support empowerment instead 
of delivering traditional compensatory and pacifying care.  

In  2008,  it  was  projected  that  there  would  be  at  least  2000  more  people  over  the  age  of  65  in  Fredericia 
municipality by 2020, including a significant increase in the number of citizens aged over 80. This development 
presents an economic challenge, which  in 2020 would lead to an additional annual  cost  in elderly care of at 
least  DKK  46  million  (around  €6.2  million).  This  innovation  in  service  delivery  started  when  the  managers  in 
Fredericia’s social care division asked themselves: 'Should we really continue to provide ever-cheaper in-home 
cleaning, cooking, and personal care to older citizens? Or should we find out what kind of life they want to live, 
and  then  invest  in  their  ability  to  live  it?'  To  meet  this  challenge,  the  City  Council  in  Fredericia  municipality 
decided to launch an ambitious project within the municipal budget.   

The aim of the project  was to turn the interaction between the elder 
citizen  and  the  municipality  180  degrees;  by  meeting  each  individual 
with  a  focus  on  his/her  resources  and  personal  experience  of 
meaningful everyday activities, rather than a reduced focus on lack of 
functions  and  limitations.  This  180-degree  turn  of  perspective  -  from 
looking at our senior citizens as passive patients, to now meeting them 
as resourceful,  active individuals - has required a  change  of  paradigm 
throughout the entire organisation. In “Life Long Living”, the elderly citizens requesting practical or personal 
assistance  from  the  municipality  are  now  offered  to  join  an  intensive  everyday-rehabilitation-programme  in 
their  own  home,  where  they  are  trained  to  regain  their  ability  to  perform  meaningful  everyday  tasks.  The 
programme is conducted by our care providers (who have gone through special training in the approach and 
methods  within  everyday-rehabilitation)  under  the  guidance  of  interdisciplinary  teams  -  with  occupational 
therapists, nurses, physiotherapists and nurses’ assistants. 

When joining “Life Long Living”, every citizen gets an individual “Citizen Plan”, with set goals for developing or 
maintaining  their  ability  to  perform  everyday  tasks.  The  goals  are  set  in  cooperation  between  the  elderly 
citizen, his/her care provider and the interdisciplinary team, to assure a focus on meaningful activities, along 
with a cross-professional assessment. A home-training-programme is set up and the need for useful assistive 
technology  to  support  independent  daily  living  is  assessed  and  tested.  The  training  is  provided  by  the  care 
providers as part of the daily care and assistance, not in a training centre, and the goals and activities in the 
“Citizen Plan” are adjusted continuously as abilities and motivation changes. Previously, these elderly citizens 
were offered compensatory and pacifying care, which  often resulted in losing  more  everyday functions, and 
their  need  for  help  almost  always  increased  over  time.  After  implementing  “Life  Long  Living”,  and  thereby 
meeting  the  elderly  citizens  with  the  expectations  and  individual  approach  in  our  everyday-rehabilitation-
programme, their need for practical and personal assistance drops off significantly. 

Independent evaluations of the economic effects and organisational outcome of “Life Long Living” for citizens 
and staff have been conducted by DSI (the Danish Institute of Health), later named KORA (National Institute of 
analysis and research in municipalities and regions).  

The results are promising. Joining the everyday-rehabilitation-programme often means a dramatic increase in 
provided  services  in  the  beginning,  followed  by  significant  decrease  in  provided  services  on  a  longer  term, 
relatively  to  the  regaining  of  abilities.  All  together  the  need  for  practical  and  personal  care  is  reduced 
considerably:  45.9%  of  the  referred  citizens  become  completely  self-reliant  and  38.9%  become  partly  self-

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

85 

reliant. The number of requested services and the total cost for the municipality has decreased significantly, 
corresponding  to  approximately  €2  million  per  year.  Along  with  the  economic  benefits,  the  impact  of  “Life 
Long Living” can be summarised as:  

  Satisfied citizens with a high degree of self-sufficiency, who express pride and improved quality of life 

by regaining independent everyday life;  

  Satisfied employees, who express significantly greater job satisfaction and commitment working with 

the new empowering model;  

  Significant reduced need for care services, leading to a considerable decrease in total costs, enabling 

the municipality to provide more welfare for the elderly for the same amount of money. 

Since  2012,  the  model  has  been  integrated  into  the  Danish  national  budget  as  best  practice  for  all  Danish 
municipalities on how to conduct care and services for the elderly in a rehabilitative and empowering manner, 
to  meet  the  requests  from  an  increasing  number  of  older  citizens  in  the  future.  By  2014,  all  Danish 
municipalities had adopted the “Life Long Living” approach in their Health and Care Departments. “Life Long 
Living” has also attracted great attention throughout Europe, on a political as well as on a professional level. 
The  municipality  of  Fredericia  has  been  a  partner  in  developing  the  European  innovative  network  on  age-
friendly environments “AFE Innovnet”, and the “Covenant on Demographic Change - Towards an Age-Friendly 
Europe”, with the main objective to share and develop age-friendly solutions across European countries. From 
outside Europe, the Singapore Senior Minister for Health and Manpower has visited “Life Long Living” with a 
delegation  of  public  and  private  stakeholders,  and  found  the  project  to  be  inspiring  for  future  homecare  in 
Singapore. 

In 2010, the Fredericia municipality received the great innovation prize for the project from Local Government 
Denmark  (LGDK),  the  interest  group  and  association  of  Danish  municipalities.  This  was  followed  by  a  best 
practice certificate in the European Public Sector Award 2011, and being honoured in The European Year for 
Active Ageing 2012 under the category “Towards age friendly environments”. In 2015, the Danish Government 
turned the “Life Long Living” approach into a legal entitlement for all citizens in Denmark. By adapting the Law 
of Social Services, it is now mandatory for every municipality to assess the potential of rehabilitation of daily 
living  functions  for  any  citizens  who  require  social  care  services.  Followed  by  an  obligation  to  offer 
rehabilitation interventions rather than compensatory care to citizens with potential. 

For further information: Britta Hallin, Adviser of communications, britta.hallin@fredericia.dk;  
See also:  presentation of “Life Long Living” by Project Manager Helle Aavild Juhl (AFE-INNOVNET Webinar 2): 
http://www.youtube.com/watch?v=48C0be6RIFE  (minutes: 28.30 – 40.40); the Covenant on Demographic 
Change - Towards an Age-Friendly Europe and http://www.healthyageing.eu/project/afe-innovnet  

Increasingly, citizens are not waiting to be asked by public administrations to contribute to  the co-
production of public services, but are making the first move instead, including seizing the initiative 
through civil society organisations (CSOs). This can involve reaching-out to administrations, or even 
by-passing them: 

  Citizen-to-government: In some cases, CSOs exploit the possibilities of modern media (e.g. 
mobile  phone  apps)  to  demand  a  more  effective  response  from  public  administrations, 
which nevertheless retain the primary responsibility for the service. A prime example is the 
FixMyStreet  platform,  launched  in  2007  by  mySociety,  a  UK-based,  not-for-profit,  social 
enterprise,  which  uses  open  source,  report-mapping  software  and  has  become  a  global 
phenomenon, taken up by citizens in countries around the world. It is most commonly used 
for reporting street problems (such as broken lighting) to local authorities, but can be fitted 
to any project that matches geographical points to email addresses.58 

58 See also cases from the two EU-funded studies in the ‘open, modular and collaborative government’ blue box 

Topic 1.2: Instruments of policy implementation 

 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

86 

  Citizen-to-citizen:  In  other  cases,  CSOs  organise  their  own  public  services,  effectively 
substituting  for  public  administrations.  The  government  plays  no  active  role,  but  may 
provide  a  facilitating  framework.  As  an  example,  ‘Lulu  dans  ma  rue’  kiosk  is  a  French 
brokerage  started  in  March  2015  by  a  non-profit  organisation  in  Paris,  which  puts  people 
searching  for  work  in  touch  with  local  residents  who  are  looking  for  a  service.  This  social 
innovation was pump-primed by the Veolia Foundation, which helped to finance a feasibility 
study and set up the first pilot operation in Paris’s 4th district. The kiosk can be visited at the 
physical concierge or through its web platform. 

Such situations59 remain relatively sporadic and dispersed still, but are growing in scale and scope. 
They  present  public  administrations  with  a  choice:  whether  to  embrace  this  new  way  of  working 
together  with  CSOs,  or  face  in  some  cases  being  replaced  by  bottom-up  initiatives.  Clearly,  co-
production is not for everybody and the costs/benefits of harnessing service users and communities 
in  the  delivery  of  a  specific  public  service  will  vary.  To  help  weigh  up  the  pros  and  cons  of  co-
production  and  ways  forward,  further  case  studies,  research  and  resources  are  available  from 
existing organisations, such as Governance International’s good practice hub.  

59 See also the 2010 EU-funded Study on Collaborative Production in eGovernment, which includes six case studies 

Topic 1.2: Instruments of policy implementation 

 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

87 

1.3  Continuous improvement and innovation 

from 
the 

Encouraging external scrutiny 

Monitoring, evaluation and audit 

One of the key qualities of good policy development is 
that implementation is subject to review and reflection, 
so  that  lessons  are  learned,  adaptations  are  made,  or 
even  policy  is  abandoned  in  response  to  findings.  This 
the  executive,  open-
requires  openness 
mindedness 
its 
electorate 
from 
representatives, and courage on both sides to embrace experimentation and not rush to judgement. 
This section examines: systematic monitoring, planning and managing evaluations, and the growing 
role of performance audits in assessing whether implementation is progressing to plan, policies are 
achieving  their  high-level  objectives,  and  value-for-money  is  being  delivered;  and  the  value  of 
external  scrutiny  in  driving  up  the  standards  of  public  administration.  In  this  spirit  of  continuous 
improvement,  the  section  looks  finally  at  public  sector  innovation  in  its  myriad  forms,  and  how 
public administrations are creating innovative cultures to stimulate new ways of working. 

Fostering innovation 

and 

1.3.1  Monitoring, evaluation and performance audit 

Monitoring, evaluation and audit 

Monitoring  and  evaluation  has  often  been  seen  as  an 
unwelcome distraction, imposed by funding providers on 
recipients: whether this is donors providing development 
aid,  the  EU  seeking  evidence  of  use  of  co-financing,  or 
budget 
seeking 
Finance  Ministries 
beneficiaries are delivering ‘something for something’. In 
these  various  scenarios,  monitoring  and  evaluation  is 
often treated as an add-on to the policy process. Increasingly, however, monitoring and evaluation is 
recognised  as  integral  to  policy  success.  Given  the  complexities  of  the  real  world  -  complicated 
choices, tough decisions and unexpected outcomes - the policy process is inevitably iterative, and a 
feedback loop is essential to make ‘adopt-and-adapt’ a reality (see topic 1.1). 

Encouraging external scrutiny 

Fostering innovation 

proof 

that 

Monitoring  is  a  term  that  is  widely  used,  but  can  mean  different  things  to  different  people.  For 
some,  monitoring  is  supervision;  for others,  it  is  synonymous with  control.  For  the  sake of  clarity, 
monitoring  is  defined  here  as  a  systematic  process  of  collecting  data  to  track  inputs,  outputs, 
outcomes and impacts throughout implementation, and to inform management and stakeholders on 
performance and progress. Monitoring can be applied to policies, programmes, projects and public 
services, but also organisations (see theme 4), and systems of governance (see theme 3).  

Most  public organisations are familiar with  evaluation  in the context of  policies and programmes, 
especially ESIF, taking the information from monitoring (and other sources) at key moments in time, 
and  systematically  assessing  relevance,  efficiency,  effectiveness,  impact  and  sustainability.  While 
monitoring provides a flow of data for real-time decisions, evaluation is a comprehensive stock-take 
of where things stand. This can occur before, during or after an intervention. However, evaluations 
are far less commonplace and are not applied comprehensively to policies and programmes across 
Europe, despite the conventional wisdom that evaluation is a major stage in the policy process and 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

88 

an  important  contributor  to  the  evidence  base.  The  gap  has  been  partly  filled  by  performance 
audits,  conducted  by  countries’  supreme  audit  institutions  (SAIs),  but  as  the  SAI’s  main  remit  is 
always financial audit, performance audits are also not a consistent feature of the European policy-
making landscape.  

In  the  interests  of  a  common  understanding,  the  following  table  summarises  the  key  differences 
between monitoring, evaluation and performance audit.60 

Monitoring 

Evaluation 

Performance audit 

What 

Why 

When 

Tracking performance, and 
progress against the plan 
(expectation) 

For operational reasons – to 
learn lessons and take 
corrective action in real-time, if 
required 
Regular intervals during 
implementation 

Who 

Managers and staff involved in 
implementation 

Assessing the relevance, 
efficiency, effectiveness, 
impact and sustainability of 
policies and programmes 
For strategic reasons – to 
ensure the policy addresses the 
identified problems and 
objectives, and learn lessons 
Usually at specific points 
(before, during and after 
implementation). 
Internal units or external 
consultants not involved in 
design or implementation* 

Examining the efficiency and 
effectiveness of government 
undertakings, programmes or 
organisations 
For accountability reasons – to 
ensure public funds are being 
used appropriately and identify 
improvements, if necessary 
Usually at specific points 
(during and after 
implementation). 
Usually, qualified auditors from 
the SAI, independent of design 
and implementation 

* The exception is ex ante evaluation, where the evaluators are expected to influence the design and future implementation 
through their independent findings. 

In  practice,  the  boundaries  between  monitoring  and  evaluation  are  increasingly  blurred,  as 
evaluation can take place in real-time, during the early stages of a new policy or programme or on an 
ongoing  basis.  The  reality  is  that  all  administrations  engage  in  some  form  of  monitoring  and 
evaluation  of  their  activities,  whatever  they  call  it.  The  only  question  is  whether  this  is  casual  or 
structured.  The  latter  is  an  essential  condition  of  receiving  EU  funds,  especially  under  ESIF 
regulations. As a systematic process, monitoring and evaluation has five steps, shown below.  

Step  1  is  orientation:  what  should 
be  monitored  and  evaluated,  why, 
and  how  will  the  information  be 
used?  This  preparatory  phase,  however  short  or  long,  informs  the  following  four  steps.  The  main 
ingredients of orientation are set out overleaf. 

60 As performance audit is an explicit responsibility of SAIs outside the administration, the rest of this topic mainly focuses 
on monitoring and evaluation, with performance audit described by way of contrast with evaluation. 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

89 

Ingredient 

Objectives 

Purpose 
There must  be clarity about  what the policy, programme, project, service or organisation is 
seeking  to  achieve.  In  other  words,  clear  goals  against  which  performance  can  be  assessed 
(which can be both strategic and specific / operational), as well as a plan to get there. 

Methodology 

Responsibility  The  role  of  defining  indicators,  collecting  data,  analysing  the  findings,  and  assessing 
performance and progress is assigned to an official or unit (or managing external experts, in 
the case of evaluation), as well as reporting lines to management and policy-makers. 
The  monitoring  arrangement  must  include  a  system  to  gather  data  and  to  analyse  the 
findings.  As  an  integral  element  of  designing  indicators,  it  must  be  known  from  the  outset 
what the source of the information will be and how often it will be produced. 
Management  Once information is gathered by the responsible official(s) on performance, including against 
indicators, this must be fed into the policy process as part of the evidence base for adjusting 
policy. This means there must be an outlet for the information to be used. For example, in the 
case  of  ESIF,  this  would  be  the  managing  authority61,  but  ultimately  the  monitoring 
committee through the annual implementation report. 

Useful guidance  in preparing for monitoring and evaluation can be  found, for example, in the  EU-
financed Community of Practice on Results Based Management (COP RBM), the OECD’s Summary of 
Key Norms and Standards for Evaluation62 and the Civicus Toolkit on Monitoring and Evaluation.  

Public  administrations  may  also  find  outcome  mapping  useful  as  a  complementary  tool  and 
approach for planning, monitoring and evaluation, as its focus is on the behavioural changes brought 
is  available  from  the 
in  socio-economic  development.  Guidance 
about  by 
aforementioned COP RBM63, and also the United Kingdom’s Overseas Development Institute and the 
Outcome Mapping Learning Community. 

interventions 

At  the  outset,  it  is  essential  that  there  is  political  buy-in  to  monitoring  and  evaluation,  to  have 
confidence that the learning points will be internalised when they emerge. This willingness to absorb 
findings, and momentum, can be built over time by collecting and presenting relevant information 
that  satisfies  the  information  needs  of  ministers,  mayors  or  councillors,  in  usable  formats  (as  per 
step 5). 

Evaluations  can  suffer  from  poor  planning,  which  leads  to  ambiguity  in  purpose  and  objectives, 
vagueness in scope, lack of rigour in the analysis, and ultimately blandness in the conclusions which 
makes  it  hard  to  extract  useful  recommendations.  Centres  of  Government  and  line  ministries  can 
improve the governance of evaluation by publishing procedures with clear guidelines on the timing 
of  evaluations  for  different  purposes  (ex-ante,  interim, etc.),  the  standards  they  should  meet,  and 
the  techniques  that  should  be  employed.  The  European  Commission’s  website  includes  useful 
guidance  on  planning  and  performing  evaluations,  especially  within  the  context  of  evaluating  ESIF 
programmes for 2014-2020 (see also topic 8.3). 

The decision to move into performance audits can only be taken by the SAI itself, with the consent of 
parliaments that vote directly for SAI funding. However, there is useful guidance on the websites of 
the  International  Organisation  of  Supreme  Audit  Institutions  (INTOSAI),  as  well  as  the  European 

61 Or possibly the intermediate body as the first port-of-call. 
62 This is aimed at developing development cooperation, but interesting nonetheless. 
63 Sourcebook, pages 179-182 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

90 

Court of Auditors (ECA). INTOSAI has published implementation guidelines64, which are the basis of 
the  ECA’s  Performance  Audit  Manual.  With  their  focus  on  effectiveness,  efficiency  and  economy, 
performance audits serve a similar purpose to evaluations. The ECA’s manual usefully distinguishes 
between  the  two  disciplines,  the  principal  difference  being  the  status  of  the  SAI  in  holding 
administrations  to  account.  Unlike  financial  audit,  which  follows  a  standardised  methodology, 
performance  audits  tailor  their  methods  to  the  subject  under  consideration,  in  accordance  with 
INTOSAI guidelines. 65 

Evaluation v performance audit 

Both activities involve the examination of policy design, implementation processes and their consequences to 
provide an assessment  of economy, efficiency and effectiveness of an entity or activity. They require similar 
knowledge,  skills  and  experience  and  involve  similar  methods  for  collecting  and  analysing  data.  The  main 
difference is the context in which they take place and the purpose of each. Performance audit is superimposed 
on an accountability framework, which implies that  the Commission and other institutions and organisations 
concerned are held responsible for the management of EU funds and should provide meaningful and reliable 
information  to  demonstrate  and  take  responsibility  for  performance  in  light  of  agreed  expectations. 
Performance audits are carried out by auditors who maintain their independence to select and determine the 
manner in which to conduct their work, and report the results to the discharge authority (European Parliament 
acting  on  the  recommendation  of  the  Council).  It  is  therefore  not  the  purpose  of  the  Court's  performance 
audits  to  deliver  comprehensive  evaluations  of  EU  activities.  This  is  the  responsibility  of  the  Commission, 
Member  States  and  other  managers  of  EU  activities.  However,  performance  audits  will  usually  include 
evaluative  elements  of  selected  subjects  and  consider  evaluation  systems  and  information  with  a  view  to 
assessing  their  quality  and,  when  they  are  considered  to  be  satisfactory  and  relevant,  use  evaluation 
information as audit evidence.    

Source: European Court of Auditor’s Performance Audit Manual 

In  accordance  with  INTOSAI  guidelines  (ISSAI  3000,  2.1),  “the  mandate  of  performance  auditing 
should cover the state budget and all corresponding government programmes. The auditor must be 
free  to  select  audit  areas  within  its  mandate.  Political  decisions  and  goals  established  by  the 
legislature  are  the  basic  frame  of  reference.  A  performance  audit  may,  as  a  result  of  its  findings, 
question  the merits  of  existing  policies.  Performance audits  are  in  general  ex  post  audits  that  deal 
with current issues. High levels of quality in the work must be promoted and secured.” 

At  the  heart  of  the  monitoring 
system  are  performance  indicators. 
These  can  be  either  quantitative  or 
qualitative, depending on the nature of the indicator (or ‘metric’). The indicators that are considered 
to be the most important are often called ‘key performance indicators’ or KPIs. As the COP RBM has 
highlighted in its Sourcebook, different terms are used for indicator types by various organisations 

64  ISSAI  3000  Implementation  Guidelines  for  Performance  Auditing:  Standards  and  guidelines  for  performance  auditing 
based  on  INTOSAI's  Auditing  Standards  and  practical  experience  and  ISSAI  3100  Performance  Audit  Guidelines  –  Key 
Principles. 
65 “Performance auditing is not overly subject to specific requirements and expectations. While financial auditing tends to 
apply relatively fixed standards, performance auditing is more flexible in its choice of subjects, audit objects, methods, and 
opinions. Performance auditing is not a regular audit with formalized opinions, and it does not have its roots in private 
auditing. It is an independent examination made on a non-recurring basis. It is by nature wide-ranging and open to 
interpretations. It must have at its disposal a wide selection of investigative and evaluative methods and operate from a 
quite different knowledge base to that of traditional auditing. It is not a checklist-based form of auditing” (INTOSAI 
Performance Audit Guidelines, ISSAIs 3000, 1.2). 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

91 

(including  the  EU  and  OECD66),  and  hence  the  following  is  a  guide  for  common  understanding  on 
terms in the rest of this section.  

Type 

Inputs 

Outputs 

Results 

Impact 

Context 

Brief description 
These  are  the  resources  used  in  the  policy,  programme,  project,  service  or  organisation, 
whether financial, human, material, or technological. In practice, this indicator is often shown 
simply in price/cost terms (the amount spent on activities).  
These  arise  from  the  activities  of  the  policy,  programme,  project,  service  or  organisation  and 
are  deliverables  (such  as  products,  services,  buildings  and  infrastructure).  Monitoring  and 
evaluation should consider not just the existence of the output, but its quality, characteristics, 
functionality, and timeliness too. 
These are the direct changes that are brought about by the activity. For example, if the output 
of  the  activity  ‘rehabilitate  railways’  was  50km  of  track  re-built,  then  the  immediate  benefits 
might be measured as higher design speed and/or higher maximum load, which would be the 
specific / operational objectives.   
These are the development effects that accrue from the results of the activity (corresponding 
to  strategic  objectives).  Using  the  railways  example,  the  impact  might  be  faster  travel  time, 
more  trains  per  hour  using  the  track,  and/or  improved  safety  record  (fewer  accidents).  The 
longer-term and more ‘global’ impact might be felt in better socio-economic performance, such 
as GDP, jobs and well-being, but these are harder to link causally to the activity and its outputs.    
These  are  variables  in  the  wider  socio-economic  environment  that  can  have  an  exogenous 
effect on the inputs, activities, outputs, results and impact. 

The convention is that each performance indicator should contain five components for the sake of 
completeness. 

Component 
Definition 

Source 

Timescale 

Baseline 

Purpose 
The indicator should be clearly stated, so that there is no ambiguity in the minds of the reader, 
or indeed the people providing or collecting the  information, about  its content  and meaning. 
While  the  indicator  may  be  described  in  shorthand  (such  as  ‘jobs  created’),  it  should  include 
explanations,  and  if  necessary  references  to  official  definitions,  statistical  sources  and 
documents, that the reader can follow to understand fully the use of specific terms (e.g. “’jobs 
created’  means  the  additional  employment  by  beneficiary  organisations  of  persons  that 
undertook the activity supported under the programme and were still in post 12 months after 
the activity was completed”). 
Again,  the  indicator  should  be  clear  on  how  information  will  be  gathered,  whether  it  is 
quantitative  or  qualitative.  This  might  involve  officially  recognised  sources  (such  as  Eurostat, 
national  statistics  agency,  ministry,  etc.)  or  other  authoritative  source  (credible  research 
organisation, academic institution, etc.). It might necessitate an original survey, in which case 
the methodology must be robust, and ideally elaborated when the indicator is designed.   
The indicator should be accompanied by a statement of the frequency in which information will 
be collected and reported, and over what timeframe. In other words, it should state whether 
the intervals that information will be gathered (e.g. ongoing/real-time, monthly, quarterly, six-
monthly,  annually,  biannually).  Depending  on  the  indicator  and  source,  this  might  be  highly 
specified to reflect publication dates of official data (e.g. “on 15 March and 15 September each 
year”).  The indicator should state ‘from when – to when’. For example, it might be appropriate 
to  start  gathering  information  as  soon  as  the  activity  starts,  or  only  a  year  after  it  has  been 
completed, and to stop assessing performance after, say, 2 years. 
In many cases, the purpose  of the indicator  will be to track performance over time, in which 
case the reader needs to know the starting point. Typically, the baseline position will be set out 
at the same time as the indicator is adopted, so that again there is no ambiguity later. 

66 For example, OECD refers to ‘results’ in the table as ‘outcomes’, and sees ‘outputs, outcomes and impact’ as 3 levels of 
‘results’ instead. 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

92 

Benchmarks  Whether  the  comparison  is  over  time  or  with  peer  performance  (other  countries,  regions, 
localities,  etc.),  it  is  common  to  establish  ‘comparators’  as  reference  points.  If  the  aim  is  to 
achieve eventually a  certain level or threshold of performance, these benchmarks are  usually 
called ‘targets’, usually stated with a time by which the target will be achieved (e.g. “in three 
years” or “by 2020”). 

Performance  indicators  are  an  easy  concept  to  grasp,  but  harder  to  design  and  operationalise  in 
practice. Generating indicators raises all sorts of questions:  

  Does the indicator reflect accurately the objective? Does it capture what we are trying to 

do and achieve? 

  Is  the  information  available?  We  can  try  and  design  the  ‘perfect’  indicator,  but  can  it  be 
measured? Is the cost of gathering information manageable? Does a baseline exist, or is it 
too late to gather the information to make one? If it is a qualitative indicator, how can we 
compare  it  over  time  or  with  our  peers’  performance?  Are  we  falling  into  the  trap  of 
designing indicators around the available data – only measuring the measurable? 

  Is  the  situation  too  complex  to  ‘collapse’  it  into  an  indicator?  Will  it  give  us  a  false 
impression  of  our  performance?  Or  worse  still,  by  simplifying  a  complex  situation  in  an 
indicator, will it push us to emphasise certain elements at the expense of others, and lead to 
‘bad policy’? 

  Does the indicator really tell us what we think it tells us?  For example, if the indicator is 
‘number  of  complaints’,  does  an  increase  mean  that  the  service  is  performing  worse  than 
before, or that the  institution has been successful in becoming more open and welcoming 
feedback? In the example, is ‘number’ the best choice of metric – what does it tell you, if the 
number of service users is also going up, should it be ‘percentage’ instead and does that tell 
you much more? 

  Does measuring performance create its own incentives? Will the presence of an indicator 
by itself change behaviour: for the better (focus implementers on what is most important) or 
for worse (concentrate on doing only enough to satisfy the indicator)?  

When  approaching  monitoring  and  evaluation,  a  careful  choice  of  indicators  is  critical,  as  is  their 
content.  This  includes  decisions  on  details  (e.g.  whether  ‘number’  or  ‘percentage’),  but  also  not 
relying  too  heavily  on  quantitative  indicators  and  especially  single  metrics.  The  risks  of  over-
simplicity can be compounded if administrations ‘go public’ with their indicators, and expect citizens 
to  judge  the  government’s  or  judiciary’s  record  on  the  outcome.  It  is  almost  impossible  to  design 
indicators  which  encapsulate  all  the  dimensions  and  subtleties  of  a  policy  in  a  single  metric, 
especially  those  which  are  behavioural,  cultural  or  otherwise  qualitative,  and  therefore  hard  to 
quantify. 

There is also the risk of the observer effect: the act of monitoring itself changing the performance of 
the  observed  (the  implementer).  It  is  essential  that  ‘the  tail  doesn’t  wag  the  dog’:  striving  to  hit 
targets  ends  up  driving  decision-making,  distorting  priorities  compared  with  the  diversity  of 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

93 

problems  being  addressed,  and  directing  resources  into  some  areas  at  the  expense  of  others. 
Indicators  themselves  are  not  solutions,  they  are  only  guides  to  whether  proposed  solutions  are 
working. 

The importance of indicators is illustrated by their absence; without them, public policies can end up 
losing direction. Targets are one step further - something to aim for, as the name suggests. If they 
are not  hit, the  challenge  is to understand why the target was missed, by how much, and to alter 
direction to get closer. In some cases, it may be the target itself that is unrealistic and unattainable, 
and the sensible response is to adjust or to abandon it, and to embark on a new course of action.  

Designing indicators for usefulness 
  Develop a portfolio of indicators which capture many different aspects of a policy challenge, to build up a 

fuller and more sophisticated picture, while avoiding information overload.  

  Don’t rely on indicators alone to inform you about performance. A more rounded assessment of 

accompanying indicators with insights into what is happening on the ground. For example, a programme 
might achieve its goal of laying 50km of highway, but unless the supervising engineer can validate that the 
road has been constructed to agreed specification and standards, the output will be poor value for money 
and not built to last. 

  Above all, emphasise interpretation (step 4) and application (step 5). Indicators should be treated as a 

management tool for improving governance and the future design of policies, programmes and 
projects/services, not an absolute test of their validity, given all sort of factors might be in play. 

Step  3  is  data  collection.  Clearly, 
how  information  is  gathered  will 
depend  on  the  source  and  the 
frequency  with  which  data  can  be  made  available.  These  factors  will  be  determined  when  the 
indicator itself is defined. The frequency may vary from continuous (in the case of price-based input 
data)  to,  at  the  other  extreme,  once  a  decade  (in  the  case  of  a  population  census,  for  example); 
administrations will wish to tend towards regular flows of information for practical reasons. Ideally, 
all chosen indicators will be capable of monitoring and evaluation at minimum administrative cost. 
Information that is automatically generated through day-to-day activities, or regularly assembled by 
official sources (such as statistics agencies, ministries or municipalities) are ideal, but administrations 
should  not  be  constrained  by  immediate  availability.  In  some  cases,  surveys,  panels,  self-
assessments and other forms of original research might be necessary – but the cost will need to be 
weighed up and the organisation factored into the planning. 

The  United  Nations  Development  Programme  has  identified  innovative  approaches  to  monitoring 
and evaluation, which include novel and user-centric ways of data collection. These include methods 
for increased participation of citizens, either directly such as providing input through SMS reporting 
or  story-telling,  or  indirectly,  with  information  being  collected  and  analysed  remotely  and  in  the 
aggregate. The example of Italy’s OpenCoesione platform and ‘monithon’ (see  topic 2.3) is a prime 
example of a participatory approach, in this case to monitoring ESIF performance. 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

94 

Innovations in monitoring and evaluating results 

Innovation 
Crowdsourcing  

Real-time, simple 
reporting  

Participatory 
statistics 
Mobile data 
collection  
The micro-
narrative 
Data exhaust  

Intelligent 
infrastructure  

Remote sensing  

Data visualisation  

Multi-level mixed 
evaluation method 

Outcome 
harvesting  

Overview 
A large number of people actively report on a situation around them, often using mobile 
phone technology and open source software platforms  
A means to reduce, to a minimum, the formal reporting requirements for programme and 
project managers and free up their time to provide more frequent, real-time updates, which 
may include text, pictures, videos that can be made by computer or mobile devices  
An approach in which local people themselves generate statistics; participatory techniques 
are replicated with a large number of groups to produce robust quantitative data  
The targeted gathering of structured information using mobile phones, tablets or PDAs using 
a special software application. 
The collection and aggregation of thousands of short stories from citizens using special 
algorithms to gain insight into real-time issues and changes in society  
Massive and passive collection of transactional data from people’s use of digital services like 
mobile phones and web content such as news media and social media interactions  
Equipping all – or a sample of – infrastructure or items, such as roads, bridges, buildings, 
water treatment systems, hand-washing stations, latrines, cook stoves, etc., with low-cost, 
remotely accessible electronic sensors  
Observing and analysing a distant target using information from the electromagnetic 
spectrum of satellites, aircrafts or other airborne devices  
Representation of data graphically and interactively, often in the form of videos, interactive 
websites, info-graphs, timelines, data dashboards, maps, etc.  
This approach includes the deliberate, massive and creative use of mixed (quantitative and 
qualitative) methods on multiple levels for complex evaluations, particularly for service 
delivery systems  
An evaluation approach that does not measure progress towards predetermined outcomes, 
but rather collects evidence of what has been achieved, and works backward to determine 
whether and how the project or intervention contributed to the change  
Source: UNDP (2013), “Innovations in Monitoring & Evaluating Results, Discussion paper” 

Step 4 is analysing the data. This is 
the point  at which monitoring and 
evaluation  have  a  shared  interest. 
Both  instruments  are  about  interpreting  information  to  learn  lessons,  the  main  differences  being 
when these reviews take place and by whom. Monitoring can be seen as a form of ongoing, internal, 
informal  ‘evaluation’  with  an  operational  focus;  formal  evaluations  tend  to  be  external,  a  more 
‘static’ snapshot, and for strategic purposes. 

In both cases, the administration needs to create the time and space for reflecting on the findings 
from performance measurement. With evaluations, this happens as an automatic by-product of the 
process;  by  commissioning  evaluators,  the  administration  is  creating  an  external  stimulus  for 
scrutinising its own performance. Even if it chooses not to publish the product or take on board the 
recommendations,  the  very  fact  of  evaluation  causes  questions  to  be  asked.  The  same  should 
happen  with  all  monitoring,  as  a  series  of  ‘mini-reviews’.  In  the  case  of  ESIF,  this  is  exactly  what 
occurs,  but  again  because  of  external  stimulus:  monitoring  is  a  regulatory  requirement.  The 
challenge  is to translate the  ESIF monitoring mentality into standard public sector practice, and to 
make  the  process  fit  the  purpose  of  day-to-day  management,  whether  of  policies,  programmes, 
projects, services or organisations. 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

95 

In  analysing  data,  administrations  should  consider  international  governance  indices.  These  are  a 
form of context indicator, in the sense of describing the global environment in which their policies, 
programmes,  projects,  services  and  organisations  are  operating,  and  in  some  cases,  constitute 
exogenous  factors  (for  example,  a  country’s  performance  on  rule  of  law  has  an  influence  on  the 
success of activities to stimulate investment). But they also offer interesting and useful benchmarks 
for  the  whole  reform  process  itself.  The  table  overleaf  provides  an  overview  of  widely  recognised 
indices  and  reports.  The  extent  to  which  global  indices  have  mushroomed  in  recent  years  is  an 
indicator  itself  of  the  growing  interest  in  administrative  reform.  The  fact  that  recent  international 
studies have featured titles like "Democracy in limbo" (EIU) and "Stuck in transition" (EBRD) is a sign 
of the pressure on public administrations to deliver good governance. 

International experts have been active in producing guidance on how to use these indices, including 
the  OECD’s  2006  “Uses  and  Abuses  of  Governance  Indicators”,  the  UNDP’s  2009  “User's  Guide  to 
Measuring  Public  Administration  Performance”  and  the  Hertie  School  of  Governance’s  2014 
“Governance Report”.  

Governance  indices  are  very  valuable,  but  should  be  used  and  interpreted  with  care,  to  avoid 
reading too much into individual numbers without understanding first what lies behind them. Many 
indices employ a ‘league table’ style to show the performance of individual countries in relation to 
others.  This  is  an  eye-catching  device  and  can  help  to  focus  hearts  and  minds  on  the  underlying 
problems,  but  must  be  treated  with  caution.  Like  all  indicators,  every  index  has  both  its  valued 
features  and  its  flaws.  Many  are  composites,  meaning  each  factor  must  be  weighted,  a  process 
which is open to debate (and hence, the authors typically present the methodology and rationale). 
Moreover, comparative positions of countries will always remain relative: there must be a first and 
last. While movement up and down the table over the years is an interesting guide to the effect of 
changes  in  policy  and  practice,  public  administrations  are  not  in  competition  except  with 
themselves.  The  prize  for  improving  governance  is  not  promotion  to  a  super-league  of  public 
authorities,  but  better  societal  outcomes:  prosperous  economies,  cohesive  societies,  sustainable 
environments.   

Hence,  the  key  is  to  dig  below  the  headline  numbers  and  ‘league’  positions,  and  to  pose  the 
following questions:  

  What are the factors that explain our performance? What are the elements of a composite 
index  that  the  author  has  considered?  What  can  we  learn  from  the  author’s  underlying 
analysis of our policies and practices?  

  What are the reasons for higher-placed countries showing a relatively better performance? 
What can we learn from their policies and practices? Is there anything that is transferable? 

  If we are showing a better / worse position over time in the ‘league table’, is this down to 
changes we have made, or has everyone got better / worse? If everyone has got worse, then 
doing better is no basis for complacency - what else can we improve? 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

96 

Index 

Corruption 
Perception Index 
(CPI) 

Source 
Transparency 
International 

Democracy Index 

Economist 
Intelligence Unit 

Doing Business 
Indicators 

World Bank 

Freedom in the 
World 

Freedom House 

Global 
Competitiveness 
Index (GCI) 

World Economic 
Forum 

How are we doing? International Governance Indices and Regular Reports 

Description 

The CPI scores and ranks countries/territories based on how corrupt a country’s public sector is perceived to be. It is a 
composite index, a combination of surveys and assessments of corruption, collected by a variety of reputable institutions. 
A country/territory’s score indicates the perceived level of public sector corruption on a scale of 0-100, where 0 means that 
a country is perceived as highly corrupt and a 100 means that a country is perceived as very clean. A country's rank 
indicates its position relative to the other countries/territories included in the index. Ranks can change merely if the 
number of countries included in the index changes. Annual update. 
The Economist Intelligence Unit’s index of democracy, on a 0 to 10 scale, is based on the ratings for 60 indicators grouped 
in five categories: electoral process and pluralism; civil liberties; the functioning of government; political participation; and 
political culture. Each category has a rating on a 0 to 10 scale, and the overall index of democracy is the simple average of 
the five category indexes. Annual update. 
Doing Business measures the quality of the business environment. In 2017, the report covers the following 11 categories: 
starting a business; dealing with construction permits; getting electricity; registering property; getting credit; protecting 
minority investors; paying taxes; trading across borders; enforcing contracts; resolving insolvency; and labour market 
regulation. The methodology evolves over time; therefore year-to-year performance comparisons might not automatically 
be meaningful. Data is not based on firm or household surveys but on expert assessments. 
Freedom in the World is a survey-based annual global report on political rights and civil liberties, composed of numerical 
ratings and descriptive texts for each country. The report’s methodology is derived in large measure from the Universal 
Declaration of Human Rights, adopted by the UN General Assembly in 1948. Freedom in the World operates from the 
assumption that freedom for all peoples is best achieved in liberal democratic societies. The report assesses the real-world 
rights and freedoms enjoyed by individuals, rather than governments or government performance per se. Political rights 
and civil liberties can be affected by both state and non-state actors. While both laws and actual practices are factored into 
the ratings decisions, greater emphasis is placed on implementation. 
The GCI defines competitiveness as the set of institutions, policies, and factors that determine the level of productivity of a 
country. It measures 12 pillars of competitiveness, of which, the first pillar concerns the "quality of Institutions". It 
considers the legal and administrative framework within which individuals, firms, and governments interact to generate 
wealth. This includes, for example, factors such as: a sound and fair institutional environment, protection of property 
rights, government attitudes toward markets and freedoms and the efficiency of its operations, in relation to excessive 
bureaucracy and red tape, overregulation, corruption, dishonesty in dealing with public contracts, lack of transparency and 
trustworthiness, inability to provide appropriate services for the business sector, and political dependence of the judicial 
system,  the proper management of public finances (pillar 3) and private sector ethics and transparency through the use of 
standards as well as auditing and accounting practices. Annual Report. The 2014 report covers 144 economies. It is based 
on a mixture of data sets – both quantitative and opinion survey based. 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

97 

Government at a 
Glance 

OECD 

Prosperity Index 

Legatum Institute 

Quality of 
Government (QoG) 

Quality of 
Government 
Institute, University 
of Gothenburg 

Sustainable 
Governance 
Indicators (SGI) 

Bertelsmann 
Foundation 

Government at a Glance provides a dashboard of key indicators contributing to the analysis and international comparison 
of public sector performance. Indicators on government revenues, expenditures, and employment are provided alongside 
key output and outcome data in the sectors of education and health. Government at a Glance also includes indicators on 
key governance and public management issues, such as transparency in governance, regulatory governance, new ways of 
delivering public services and HRM and compensation practices in the public service. It is published every two years, and 
covers only OECD members, thus some EU countries are not included (BG, CY, HR, MT, LT, LV, RO). 
The Legatum Prosperity Index offers an insight into how prosperity is forming and changing across the world. The Index is a 
measurement of prosperity based on both income and well-being. Traditionally, a nation’s prosperity has been based 
solely on macroeconomic indicators such as a country’s income, represented either by GDP or by average income per 
person (GDP per capita). However, most people would agree that prosperity is more than just the accumulation of material 
wealth. It is also the joy of everyday life and the prospect of being able to build an even better life in the future. 
One of the eight pillars of the index covers governance. Variables for assessing governance include government stability, 
government effectiveness, and rule of law, including subcategories of: regulation, separation of powers, political rights, 
government type, political constraints, efforts to address poverty, confidence in the judicial system, business and 
government, corruption, government effectiveness, environmental preservation, separation of powers, government 
approval, voiced concern, confidence in military, confidence in honesty of elections. The annual report covers 142 
countries.  
The QoG is a survey with an information data set on the structure and behaviour of public administration. The data is 
based on a web survey of 1294 experts from 159 countries. The dataset covers different dimensions of Quality of 
Government, such as politicisation, professionalisation, openness, and impartiality. The QoG web survey is an ongoing 
project and data is continuously updated to increase the number of participating experts and the number of countries 
represented by the survey.  
SGI is a platform built on a cross-national survey of governance that identifies reform needs in 41 EU and OECD countries. 
The SGI brings together a broad network of experts and practitioners aiming to understand what works best in sustainable 
governance. SGI themes include: Policy Performance – including economic policies, social policies, environmental policies; 
Democracy - quality of democracy; and Governance - executive capacity, executive accountability 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
Transformation 
Index 

Bertelsmann 
Foundation 

Transition Report 

European Bank for 
Reconstruction and 
Development 

WJP Rule of Law 
Index 

World Justice Project 

Worldwide 
Governance 
Indicators (WGI) 

World Bank Group 

Quality of Public Administration – A Toolbox for Practitioners 

98 

The Bertelsmann Transformation Index (BTI) is based on 129 country reports. As the focus is on emerging economies, just 
11 EU countries from the latest accessions are covered (not CY). The BTI analyses and evaluates whether and how 
developing countries and countries in transition are steering social change toward democracy and a market economy. The 
BTI aggregates the results of this comprehensive study of transformation processes and political management into two 
indices: The Status Index and the Management Index. The Status Index, with its two analytic dimensions – one assessing 
the state of political transformation, the other the state of economic transformation – identifies where each of the 129 
countries stand on their path toward democracy under the rule of law and a market economy anchored in principles of 
social justice. Focusing on the quality of governance, the Management Index assesses the acumen with which decision-
makers steer political processes. The BTI is published every two years. The BTI does not cover countries that were 
members of the Organisation for Economic Cooperation and Development (OECD) by the year 1989. The Transformation 
Index is based on a qualitative expert survey in which written assessments are translated into numerical ratings and 
examined in a multi-stage review process, to make them comparable both within and across regions. Assessed variables 
include: stateness; political participation; rule of law; stability of democratic institutions; political and social integration; 
level of socioeconomic development; organisation of the market and competition; currency and price stability; private 
property; welfare regime; economic performance; sustainability; transformation management (level of difficulty & 
management performance); steering capability; resource efficiency; consensus building; internal cooperation; and strategic 
outlook. 
The Transition Report 2016-17 deals exclusively with the subject of inequality and economic inclusion. It draws on the third 
round of the Life in Transition Survey (LiTS III), a household survey conducted by the EBRD and the World Bank in 34 
countries in 2015-2016, comparing the results with the first two rounds (2006 and 2010). It also uses the results of a 
unique survey of bank managers (the EBRD’s Banking Environment and Performance Survey) and other data sources. Of 
the EU countries, it only covers the Member States of the 2004 accession onwards.  
The WJP Rule of Law Index offers a detailed, multidimensional view of the extent to which countries adhere to the rule of 
law in practice, and is the most comprehensive index of its kind. The Index’s scores are built from the assessments of 1,000 
respondents per country and local legal experts, ensuring that the findings reflect the conditions experienced by the 
population, including marginalised sectors of society. The 2016 edition expands coverage to 113 countries and jurisdictions 
(from 102 in 2015), relying on more than 110,000 household and expert surveys. Performance is measured using 44 
indicators across eight primary rule of law factors, each of which is scored and ranked globally and against regional and 
income peers: constraints on government powers, absence of corruption, open government, fundamental rights, order and 
security, regulatory enforcement, civil justice, and criminal justice. 
The WGI project reports aggregate and individual governance indicators for 215 economies over the period 1996–2013, for 
six dimensions of governance: Voice and Accountability; Political Stability and Absence of Violence; Government 
Effectiveness; Regulatory Quality; Rule of Law; Control of Corruption. These aggregate indicators combine the views of 
enterprise, citizen and expert survey respondents in industrial and developing countries.  They are based on 32 individual 
data sources produced by a variety of survey institutes, think tanks, non-governmental organisations, international 
organisations, and private sector firms. 

Topic 1.3: Continuous improvement & innovation 

 
 
Quality of Public Administration – A Toolbox for Practitioners 

99 

Some indices use the difference in position between individual countries and ‘best in class’ to show 
the  distance  that  the  country  still  needs  to  cover.  This  concept  of  a  journey  is  helpful,  as  long  as 
contextual  factors  are  also  taken  into  account,  because  it  focuses  on  the  absolute  improvements 
that are needed in the administration under consideration, and in which areas. 

As  a  feedback  mechanism,  monitoring  is  invaluable  in  steering  policies  and  programmes  towards 
success, but sometimes more in-depth explanations are needed for what is happening and why. For 
more  fundamental  reviews  of  plans  and  performance,  one  step  removed  from  implementation, 
public administrations should engage in evaluation, drawing on monitoring data where it is available, 
and  conducting  original  research  (interviews  and  surveys)  where  it  is  not.  Both  evaluation  and 
performance audit have efficiency and effectiveness as core concepts, but also take account of the 
sustainability of policy outcomes. 

Ratio 
Efficiency 

Effectiveness 

Key questions 
How well have the various inputs and resources (funds, expertise, time, etc.) been converted 
through  activities  into  expected  outputs?  Could  the  same  outputs  have  been  achieved  at 
lower cost, or better outputs for the same cost? 
To what extent have the objectives (specific / operational and strategic) been achieved or are 
expected to be achieved? Have the interventions and instruments used produced the desired 
outcomes  (results  and  impact),  or  could  better  effects  be  obtained  by  using  different 
instruments? 

Sustainability  Are  the  policy  outcomes  (including  institutional  changes)  durable  over  time?  Will  they 

continue if there is no more public funding? How resilient are they? 

The relationship between inputs, outputs, results and ultimate impact is illustrated below: 

Evaluations  and  performance  audits  should  also  consider  causality  and  the  magnitude  of  effects. 
This  is  about  assessing  the  extent  to  which  policy  interventions  create  the  expected  effects,  or 
whether  there  are  other  exogenous  factors  which  influence  outcomes  and  led  to  unintended 
consequences.  This has two components: contribution and attribution. 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

100 

Factor 
Contribution 

Key questions 
Is the intervention in fact one of the causes 
of observed change? 

Attribution  What  proportion  of  the  observed  change 
can  really  be  attributed  to  the  evaluated 
intervention?  

Analysis 

intervention  among  the 

Rank  the  assessed 
various causes explaining the observed change 
Build  a counterfactual  scenario:  what  would 
have happened without the intervention? 

For  an  inspiring  example  of  using  counterfactuals,  please  see  employment  programme  innovation 
practised in Denmark using randomised controlled trial experiments (topic 1.3.3). 

is  no  merit 

Step  5  of  the  process 
is  action. 
in 
Ultimately,  there 
monitoring,  evaluation  or  audit 
unless  it  affects  performance.  If  there  is  deviation  from  the  plan,  which  might  be  positive  or 
negative,  the  point  is  to  understand  why  and  to  make  adjustments  (or  not)  in  either  the  policy, 
programme, project / service or indeed the plan itself, to achieve the objective. 

This  starts  with  reporting:  the  format  in  which  performance  information  is  presented  should  be 
appropriate to the target audience, which includes management, but often external audiences too, 
including politicians and the public. This may require different styles and levels of detail. 

Evaluations often have limited impact on policy-making, either because: their timing is out of step 
with the policy process (the findings arrive too late to influence policy design); there are conflicts of 
interest  (the  same  staff  that  designed  the  policy  also  supervise  or  conduct  its  evaluation);  the 
recipients are not interested or incentivised to learn lessons from the past; or they are simply not 
commissioned  in  the  first  place.  The  independence  of  performance  audits  is  one  of  their  key 
strengths,  although  this  can  also  be  a  weakness  if  their  timing  is  uncoordinated  with  the  policy 
process,  in  which  case  the  public  administration’s  main  concern  can  be  defending  past  decisions, 
rather  than  focusing  on  the  future.  The  following  table  sets  out  some  potential  answers  to  these 
weaknesses. 

Challenge 
Evaluation are not a 
systematic part of 
the policy process 

Evaluation is not 
sufficiently impartial 

Evaluation findings 
are ignored 

Solution 

Introduce a law or code of conduct which commits the administration to evaluate 
policies and programmes, subject to the expected benefits exceeding the costs; 

  Publish an annual evaluation plan, which sets out the priorities for evaluations 

over the coming year with a clear timetable.  

  Assign responsibility for evaluation to a unit which is separate from the ministry or 

department which is responsible for the policy; 
  Request the SAI to audit the evaluation process. 
  Pre-commit through a law or code of conduct to publish all evaluation findings on 

the government’s website; 
Introduce ‘real-time’ evaluations, which are conducted in parallel with 
implementation, and hence have a greater chance of influencing ongoing policy 
development. 

The evaluation process itself can be used to take forward organisational learning, by involving the 
public  administration  in  its  preparation  and  implementation,  not  just  as  the  recipient  of  the 
report(s). This is where it is very important to build capacity within the administration to plan and 
oversee evaluations, and use the findings (see also topic 8.3.3 on working with consultants). 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

101 

In  addition  to  formal  evaluations,  peer  reviews  (including  ‘gateway  reviews’67)  can  be  valuable  in 
drawing on the knowledge of expert practitioners - independent from the process - in short, focused 
inputs  to  strengthen  policy  design  and  implementation.  Another  alternative  to  evaluation  being 
organised  solely  by  governments  is  co-evaluation:  the  active  involvement  of  stakeholders  in 
evaluating public policy and programmes, as exemplified by Italy’s pilot ‘Civil Evaluation’ (see topic 
5.5), which has the innovative aim of promoting wide collaboration between public administrations 
and  citizens  (users)  in  assessing  public  services.  The  principles  underlying  this  case  study  are 
explored further in the context of customer satisfaction of service delivery.  

1.3.2  Encouraging external scrutiny 

Monitoring, evaluation and audit 

The  transparency  of  government  helps  to  stimulate 
policy  development  in  public  administrations,  much  in 
the  same  way  that  competition  entices  enterprises  to 
find  better  ways  to  satisfy  customers’  needs,  through 
external  pressure.  Governments  at  all  levels  are  held  to 
account  by  parliaments  and  assemblies,  aided  by  SAIs, 
independent  regulatory  bodies,  and  Ombudsmen  that 
conventionally  report  directly  to  them.  They  channel  the  views  of  the  electorate  and  ensure  that 
their expectations have an outlet.  

Encouraging external scrutiny 

Fostering innovation 

Other  institutions  outside  of  the  public  sector  also  play  essential  roles.  An  independent  and 
investigative  media  may  not  always  be  welcomed  by  governments,  but  it  provides  a window  into 
the  workings  of  public  administrations  and  a  source  of  scrutiny  that  drives  up  the  standards  of 
government  and  is  especially  valuable  in  putting  ethics  and  integrity  in  the  spotlight.  Through 
discourse  and  dissent,  the  media  provides  a  ‘safety  valve’  that  is  vital  for  political  stability  and 
economic prosperity.  

Similarly,  the  ‘third  sector’  of  civil  society  organisations  (CSOs)  provides  a  voice  to  local 
communities and interest groups, with a combination of campaigning energy and expertise, often in 
specific policy domains, such as environment, enterprise, etc. Civil society is highly heterogeneous, 
covering  everything  from  humanitarian  aid  to  lobbyists  and  think  tanks,  business  associations  to 
trade unions to educational bodies, and active in all the fields in which both the public and private 
sectors  are  also  present.  What  CSOs  have  in  common  is  two  ‘negatives’:  not  being  part  of  the 
government and not distributing profit to their members. However, while any surpluses are retained 
to  reinvest  in  activities,  the  reality  is  that  CSOs  frequently  experience  financial  insecurity,  being 
dependent on donations and project funding. As few CSOs have the scale to operate at national or 
international levels, most tend to remain small and localised, leaving the sector fragmented, fragile 
and constantly facing an uncertain future. This is where public administrations can intervene to good 
effect,  while  preserving  the  CSO’s  independence.  While  many  CSOs  rely  on  ESIF  funding  for  their 
project-based finance (see theme 8), there is also the option of core funding from national budgets 
voted by parliaments.  

67 Conducted at key stages in the implementation of a programme or project.  

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

102 

The ongoing focus on finding funds distracts many CSOs from pursuing their primary objectives, but 
weaknesses  in  governance  structures  and  coordination  can  also  play  their  part.  The  sector  is 
sometimes characterised more by competition over scarce resources than cooperation over a shared 
vision,  making  the  CSO  community  reactive  to  the  public  administration’s  agenda,  rather  than 
anticipating  and  advocating  change.  Inadequate  networking  among  CSOs  can  undermine  their 
effectiveness and miss the opportunity to engage better with businesses and citizens over common 
causes. Some public administrations have reached out to representatives of the CSO community, to 
better  understand  their  development  needs  and  to  formalise  their  advocacy  role  in  an  advisory 
capacity with standing committees, such as Croatia’s Council for Civil Society Development (CCSD), 
whose members are elected by NGOs.  

Inspiring example: Fostering structured civil dialogue (Croatia) 

Developed  as  the  most  important  institutional  mechanism  for  civil  dialogue,  the  Council  for  Civil  Society 
Development (CCSD) aims at involving wider civil society in the shaping of public policies in Croatia. Given the 
difficulties  most  governments  face  when  trying  to  set  criteria  for  identifying  and  nominating  civil  society 
representatives  (CSRs),  the  procedure  for  the  election  of  CSRs  is  an  example  of  good  practice,  with  a  great 
potential for replication in other countries. 

The  CCSD  was  established  in  2002  as  an  advisory  body,  providing  a  forum  for  direct  and  formal  dialogue 
between government bodies and civil society. It is composed of 31 members:  

  15 representatives of government bodies;  
  16  representatives  of  civil  society  organisations,  namely:  13  representatives  of  NGOs/citizens’ 
associations,  elected  by  NGOs  themselves  through  public  elections;  one  representative  of  trade 
unions, nominated by the coordination of trade union federations; one representative of employers’ 
associations, nominated by the Croatian Union of Employers; and one representative of foundations, 
nominated by Croatian Network of Foundations;  

The  CCSD  is  based  on  the  notion  of  openness  and  inclusion  as  drivers  to  a  more  efficient  and  transparent 
public administration. The idea is to achieve structured co-operation between public administration and civil 
society, as well as creating the conditions for sustainable development. In line with  the actual problems, i.e. 
the lack of a structured dialogue and the social doubts about the transparency within public administrations, 
the  introduction  of  the  CCSD  brought  a  new  dimension  to  the  system.  The  Council  enables  regular  and 
valuable  exchange  of  opinions,  know-how  and  experience  between  the  representatives  of  different  sectors, 
contributing also to building mutual trust and understanding. 

Members  of  the  Council  representing  NGOs  (as  the  most  numerous  and  diverse  actors  of  civil  society)  are 
elected  by  NGOs  themselves  through  a  transparent  and  democratic  procedure,  based  on  a  public  call  for 
nominations  and  public  call  for  voting  for  eligible  candidates.  The  two-stage  procedure  of  electing  Council 
members  begins  with  a  public  call  for  nominations  widely  disseminated  and  published  in  all  media.  NGOs 
nominate candidates solely for the area in which they operate, taking into account that every NGO or a formal 
network or association of NGOs can nominate only one candidate for member and substitute member of the 
Council.  Nominations  are  sent  by  post  using  a  standard  nomination  form  with  a  set  of  required  supporting 
documents.  The  expert  commission checks the eligibility  of candidates and establishes the list  of candidates 
with valid nominations, which is published on the website of the Office for Cooperation with NGOs. After that, 
a  public  call  for  voting  for  eligible  candidates  is  published  on  the  Office  website  and  NGOs  vote  for  eligible 
candidates  submitting  a  standard  voting  form  available  on  the  Internet  by  regular  post,  respecting  the 
principle that one organisation can vote only once. Finally, the expert commission (nominated by the previous 
Council) proceeds to counting votes, and those candidates with the largest number of votes are proposed to 
be appointed members and substitute members of the Council by the Act of the Government of the Republic 
of  Croatia.  For  the  purposes  of  ensuring  transparency,  detailed  information  on  all  valid  and  invalid  votes  by 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

103 

candidates is available on the Internet. In addition, all organisations can access all supporting documentation 
and check the validity of votes sent by post, which contributes to the transparency of the process. 

At the 2010 CCSD elections, 787 NGOs sent their voting ballots for candidates for the council members (and 
substitute members) in different sectors, showing a high level of concern and interest in who represents the 
civil society and what kind of expertise will be added to this advisory body of the Croatian Government.  

The CCSD’s work is well complemented by the activities of the Government Office for Cooperation with NGOs 
(founded  in  1998)  and  the  National  Foundation  for  Civil  Society  Development  (established  in  2003).  These 
three  institutions  form  a  three-pillar  framework  for  creating  a  more  enabling  environment  for  civil  society 
development in Croatia. The role and importance of the CCSD has been continuously growing over the past 12 
years. Its contribution has been additionally recognised with regards to ensuring participatory programming of 
EU  funds,  improving  standards  of  public  funding  for  CSO  programmes,  as  well  as  developing  the  normative 
framework for public consultations on new draft laws, other regulations and acts.  The widespread acceptance 
and  positive  reviews  of  its  work  show  the  potential  for  the  CCSD  to  contribute  towards  more  open, 
participatory and collaborative approaches to public policy-making across the Croatian public sector. 

For further information: Igor Vidačak, Head of the Office, info@uzuvrh.hr  

1.3.3  Fostering innovation 

Encouraging external scrutiny 

Monitoring, evaluation and audit 

Innovation is central to achieving the goals of Europe 2020 
-  and  public  administrations  are  pivotal  to  stimulating 
innovation.  The  public  sector  is  directly  responsible  for 
around  one-third  of  research  and  development  (R&D)  in 
Europe,  as  well  as  ensuring  and  enforcing  the  right 
through 
regulatory  environment 
for  private  R&D, 
patenting  and  protecting 
intellectual  property  rights. 
Through  public  funding,  governments  have  driven  the  development  and  demonstration  of  key 
technologies,  such  as  renewable  energy  sources,  biotechnology  and  nanotechnology,  at  the  early 
stages, when the returns are too long-term and the risks too high for private investors. Through the 
state’s role in funding healthcare, education, defence and infrastructure, public procurement offers 
industries,  such  as 
considerable 
pharmaceuticals, ICT, aerospace, transport, energy and environment (see also topic 8.2.4). The role 
of innovation within public administrations is equally important in improving services, strengthening 
productivity, and bringing new thinking to old problems.  

leverage  over  private 

Fostering innovation 

technology 

innovation 

in  high 

Making the case for public sector innovation 

In addition to the public sector's role in catalysing innovation in the wider economy, there is an urgent need to 
power  innovation  within  the  public  sector  itself  to  unlock  radical  productivity  improvements  and  efficiency 
gains, to foster the creation of more public value and a better response to societal challenges. Innovation in 
the public sector … can be defined as the process of generating new ideas and implementing them to create 
value for society, covering new or improved processes (internal focus) and services (external focus). It takes on 
a variety of forms, ranging from smarter procurement, mobilising new forms of innovation financing, creating 
digital  platforms  and  citizen-centric  services,  as  well  as  driving  a  new  entrepreneurial  culture  among  public 
managers. 

Source: European Commission, Directorate-General for Research and Innovation (2013), Powering European 
Public Sector Innovation, Towards A New Architecture, Report of the Expert Group on Public Sector Innovation. 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

104 

Statistics indicate that the demand for public services in many advanced countries is growing faster 
than the rest of the economy - a trend started before the onset of the recent global financial and 
economic  crisis,  and  the  resultant  fiscal  deficits.68  To  reduce  indebtedness,  the  public  sector  is 
subject  to  major  budgetary  constraints,  but  the  expectations  of  public  services  have  never  been 
higher. Innovation is vital for increasing public sector efficiency (value for money, better for less) and 
for delivering new and better-quality services. In fact, innovation in public sector organisations can 
be defined even more broadly, as comprising seven dimensions:69  

Type 
Product 
Process 

Service 

Position 

Strategic 

Governance 

Rhetorical 

Clarification 
Developing new or enhanced products, such as electronic ID cards, better laws and regulations. 
Re-designing organisational processes to improve their performance and efficiency, such as 
lean production, reorganisation of back-office processes, etc. (see theme 5) 
Discovering new ways to provide public services to citizens and businesses, such as through 
smartphones, social media, co-delivery, etc. (see theme 5 and theme 6) 
Identifying new contexts or ‘customers’ for public services, and increasing the tailoring and 
targeting towards specific groups and individuals, such as offering personalised online services 
through MyPage, or repositioning the relationship between government and immigrants 
Defining new goals or purposes for the organisation, such as the role of public sector in the 
sustainability and social responsibility debate. 
Finding new forms of citizen engagement and democratic institutions, such as area forums, e-
Participation, devolved administration, etc. 
Introducing new language and concepts into public administration, such as the concept of 
‘congestion charging’ in city centres, or ‘nudging’ to influence the behaviour of citizens and 
businesses to achieve policy goals. 

The stimulus for innovation can come from different sources, and can be policy-driven, organisation-
driven,  professional-driven  or  user-driven.  Unfortunately,  evidence  suggests  that  public  sector 
innovation today mostly happens through ad hoc and uncoordinated initiatives, rather than due to 
deliberate  and  systematic  efforts.70  The  European  Commission’s  Expert  Group  on  Public  Sector 
Innovation  has  identified  internal  barriers  which  hold  back  public  administrations  from  becoming 
more innovative in four broad categories:71 

68 See: G. Mulgan (2007), Ready or not? Taking innovation in the public sector seriously, NESTA; and S. Kelman (2008), The 
‘Kennedy School’ of research on innovation in government, pp28-51 in S. Borins (ed.) (2008) ‘Innovations in government: 
research, recognition and replication’, Brookings Institution 
69 See also J. Hartley (2005), Innovation in governance and public services:  past and present, Public Money and 
Management, 25:1. 
70  European  Commission,  Directorate-General  for  Research  and  Innovation  (2013),  Powering  European  Public  Sector 
Innovation, Towards A New Architecture, Report of the Expert Group on Public Sector Innovation, p.5 
71 Op cit. 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

105 

Obstacles to innovation 
Weak enabling factors 
or unfavourable 
framework conditions 

Lack of innovation 
leadership at all levels 

Limited knowledge and 
application of 
innovative processes 
and methods 
Insufficiently precise 
and systematic use of 
measurement and data 

Challenges 
Public sector organisations face structural obstacles (scattered competences, 
ineffective governance mechanisms, and diverse legal and administrative cultures), 
resource constraints to develop and deploy staff and to finance roll-out, and 
inadequate coordination within and across organisations to share, spread and 
scale-up successful initiatives.  
Ultimately, innovation is down to individuals. Public administrations need leaders 
that can envisage and manage change at all levels: politicians, top civil servants, 
mid-level managers, policy advisers, front-line staff, etc. However, the public sector 
can be slow to recognise and reward innovation, tending to prefer caution 
(avoiding failure) to creativity (finding new paths to success). Rigid rules and risk-
averse managers can discourage staff and stifle the diffusion of innovative ideas. 
To move from theory to practice, innovation needs access to capabilities (systems, 
skills, tools and methods), which is often absent, and collaboration (with other 
parts and levels of government, businesses, citizens and third sector organisations) 
which needs to be nurtured.  
Public administrations are constrained in pursuing innovation by inadequate 
information on sources of new and improved products, processes and services, and 
monitoring data to sell the benefits for policy outcomes. 

By its nature, there is no blueprint for innovation: do A, get B. Tomorrow’s innovations are unknown, 
otherwise  they  would  be  today’s.  The  main  challenge  for  the  public  sector  is  to  engineer  the 
conditions and the climate for creativity to flourish. This is about organisational culture and can only 
happen  with  the  acquiescence  of  the  top  leadership  (politicians  and  senior  management), 
communicated  to  all  levels  and  units,  and  backed-up  by  systems  and  day-to-day  experiences. 
Innovation can be embedded into institutions, if employees are encouraged and enabled to act as 
public sector ‘entrepreneurs’.  

Entrenching innovation in organisational values 

An organisational culture that supports innovative working encourages risk taking and the exchange of ideas; 
promotes  participation  in  decision-making;  has  clear  goals  and  rewards  for  innovation;  and  provides 
psychological  safety in relation to idea  generation.  The evidence  shows there to be clear sector differences. 
Further, innovation behaviours, such as challenging current thinking and non-conformity, in one organisation, 
may manifest differently in another.  

Our survey results confirm that organisations that actively promote and reward innovation are most effective 
at bringing about  innovation. For  example, providing ‘individual and team incentives or reward programmes 
that encourage innovation’ and having ‘work time devoted to developing new ideas’ were listed as among the 
most effective initiatives for facilitating innovation. Evidence from our interviewees also reflects the literature 
findings.  To  flourish,  innovation  must  be  entrenched  as  one  of  the  core  values  of  the  organisation  and  the 
organisational objectives must be visibly aligned with those values.  

“Culture and history are the main catalysts. The view that innovation is the right way forward for the business 
occurs from the top down and it has been engrained in the company since its formation, over 170 years ago. It 
is absolutely true that innovation is the lifeblood of the organisation” (Mike Addison, Open Innovator, Procter 
&  Gamble).  “P&G  now  also  augments  its  internal  innovation  by  constantly  striving  to  create  and  nurture  an 
organisational culture that is always looking externally for solutions and is proudly championing the adoption 
of ideas found elsewhere. We believe that the customer is the boss, and we constantly strive to make products 
more relevant – using the best ideas, regardless of their origin to achieve this”. 

Different  challenges  in  developing  a  culture  to  support  innovation  are  observed  across  sectors  and 
organisations.  For  example,  one  interviewee  in  local  government  suggested  that  middle  to  senior  managers 
are the key to thinking creatively and having the confidence to move forward with new ideas. “The culture is 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
  
Quality of Public Administration – A Toolbox for Practitioners 

106 

risk adverse in the organisation, the community and the press. Many councils may have a political dogma with 
[political] members in place for many years – their response is often, ‘we’ve been so successful in the past why 
change?’ Innovation to them sounds risky. Changing culture is about influencing member’s innovativeness as 
well  as  the  employees  of  the  council.”  (Martin  Collett,  Head  of  Organisational  Development,  Tameside 
Metropolitan Borough Council).  

Source: Extracted from NESTA (2009), “Everyday innovation: How to enhance innovative working in employees 
and organisations”, http://www.nesta.org.uk/publications/everyday-innovation  

The aim should be an atmosphere in which it is accepted - and expected - that public servants can 
think laterally and radically about problems and policy solutions. They should be able to put forward 
their ideas internally, without inhibition. The policy-making process still needs to be evidence-based 
and rigorous, with robust options appraisal (see topic 1.1). But staff should be actively encouraged 
to  challenge  conventional  wisdoms,  question  the  assumptions  that  underlie  how  things  are  done 
currently, and  not  accept existing  paradigms  at  face  value.  A  good  place  to  start  is objectives  and 
outcomes: what does the beneficiary want and how can we help them to achieve it? 

In a sense, this makes the creative process a form of ongoing ‘internal consultation’. When public 
bodies plan to go outside to consult citizens, businesses and other organisations on their policies and 
programmes, they have unlimited expectations of the reaction they might receive. Some of the ideas 
from beneficiaries might be very insightful and open whole new avenues of thinking to be explored, 
for  example,  while  others  might  add  little  value  and  can  be  discarded  quickly.  For  external 
consultations,  public  administrations  can  frame  the  discussion  with  their  choice  of  questions,  but 
they cannot control the feedback, only their response to it, irrespective of whether the format is a 
survey, a focus group, an SME panel, a public hearing, or a suggestions box (see theme 5). All invited 
views are valid, but must be screened against the criteria of what is achievable and desirable, and a 
decision taken on next steps. The  same  principle  should apply  to ‘internal consultations’, but on a 
continuous basis. This does not mean that permanent revolution serves the public interest: it takes 
time  for adopted  innovations to be  implemented and intended beneficiaries to adjust  to them,  to 
give change a chance.  

Public  administrations  can  create  the  mechanisms  for  ‘innovation  through  internal  consultation’ 
through systems for both structures and staffing: 

Mechanism 
Structures 

Challenge 

  Nominate ‘innovation champions’ or ‘innovation coordinators’ across the administration to 
campaign for innovation, to encourage fresh ideas and to spot new practices (products, 
processes, services, etc.) that can be disseminated across the organisation and into other 
public bodies (see theme 3). 

Staffing 

  Encourage and incentivise innovation by targeting creative talent through recruitment, 

staff development, performance appraisal and bonus payments, and integrating innovation 
as a competence into profiles and frameworks (see theme 4).  

Some  Member  States  have  sought  to  institutionalise  innovation  by  creating  dedicated  units  like 
Denmark’s MindLab  (see  topic  1.1).  These  units are assigned  a  specific  agenda  to think  creatively, 
and to act as catalysers of innovation within the administration, such as the example of ‘Know Share’ 
(re-branded as 'REGIO Communities of Practitioners’), a co-design initiative of the Commission JRC’s 
EU Policy Lab and DG REGIO. 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

107 

REGIO ‘Know Share’ (re-branded as 'REGIO Communities of Practitioners’): co-designing a bottom-
up community to improve ERDF and Cohesion Fund management 

In  2015,  the  European  Commission’s  Directorate-General  for  Regional  and  Urban  Policy  (DG  REGIO)  and  the 
JRC’s EU Policy Lab72 launched a joint "design for policy" project engaging with managing authorities (MAs) and 
intermediate  bodies  (IBs)  to  improve  the  management  of  the  European  Regional  Development  Fund  (ERDF) 
and  the  Cohesion  Fund  (CF).73  Across  the  Member  States,  there  are  estimated  to  be  around  24,000  public 
officials  working  in  the  MAs  and  IBs  at  national  and  regional  levels  on  ERDF  and  the  CF,  which  is  a  huge 
resource  of  know-how.  The  question  was:  how  best  to  foster  exchange  of  this  collective  knowledge  among 
MAs to improve management of the funds and achieve better results?   

As a first step, the EU Policy Lab organised a series of lab session with DG REGIO to unpack the problem and 
co-design a process aimed at creating a user-led ‘platform’ to forge new relationships between DG REGIO and 
the  MAs  and  IBs,  based  on  collaboration  and  shared  responsibility.  This  resulted  in  a  three-phase  process 
based on a participatory design approach:  

Phase 

Co-design 

1 

2 

3 

Analysis of the problem, co-design and 
prototyping of solutions with MAs/IBs 

Functional prototyping and testing 
Further development, upscaling and 
mainstreaming 

Community-building 
Planting the seeds of a community, identifying shared issues 
and champions among the MAs and IBs willing to take 
responsibility for action 
Growing the community and supporting the champions 

Harvesting the results, making the process self-sustained.    

The  EU  Policy  Lab  lead  the  implementation  of  the  1st  phase  from  December  2015  to  March  2016,  with  DG 
REGIO  taking  forward  the  following  phases,  including  through  the  organisation  of  three  policy  laboratories 
during  the 2016 European Week of Regions and Cities.  

The methods applied are people-centred, visual, with strong user involvement to gain a better understanding 
of  contextual  factors,  and  find  solutions  through  iterations.  A  basic  principle  is  that  there  should  be  no 
preconceived idea about what the outcome should look like. 

During  phase  1,  a  group  of  around  30  individuals  from  MAs  and  IBs  in  16  Member  States  were  engaged 
through workshops and webinars, with some additional 20 persons that have followed the project. The initial 
group consisted of those invited to the kick-off workshop (including those that were not able to join), but has 
been  gradually  enlarged  through  peer-to-peer  contacts  within  and  between  organisations.  The  process  was 
structured  to  build  trust  among  peers  and  generate  collaboration  around  priority  issues  identified  by  the 
participants  themselves  (such  as  State  aid,  the  capacity  of  ERDF  beneficiaries,  monitoring  of  projects  and 
communication  of  their  results).  The  MAs  have  then  conducted  real  world  experiences  applying  innovative 
tools  to  address  concrete  issues.  Building  on  this,  they  have  co-designed  with  DG  REGIO  a  prototype  of  the 
sharing  and  collaboration  space  that  is  now  been  tested.  This  space  will  complement  and  amplify  existing 
support activities implemented by REGIO, notably the TAIEX REGIO Peer 2 Peer initiative (see topic 8.3). 

Read more about the project here and in this concept note that summarises the first phase of the project, its 
outcomes, and the recommendations for phases two and three.   

As the first step to facilitating exchanges of experience and linking innovation across different levels 
of  governance,  the  EU  Policy  Lab  has  conducted  a  mapping  of  public  policy  labs  across  the  EU’s 
Member  States,  and  organised  Lab  Connections  in  October  2016,  the  first  gathering  of  European 
policy  labs.  The  UK’s  Behavioural  Insights  Team  (or  ‘Nudge  Unit’  as  it  is  known)  is  another  well-
known  case  of  creating  a  unit  that  is  tasked  with  thinking  ‘outside  the  box’  -  in  this  case  on  how 

72 For an overview of the JRC and the EU Policy Lab activities, please see also topic 1.1.1  
73 See also topic 8.3 on managing European Structural and Investment Funds (ESIF). 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

108 

behavioural sciences can be employed to incentivise certain policy outcomes (see topic 1.2.2). The 
aim is to understand how individuals take decisions in practice and how they are likely to respond to 
options. These insights are then employed to design policies or interventions that can encourage and 
enable people to make better choices for themselves and society. 

Inspiring example: Behavioural Insights Team (United Kingdom) 

Governments have always used a wide range of tools to achieve policy objectives. ‘Traditional’ tools, including 
legislation,  regulation  or  fiscal  measures  (tax  and  spending)  have  been  used  throughout  history  to  provide 
incentives to people to behave in certain ways. Many of the most dramatic improvements in the quality of life 
of British citizens have resulted from the use of instruments of this kind. The background thinking here is that 
many of the most pressing policy issues faced today are equally influenced by how we, as individuals, behave. 
We can all cite instances in which we know we should act differently in our own self-interest or in the wider 
interest,  but  for  one  reason  or  another  do  not.  The  traditional  tools  of  Government  have  proven  to  be  less 
successful in addressing these behavioural problems. We need to.  

The  Behavioural  Insights  Team  (BIT),  often  known  as  the  ‘Nudge  Unit’,  has  been  established  to  think  about 
ways  of  supplementing  the  more  traditional  tools  of  Government  with  policy  that  helps  to  encourage 
behaviour  change  of  this  kind.  It  does  this  by  applying  insights  from  academic  research  in  behavioural 
economics, psychology, and social anthropology to public policy. It is a small team of 13, with backgrounds in 
academia  (behavioural  sciences  and  experimental  methodology),  policy-making  and  marketing.  Starting  life 
inside the UK Prime Minister’s Office, the BIT became incorporated as a social enterprise with three owners: 
the  employees,  the  UK  government,  and  NESTA  (the  UK's  leading  innovation  charity)  as  the  winners  of  a 
competitive process to become the team's joint venture partners. 

Behavioural insights interventions are usually simple, highly cost-effective, and often yield surprising results. 
For example: 

  Automatically  enrolling  individuals  on  to  pension  schemes  has  increased  saving  rates  for  those 

 

employed by large firms in the UK from 61 to 83%; 
Informing  people  who  failed  to  pay  their  tax  that  most  other  people  had  already  paid  increased 
payment rates by over 5 percentage points;   

  Encouraging jobseekers to actively commit to undertaking job search activities increased their chance 

of finding a new job;   

  Prompting  people  to  join  the  Organ  Donor  Register  using  reciprocity  messages  (‘if  you  needed  an 

organ, would you take one?’) adds 100,000 people to the register in one year. 

The Behavioural Insights Team has a methodological approach with four discrete steps.  

1.  Define your outcome 
2.  Understand the context 
3.  Build your behavioural insights 
4.  Test, learn, adapt 

It  begins  by  defining  the  outcome  that  you  want  to  see  –  whether  it  is  more  people  back  in  to  work;  more 
people  saving  for  a  pension;  or  fewer  people  failing  to  pay  their  tax  on  time.  The  next  step  draws  on 
ethnography  to  understand  better  how  individuals  experience  the  service  or  situation  in  question.  This 
understanding allows BIT to move to the next stage: building new interventions to improve outcomes. During 
this  third  phase,  BIT  draws  explicitly  on  its  own  MINDSPACE  and  EAST  frameworks,  as  well  as  relevant 
academic  studies.  Finally,  BIT  tests  and  trials  its  interventions,  often  using  randomised  controlled  trials  that 
enable BIT to demonstrate how effective the new intervention is relative to the old way of operating.  The BIT 
paper  ‘Test,  Learn,  Adapt’  sets  out  nine  steps  for  running  randomised  controlled  trials.  This  four-part 
methodology enables the user to identify what works and what can be scaled up, as well as what is less likely 
to be effective. 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

109 

The  BIT  continues  its  programme  of  work  with  No.10  and  the  Cabinet  Office  and  has  had  a  two-year 
programme of work with the Government of New South Wales, supporting them to put in place a behavioural 
insights team which  is testing interventions that have previously proved highly successful in the UK. The BIT 
has  been  working  with  Jobcentre  Plus,  the  Department  for  Energy  and  Climate  Change,  the  Department  for 
Business,  Innovation  and  Skills,  the  Metropolitan  Police,  and  numerous  other  organisations  in  the  UK  and 
overseas  to  develop  and  test  new  interventions  that  draw  on  insights  from  the  behavioural  sciences.  In 
addition  to  providing  workshops  for  central  Government  departments  and  local  authorities,  the  BIT  has 
supported  the  United  Nations  Development  Programme,  UNICEF,  the  World  Bank,  the  Government  of 
Singapore and numerous other organisations that have wanted to understand better how they can draw on 
insights from the behavioural sciences to help them deliver more efficient and effective services.  

Source: http://www.behaviouralinsights.co.uk/about-us.74 

The ‘Nudge Unit’ works in partnership with an array of bodies, which is essential to maximising the 
influence and impact. Apart from the obvious advantages of a concentration of expertise on tap, it is 
not  necessary  to  create  specialist  units  to  achieve  this  type  of  cooperation  and  co-creation.  The 
Netherlands’ Smarter Network is an alternative model which seeks to connect public servants with 
innovative ideas from across all levels and territories of the Dutch public administration. 

Inspiring example: The Smarter Network (The Netherlands) 

The main goal of ‘The Smarter Network’ (in Dutch, het slimmer network) is to connect networks of innovating 
professionals throughout the public sector, and to design support structures and learning programs that help 
them increase their innovative capacity. As a secondary goal, Slimmernetwerk has been used as an object of 
research  on  the  question:  to  what  extent  can  bottom-up  innovation  stimulate  the  innovative  capacity  of 
professionals and organisations, as a mean to improve public value creation? 

In practice, the network is organised through a set of activities and approaches: 

  An online information knowledge-hub that serves the community (www.slimmernetwerk.nl);  

  A  ‘doetank’:  an  iterative  innovation  approach,  consisting  of  several  fixed  steps  that  will  guide  small 
interdisciplinary groups of professionals through the process of  multi-stakeholder problem defining, 
through  in  situ  experimenting  and  reflection,  to  implementing.  Doetanks  are  assisted  by  coaches, 
gatekeepers (to safeguard progress) and supported by high commissioners from government, science 
and society; 

  Smarter Network Cafés (Slimmernetwerk Cafés): meetings aiming to support the community in terms 

of physical meeting, stand-up inspiration and the connection of Doetanks to networks; and  

  A  LinkedIn  discussion  group  named  ‘slimmer  werken  in  de  publieke  sector’  (smarter  working  in  the 

public sector) to share knowledge, thoughts and connect participants online. 

The 4-year programme underlying the network is in its final stage. It is highly expected that some organisations 
within government will adopt the network and some of its activities in some form.  

For further information: Adri Stet, Senior Policy Advisor, Ministry of the Interior and Kingdom Relations, 
adri.stet@minbzk.nl; 

In  the  context  of  good  fiscal  governance  (see  theme  8),  public  administrations  should  take  an 
informed  approach  to  risk-taking  with  innovation.  To  make  progress,  administrations  need  the 
freedom to exercise ‘trial and error’, which will inevitably bring both successes and failures. It also 

74 See also D. Halpern (2015), Inside the Nudge Unit, W H Allen. 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
                                                           
Quality of Public Administration – A Toolbox for Practitioners 

110 

encourages prototyping and controlled testing before proliferation, to manage the risk with small-
scale  and  iterative  experimentation.  One  example  is  the  use  of  randomised  controlled  trials  in 
Denmark’s employment service, which has been practised since 2005.  

Inspiring example: Employment programme innovation via randomised controlled trials (Denmark) 

To  ensure  evidence  based  policy  development,  and  to  underpin  implementation  of  effective  measures,  the 
Danish  Agency  for  Labour  Market  and  Recruitment  develops  randomised  controlled  trials  (RCTs).  In  these 
policy  experiments,  two  comparable  groups  are  randomly  selected.  One  group  receives  the  new  treatment, 
the control group gets the normal treatment.  

A  number  of  municipal  jobcentres  and  unemployment  insurance  funds  are  involved  and  supported  in  these 
trials. Input and outcome are measured, as well as the costs and benefits. The impact of the new treatment is 
evaluated by external evaluators with qualitative and quantitative methods, providing the answer to whether 
programme  is  working  as  expected  and  is  effective  in  bringing  unemployed  people  into  ordinary  jobs  or 
education, and whether it is cost-effective. 

The  results  from  RCTs  are  included  in  the  evidence  base  that  the  Danish  Agency  for  Labour  Market  and 
Recruitment  is building.  The  Danish Agency  for Labour Market and Recruitment  has published the results of 
this work on the web-based knowledge bank  “jobeffekter.dk”. The evidence base helps support  the political 
decision-making process. 

The RCT “Hurtigt I gang” (fast moving into jobs) is an example of a treatment of frequent client meetings with 
the  jobcentre  (every  second  week)  that  proved  very  effective.  It  reduces  the  length  of  the  first  spell  of 
unemployment  compared to  ordinary client  meetings held every three  months. Another example is the RCT 
“mentor  til  udsatte  unge  uden  uddannelse”  (mentoring  vulnerable  young  benefit  recipients  without 
education). A mentor increases the share of young people entering education or getting a job compared not 
getting a mentor. 

The Danish Agency for Labour Market and Recruitment is increasingly focusing on building up evidence about 
the  use  of  insights  from  behavioural  economics  and  ‘nudging’  in  labour  market  policy.  The  new  RCT 
“Tjeklisteforsøget”  (the  checklist)  is  an  experiment  whereby  unemployed  people  receive  an  online  check-list 
with good advice to go into jobs, the intention being to help them become more effective job searchers. The 
results from this RCT will be known by the end of 2017. 

For further information:  Simon Lamech, Head of Section, Danish National Labour Market Authority, sil@star.dk 

Such exercises need to be financed by public funds. The ideal scenario is that policy experimentation 
is  integrated  into  budget  preparation,  so  that  each  ministry  or  municipality  has  dedicated  monies 
assigned to policy R&D. This set-aside should be viewed as public sector ‘venture capital’, higher risk 
than  mainstream  public  spending,  but  with  the  opportunity  for  longer-term  returns  from  better 
policy outcomes to justify the investment. 

An indirect but cost-effective alternative to public expenditure is to build the incentive to innovate 
into the regulatory framework, by establishing a ‘right to challenge’ principle, which exempts public 
authorities,  businesses  and/or  third  sector  organisations  from  the effects  of  legislation  if  they  can 
demonstrate  they  can  achieve  the  policy  objective  more  effectively  or  efficiently  with  their  own 
innovation (links to topic 1.2). 

The dissemination of good practice relies on high calibre intelligence on what works in public sector 
including  the  European 
innovation. 

In  recent  years,  several  pan-European  organisations, 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
 
   
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

111 

Commission, EIPA and OECD, have established networks, awards or best practice websites to collect 
and diffuse innovative practices. 

Benchmarking and learning from good practice 

Benchmarking and learning from good practice are widely recognised as having a positive impact on peers. In 
particular, good practice serves as a source of inspiration for decision makers in the public sector for their own 
plans  to  innovate.  Good  practice  awards  have  undoubtedly  great  potential  to  drive  innovation  through  the 
recognition  of  achievements  and  the  fact  that  they  provide  role  models  for  imitation  and  replication.  The 
ultimate  goal  must  be  to  become  part  of  a  continuous  learning  process  within  and  outside  public 
administration.  This  implies  inspiring  and  empowering  people  to  take  part  in  that  process.  It  further  implies 
sharing  and  transferring  knowledge  and  know-how  thereby  influencing  the  learning  curve  of  a  given 
organisation saving time and money. 

The  oldest  source  is  the  European  Public  Administration  Network  (EUPAN),  an  informal  network  of  the 
Directors-General  responsible  for  Public  Administration  in  the  Member  States  of  the  European  Union,  the 
European  Commission  and  observer  countries.  The  informal  structure  of  the  Network  is  steered  by  the 
Ministers responsible for Public Administration. It is EUPAN’s mission to improve the performance and quality 
of European public administrations by developing new tools and methods in the field of public administration, 
based  on  the  exchange  of  views,  experiences  and  good  practices  among  EU  Member  States,  the  European 
Commission, observer countries and other organisations.  

The  OECD’s  Observatory  of  Public  Sector  Innovation  (OPSI)  collects  and  analyses  examples  and  shared 
experiences of public sector innovation to provide practical advice to countries on how to make innovations 
work. It provides a place for sharing, discussing and co-creating solutions that work. The OPSI’s online platform 
is  a  place  where  users  interested  in  public  sector  innovation  can:  access  information  on  innovations;  share 
their  own  experiences;  and  collaborate  with  other  users.  The  Observatory  is  led  by  a  Task  Force  of  OECD 
countries, chaired by Canada and France. OPSI benefits from a wide range of experience and research that is 
relevant to public sector innovation but exists beyond national governments. For instance, highly distinguished 
experts on public sector innovation from academia and research organisations provide analytical advice to the 
project. 

During  almost  a  decade  (2001-2010),  the  European  eGovernment  Awards  aimed  to  promote  technology-
enabled  innovation  in  government  and  the  health  sector.  The  purpose  of  these  Awards  was  to  encourage 
community-building  and  knowledge  exchange  on  eGovernment  across  the  EU.  A  strong  policy  link  was 
established to the Ministerial Declarations and the European Commission Action Plans on eGovernment under 
the umbrella of the Lisbon Strategy. The projects and lessons learned were utilised in a number of EU-funded 
studies and the ePractice portal. 

Another example is the European Public Sector Award (EPSA) launched in 2007. Since 2009 this biannual award 
scheme  is  being  organised  and  managed  by  the  European  Institute  of  Public  Administration  (EIPA)  with  the 
support  of  several  EU  Member  States,  the  European  Commission,  the  City  of  Maastricht  and  the  Dutch 
Province of Limburg. EPSA aims to support national modernisation processes by awarding projects which have 
proven their success by tangible results and impact. EPSA wants to make this experience transparent, available 
and  usable.  It  targets  all  public  administrations  at  the  different  levels  of  government,  with  an  emphasis  on 
specific  themes.  EPSA  focuses  on  recognition  and  dissemination  of  good  practices  by  means  of  the  awards, 
knowledge-transfer activities and publications.  As a  result of the three schemes run by EIPA, more than 800 
fully  structured  and  thoroughly  assessed  cases  from  36  European  countries  and  EU  Institutions  are  now 
available in its database. The EPSA 2013 edition under the theme 'Weathering the Storm  – Creative Solutions 
in a Time of Crisis' received 230 entries from 26 European countries and three European Institutions. 

In  2012,  the  European  Commission  launched  the  Prize  for  Innovation  in  Public  Administrations  (PIPA).  PIPA 
aims  to  highlight  excellence  and  vision  in  public  administrations.  The  prize  gives  visibility  to  the  most 
innovative  public  administrations  and  their  initiatives,  which  should  serve  as  best  practices  to  inspire  other 
public administrations in Europe to innovate. Furthermore, PIPA intends to challenge the negative stereotypes 
of public administration in the perception of the people. PIPA celebrates the most innovative, public initiatives 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

112 

which  benefit  citizens,  firms,  or  the  education  and  research  sector.  In  2013  there  were  203  entries  to  the 
competition  coming  from  all  corners  of  the  European  Union.  The  nine  winners  were  selected  by  an 
independent  jury  on  the  basis  of  four  criteria:  the  economic  impact  of  their  initiative;  its  relevance  to 
challenges  facing  society; how original and easy to replicate the idea  is; and how they plan to use the prize 
money (€100,000 each to scale up). Winning initiatives included integrated healthcare information accessible 
on a phone, a web-based platform for funding opportunities for firms, and a nationwide plagiarism detection 
system for higher education institutions. 

Source: European Commission, Directorate-General for Research and Innovation (2013), Powering European 
Public Sector Innovation, Towards A New Architecture, Report of the Expert Group on Public Sector Innovation. 

More  recently,  the  European  Commission  launched  the  European  Public  Sector  Innovation 
Scoreboard (EPSIS) with a view to improving our ability to benchmark the innovation performance of 
the public sector in Europe. 

European Public Sector Innovation Scoreboard 

Following  the  Europe  2020  flagship  initiative  ‘Innovation  Union’,  the  European  Commission  launched  a  pilot 
European Public Sector Innovation Scoreboard (EPSIS) with a view to improving our ability to benchmark the 
innovation performance of the public sector in Europe. The ultimate ambition is to capture and present public 
sector innovation in a  similar way to countries’ innovation performance in the  Innovation Union Scoreboard 
(IUS) and thereby encourage and facilitate innovation activity across the public sector. The 2013 pilot EPSIS is 
the  first  EU-wide  attempt  to  better  understand  and  to  analyse  innovation  in  the  public  sector.  It  was 
developed based on the experience of earlier national and regional projects, tested widely and discussed with 
a number of key relevant experts. The work will continue. 

For EPSIS, the definition of public sector innovation follows that used in the Innobarometer 2010 (EC, 2010): 
An  innovation  is  a  new  or  significantly  improved  service,  communication  method,  process  or  organisational 
method.  Based  on  available  data,  the  pilot  EPSIS  distinguishes  seven  innovation  dimensions  ranging  from 
human  resources  to  drivers  and  barriers  to  innovation,  encompassing  22  indicators,  with  data  taken  from 
multiple  sources  including  Eurostat,  OECD,  World  Bank,  World  Economic  Forum  and  the  2010  and  2011 
Innobarometer surveys. 

The general results demonstrate that public sector  in Europe innovates but  it faces a  number of  challenges. 
The  first  results  show  that  the  involvement  of  managers  and  employees  makes  it  more  likely  that  a  public 
administration  develops  process  innovations.  The  presence  of  internal  barriers  to  innovation  (e.g.  lack  of 
management support, staff resistance or risk-averse culture) not only has a negative effect on innovation but 
also on the government’s  effectiveness  in general. Government  procurement  can not  only act as a  driver of 
business performance by demanding innovative solutions, but procurement of innovations can also contribute 
to an increased efficiency of the government sector. However, there is a clear divide in the opinion of public 
administration  officials  and  businesses  as  to  the  importance  of  innovation  versus  costs  for  winning 
procurement  tenders  with  business  having  a  much  firmer  belief  in  offering  low  costs.  The  results  also  show 
that the introduction of new and improved public services have a significant impact on business performance. 
E.g.  by  investing  in  advanced  ICT  infrastructure,  governments  have  managed  to  considerably  increase  the 
online availability of public services for businesses. 

The  Innobarometer  2010  on  innovation  in  public  administrations  shows  that  public  administration  is  highly 
innovative  with  two  out  of  three  public  administration  organisations  having  introduced  at  least  one  service 
innovation. Most drivers are ‘structural’ with the single most important driver being the introduction of new 
laws and regulations.  Barriers to innovation are probably as important  as drivers: lack of human or financial 
resources,  regulatory  requirements  and  lack  of  management  support  and  incentives  for  staff  are  the  most 
important  barriers  to  innovation  in  public  administration.  Ideas  from  staff,  management  and  clients  are  the 
major sources of information used in developing innovations. Innovation in public administration has positive 
effects on improved user access to information, improved user satisfaction and faster delivery of services. 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

113 

Results of the Innobarometer 2011 show the importance of public sector innovation for business performance. 
For  example,  companies  that  report  benefits  from  using  improved  public  administration  procedures  (e.g. 
online  completion  of  government  forms  or  access  to  online  information  on  government  services)  are  more 
likely to be an innovator and to have increasing sales. Public services innovations have a positive impact on the 
probability that a company will innovate. The results also confirm that government procurement has a positive 
impact  on  the  probability  that  a  company  will  innovate.  These  results  suggest  that  in  countries  where 
governments manage to provide improved public services for innovation and create a more business-friendly 
environment, companies show improved economic and innovative performance. Innovative and high  quality 
public services act as a driver of business performance. 

The  pilot  EPSIS  does  not  provide  a  ranking  of  countries’  performance,  since  the  availability  of  data  is  still 
limited  and  does  not  fully  capture  all  parts  of  the  public  sector  or  all  aspects  of  innovation.  However,  it  is 
sufficient  to  give  a  sense  of  the  strengths  and  weaknesses  across  countries.  In  many  countries,  services  of 
public sector are being delivered by many different types of organisations, and not just public administrations. 
The feedback from public officials consulted as part of the “Trends and Challenges in Public Sector Innovation 
in Europe” study confirms that further efforts to develop the measurement and benchmarking of public sector 
innovation  would  be  of  interest  to  most  if  not  all  Member  States  and  that  this  is  an  area  where  European 
policy should continue to show leadership. 

Thus, further work is needed to capture the full spectrum of innovation in public sector. Very  much more and 
better data is needed if EPSIS is going to continue and attain the coverage and robustness achieved with IUS. 
For this purpose, strong and coordinated efforts at the European and Member States level are needed. 

Topic 1.3: Continuous improvement & innovation 

 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

114 

1.4  Conclusions, key messages and inspiration for future action 

The  policy  process  is  not  composed  of  sequential  stages,  but  inter-linked  and  inter-dependent 
elements.  The  impact  of policy decisions should always be  anticipated, but  can never be  perfectly 
predicted,  hence  feedback  mechanisms  are  essential  to  allow  corrections  in  direction  to  be  made 
and new paths to be laid, if policy is straying too far from its goal. Furthermore, the parties that are 
most  affected  by  policy  decisions,  particularly  citizens  and  businesses,  need  to  become  active 
participants in the process: true stakeholders.  

Governments  and  judiciaries  operate  in  a  rapidly  changing  world,  in  which  ICT  opens  up  new 
possibilities  but  also  brings  heightened  expectations  from  ‘connected’  citizens  and  businesses. 
European  administrations  have  always  faced  a  stream of  evolving  and emerging  policy  challenges, 
but  now  also  the  added  constraint  of  high  public  debt  and  growing  liabilities  from  an  ageing 
population. This context calls for different types of administrative capacity that are ready to respond 
and fit for purpose, as well as new forms of cooperation and partnership. It also requires officials to 
be  inventive,  a  trait  which  has  traditionally  been  associated  with  the  private  sector,  but  which  is 
increasingly  demanded  of  public  servants  facing  complex  scenarios,  conflicting  goals  (‘more  with 
less’) and tough choices. 

  The first type to reinforce is analytical capacity: the resources to develop a robust evidence 
base,  engage  in  innovative  and  forward  thinking,  and  come  up  with  fresh  solutions  to 
ingrained problems. This requirement is not confined to policy units in central government 
alone  -  it  can  apply  to  any  level  of  government  or  official,  depending  on  structures  and 
responsibilities. However, some administrations may want to follow the lead of others, and 
establish  specialist  bodies  (like  the  Dutch  CPB,  Denmark’s  MindLab  or  UK’s  ‘Nudge  Unit’) 
which are tasked with original research or thinking ‘outside the  box’,  to advise on present 
problems and/or to engage in future scenario-building. Equally, administrations might want 
to take the taskforce approach, and assemble groups of officials from different institutions 
and  disciples  for  problem-solving,  on  a  permanent,  ad  hoc  or  flexible  basis  (like  Finland’s 
futures research, or the Netherlands Smarter Network). Whether this capacity is standalone 
or  integrated  into  the  system,  administrations  may  wish  to  consider  drawing  on  external 
insights from specialists (e.g. experts in specific fields), and/or from stakeholders to exploit 
the advantages of co-design. 

  The second type that requires the right balance is delivery capacity: the flexibility to develop 
and  adapt  implementation  solutions  to  serve  policy objectives,  meet  the  needs  of  citizens 
and businesses, and maximise cost-effectiveness at the same time. Clearly, the public sector 
is  a  major  provider  of  infrastructure  and  services  in  every  economy.  Compared  with  the 
private  sector,  which  also  faces  the  ‘make  or  buy’  decision  (in-house  or  out-source), 
governments  have  other  instruments  in  their  armouries  to  achieve  their  goals,  including 
information/persuasion,  regulation and  co-creation with ‘customers’. The careful choice of 
instrument  to  fit  the  circumstances  -  weighing  up  the  pros,  cons,  costs  and  benefits  – 
requires  rigorous  analytical  capacity.  Once  the  decision  is  taken,  however,  administrations 
face  a  continuous  pressure  to  identify  incremental  or  even  radical  improvements,  to 

Topic 1.4: Conclusions & key messages 

 
 
 
 
 
 
Quality of Public Administration – A Toolbox for Practitioners 

115 

enhance  service  delivery  (see  theme  5),  to  simplify  administration  and  reduce  the 
information  burden  on  businesses  and  citizens  (see  also  theme  6),  and  to  find  efficiency 
savings (see also theme 8). Again, co-responsibility in its various forms, including co-decision, 
co-budgeting and co-production, can enhance the quality of outcome. 

  The third type  that  is often  neglected  is oversight or regulatory  capacity: the  expertise  to 
scrutinise policy decisions and their delivery, and the authority to speak up and to question 
whether changes should be made, in the interests of continuous improvement. This form of 
capacity  should  be  both  integral  to  delivery  (monitoring),  but  also  independent  from  it 
(evaluation  and  performance  audit).  Stakeholders  should  be  invited  in  to  express  their 
insights  (co-evaluation),  but  just  as  important  is  to  ensure  a  healthy  external  audience  of 
independent  media  and  civil  society  which  can  hold  the  administration  to  account  and 
maintain its focus on the public interest (see also theme 2).  

These three types of capacity do not co-exist in isolation from each other. They overlap substantially, 
even within individual officials who can be expected to analyse needs, organise implementation, and 
scrutinise  performance  using  monitoring  data,  as  a  process  of  self-reflection.  Equally,  however, 
significant  separation  of  roles 
inevitable  and  desirable.  Whatever  the  arrangements, 
administrations should carefully consider their strengths and weaknesses in all three domains, and 
look  at  their  organisational  and  human  resources  management  (e.g.  incentives,  appraisal,  training 
and development) to ensure that the institutional focus and the best talent is not over-concentrated 
in one area (possibly, analysis) at the expense of the others (see also theme 4).  

is 

Topic 1.4: Conclusions & key messages 

 
 
 
 
 
Getting in touch with the EU 

In person 
All over the European Union there are hundreds of Europe Direct Information Centres. You can find the address of the centre 
nearest you at: http://europa.eu/contact 

On the phone or by e-mail 
Europe Direct is a service that answers your questions about the European Union. You can contact this service  
– by freephone: 00 800 6 7 8 9 10 11 (certain operators may charge for these calls),  
– at the following standard number: +32 22999696 or  
– by electronic mail via: http://europa.eu/contact 

Finding information about the EU 

Online 
Information about the European Union in all the official languages of the EU is available on the Europa website at: 
http://europa.eu   

EU Publications 
You can download or order free and priced EU publications from EU Bookshop at: http://bookshop.europa.eu. Multiple copies of 
free publications may be obtained by contacting Europe Direct or your local information centre (see http://europa.eu/contact) 

EU law and related documents 
For access to legal information from the EU, including all EU law since 1951 in all the official language versions, go to EUR-
Lex at: http://eur-lex.europa.eu 

Open data from the EU 
The EU Open Data Portal (http://data.europa.eu/euodp/en/data) provides access to datasets from the EU. Data can be 
downloaded and reused for free, both for commercial and non-commercial purposes. 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
K
E
-
0
4
-
1
7
-
9
3
3
-
E
N
-
N

DOI 10.2767/879305