COMMISSION STAFF WORKING DOCUMENT Final evaluation of the European statistical programme 2013-2020 Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL on the final evaluation of the implementation of the European statistical programme 2013-2020
Date: 2021-12-15
Institutions: European Commission, Eurostat
Subjects: EU programme, EU statistics, Eurostat, Internet site, data collection, innovation, official statistics, statistical method

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 15.12.2021  SWD(2021) 383 final  COMMISSION STAFF WORKING DOCUMENT Final evaluation of the European statistical programme 2013-2020 Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL on the final evaluation of the implementation of the European statistical programme 2013-2020 {COM(2021) 794 final}  
 
 
Table of contents 

1. 

INTRODUCTION ................................................................................................................................. 5 

1.1 Purpose and scope ........................................................................................................................ 5 

2. 

BACKGROUND TO THE INTERVENTION ..................................................................................... 6 

2.1 Description of the intervention and its objectives ........................................................................ 6 

2.2 Baseline and points of comparison ............................................................................................. 12 

3. 

IMPLEMENTATION/STATE OF PLAY .......................................................................................... 15 

3.1 Description of the current situation ............................................................................................ 15 

4.  METHOD ............................................................................................................................................ 18 

4.1 Short description of methodology .............................................................................................. 18 

4.2 Limitations and robustness of findings ....................................................................................... 20 

5.  ANALYSIS AND ANSWERS TO THE EQS .................................................................................... 20 

5.1 Relevance .................................................................................................................................... 20 

5.2 Effectiveness ................................................................................................................................ 28 

5.3 Efficiency ..................................................................................................................................... 41 

5.4 Coherence ................................................................................................................................... 52 

5.5 EU added value ............................................................................................................................ 59 

5.6 Evaluation of the points raised in Article 15 of the ESP .............................................................. 62 

6. 

CONCLUSIONS ................................................................................................................................. 66 

ANNEX 1: PROCEDURAL INFORMATION ............................................................................................ 73 

ANNEX 2: STAKEHOLDER CONSULTATION ....................................................................................... 75 

ANNEX 3: METHODS AND ANALYTICAL MODELS ........................................................................... 90 

ANNEX 4: FINAL REPORT OF THE CONTRACTOR ............................................................................. 99 

1 

 
 
Glossary 

Term or acronym 

Meaning or definition 

AFS 

AAR 

CMFB 

DG 

EBS 

EC 

ECB 

EEA 

EFTA 

EQ 

EQM 

ESAC 

ESCB 

Anti-fraud strategy 

Annual activity report 

Committee on monetary, financial and balance of payments statistics 

Directorate-General 

European business statistics 

European Commission 

European Central Bank 

European Economic Area 

European Free Trade Association 

Evaluation question 

Evaluation questions matrix  

European Statistical Advisory Committee 

European System of Central Banks 

ESGAB 

European Statistical Governance Advisory Board 

ESF 

ESP 

ESS 

ESSC 

ESS.VIP 

FAO 

FTE 

KPI 

IAS 

ISG 

European Statistical Forum 

European statistical programme 

European Statistical System 

European Statistical System Committee 

European Statistical System vision implementation programme 

Food and Agriculture Organization 

Full-time equivalent 

Key performance indicator 

Internal Audit Service 

Interservice steering group 

2 

 
 
 
JC 

MS 

MAP 

NCB 

OECD 

OLAF 

ONAs 

PC 

PMR 

SDMX 

SDG 

SMP 

SWD 

UN 

UNECE 

UNESCO 

UNSD 

UN-ICSC 

USS  

WHO 

Judgement criterion 

Member State 

Multiannual action plan 

National central bank 

Organisation for Economic Cooperation and Development 

European Anti-Fraud Office 

Other national authorities 

Public consultation 

Planning, monitoring and reporting 

Statistical data and metadata exchange 

Sustainable development goals 

Single market programme 

Staff working document 

United Nations 

United Nations Economic Commission for Europe 

United Nations Educational, Scientific and Cultural Organisation 

United Nations Statistics Division 

UN International Civil Service Commission 

User satisfaction survey 

World Health Organization 

3 

 
Definitions 

Term 

Meaning or definition 

Users 

Producers 

Users of European statistics include different categories: decision-makers 
at EU, national and local level; international organisations; academia and 
researchers; businesses; NGOs; the media; and European citizens in 
general. 

Producers or data providers are mostly national statistical institutes (NSIs) 
and other national authorities (ONAs). Other international organisations 
producing statistics also provide data in some specific cases. 

4 

 
 
 
 
 
1. 

INTRODUCTION 

1.1 Purpose and scope 

Article 13 of Regulation (EC) 223/20091 on European statistics states that ‘the European 
statistical programme shall provide the framework for the development, production and 
dissemination of European statistics, setting out the main fields and the objectives of the 
actions  envisaged  for  a  period  corresponding  to  that  of  the  multiannual  financial 
framework’. 

The  European  statistical  programme  2013-2020  was  created  by  Regulation  (EU) 
99/20132  for  the  years  2013-2017.  It  was  extended  by  Regulation  (EU)  2017/19513  to 
cover  the  remaining  period  of  the  previous  EU  multiannual  financial  framework,  i.e. 
2018-2020. Throughout this document, it is referred to as ‘the programme’ or simply ‘the 
ESP’, while the Regulation creating it is referred to as ‘the amended Regulation’. 

The general objective of the ESP is for the European Statistical System (ESS) to continue 
to be the leading provider of high-quality statistics on Europe. As a spending programme, 
the ESP constitutes the overall framework for developing, producing and disseminating 
European  statistics.  The  ESP  therefore  provides  the  financing  for  developing  and 
maintaining Eurostat’s statistical infrastructure and the ESS as managed by Eurostat. The 
ESP also provides financial support to Member States. 

The  requirement  to  carry  out  a  final  evaluation  and  present  a  report  on  the  ESP  is 
enshrined in Article 15 of the amended Regulation. Article 15(3) states: 

‘By  31 December 2021,  the  Commission  shall,  after  consulting  the  ESSC  and  the 
European  Statistical  Advisory  Committee,  submit  a  final  evaluation  report  on  the 
implementation  of  the  programme  to  the  European  Parliament  and  to  the  Council.  The 
report shall in particular evaluate: 

a)  the outcome of the reprioritisation and cost evaluation of statistical products; 

b)  the actions taken by the ESS to reduce the implementation and production costs 
for Member States and to limit  the overall burden stemming from  the statistical 
projects and fields covered by the programme; 

c)  the  progress  on  rendering  access  to  official  statistics  easier  and  more 

user-friendly, including the provision of data on the Eurostat website; and 

d)  the  progress  on  the  improvement  of  data  availability,  including  on  social 

economy activities and on the Europe 2020 indicators.’ 

The evaluation also serves to fulfil the requirements of the Commission’s push for better 
regulation, and it must therefore follow the  better regulation  guidelines for evaluations. 
The  evaluation  therefore  covers  the  five  compulsory  evaluation  criteria  (effectiveness, 
efficiency, relevance, coherence and EU added value). 

1   OJ L 87, 31.3.2009, p. 164. 
2   OJ L 39, 9.2.2013, p. 12. 
3   OJ L 284, 31.10.2017, p. 1. 

5 

 
                                                           
The  conclusions  of  the  evaluation  could  also  be  useful  when  performing  the  mid-term 
evaluation  of  the  single  market  programme  (SMP).  The  SMP  has  an  objective  of 
producing high-quality European statistics and includes the successor of the ESP. 

Even though the current evaluation focuses on the 2018-2020 period (because 2013-2017 
has  already  been  covered  by  two  previous  mid-term  evaluations  of  the  ESP4;5),  the 
conclusions of the current evaluation cover the implementation period of the programme 
in its entirety (2013-2020). 

The  current  evaluation  encompassed  all  Member  States  and  other  countries  covered  by 
the  programme.  Participation  in  the  programme  was  open  to  countries  in  the  European 
Economic  Area  (EEA)/European  Free  Trade  Association  (EFTA),  Switzerland,  and 
countries  to  which  the  EU’s  enlargement  policy  applies.  Regulation  (EU)  2017/951  on 
the ESP has no related implementing/delegated acts. 

2.  BACKGROUND TO THE INTERVENTION 

2.1 Description of the intervention and its objectives 

The general objective of the ESP is for the ESS to continue to be the leading provider of 
high-quality statistics on Europe. Official European statistics6 provided under the ESP are 
indispensable for EU decision-making and for measuring the performance and impact of 
EU initiatives. High-quality European aggregates, together with comparable statistics on 
all  individual  Member  States,  underpin  the  investment  plan  for  Europe;  the  European 
semester  exercise;  the  European  pillar  of  social  rights;  the  European  agenda  on 
migration; and the energy union. Statistics are also key instruments which strengthen the 
transparency and accountability of EU policies and which enable EU citizens to engage 
with  –  and  democratically  participate  in  –  political  life.  European  statistics  serve  the 
needs of a wide range of users, including: (i) decision-makers at EU, national and local 
level; (ii) international organisations; (iii) academia and researchers; (iv) businesses; (v) 
NGOs; (vi) the media; and (vii) the European public in general. 

As  a  spending  programme,  the  ESP  constitutes  the  overall  framework  for  developing, 
producing  and  disseminating  European  statistics.  Its  duration  was  originally  planned  to 
be  5  years  but  this  was  extended  to  correspond  to  the  duration  of  the  multiannual 
financial framework. The ESP provides the financing for developing and maintaining the 
statistical infrastructure of Eurostat and the ESS as managed by Eurostat. The ESP also 
provides  financial  support  to  Member  States.  Member  States  use  this  support  to:  (i) 
improve  their  national  statistical  systems;  (ii)  implement  measures  to  initiate  new  data 
collections;  and  (iii)  strengthen  the  quality  and  efficiency  of  statistical  production 
through innovative statistical methods and tools. 

4   First mid-term evaluation: https://europa.eu/!tcF9J4. 
5   Second mid-term evaluation: https://europa.eu/!QCyfpB. 
6   Article 1 of Regulation (EC) 223/2009 states that ‘European statistics are relevant statistics necessary 
for  the  performance  of  the  activities  of  the  Community.  European  statistics  are  determined  in  the 
European statistical programme.’ This document therefore uses the term ‘European statistics’ to refer 
to all statistics developed, produced and disseminated by Eurostat under the ESP. 

6 

 
                                                           
The statistical infrastructure enables statistics to be regularly produced and disseminated 
in the policy fields. Depending on the specific needs and uses in these policy fields, the 
statistical  infrastructure  also  enables  new  statistics  to  be  developed.  The  programme’s 
budget  can  be  roughly  split  between:  (i)  developing  and  maintaining  the  general 
methodological  and  IT  infrastructure  on  the  one  side;  and  (ii)  providing  support  to  the 
production, dissemination and development of statistics in the specific policy domains on 
the other. The total budget allocated for the implementation of the ESP in 2013-2020 was 
EUR 489 million.  

The ESP has four specific objectives: 

 

‘Objective  1:  provide  statistical  information  in  a  timely  manner,  to  support  the 
development,  monitoring  and  evaluation  of  the  policies  of  the  Union  properly 
reflecting  priorities,  while  keeping  a  balance  between  economic,  social  and 
environmental fields and serving the needs of the wide range of users of European 
statistics, including other decision-makers, researchers, businesses and European 
citizens in general, in a cost-effective manner without unnecessary duplication of 
effort; 

  Objective 2: implement new methods of production of European statistics, aiming 

at efficiency gains and quality improvements; 

  Objective  3:  strengthen  the  partnership  within  the  ESS  and  beyond  in  order  to 
further  enhance  its  productivity  and  its  leading  role  in  official  statistics 
worldwide; 

  Objective  4:  ensure  that  delivery  of  such  statistics  is  kept  consistent  throughout 
the whole duration of the programme, provided that  this does  not  interfere with 
the priority-setting mechanisms of the ESS.’ 

The programme is divided into three parts, each focusing on a specific priority area: 

I. statistical outputs; 

II. production methods of European statistics; 

III. partnership. 

Objectives 1 and 4 are covered by the measures undertaken in priority area I; objective 2 
corresponds to priority area II; and objective 3 to priority area III. 

The three priority areas are themselves divided into one or two levels of sub-areas. 

There is  a total  of 23 second-level,  detailed, more operational  objectives, spread across 
the  three  priority  areas.  A  set  of  achievement  indicators  is  provided  for  each  of  these 
objectives. 

Financial contributions in the form of grants to Member States and procurements made 
by  Eurostat  are  used  to  achieve  the  programme’s  objectives.  Eurostat  mainly  uses 
procurement to develop and maintain the general infrastructure, whereas grants help the 
national statistical systems to develop statistics in specific policy domains. 

A schematic view of the ESP’s intervention logic is presented in Figure 1 below. The key 
elements  included  in  the  intervention  logic  are  set  out  in  the  following  bullet  points 
(working from the bottom to the top of Figure 1). 

7 

 
  Needs  and  challenges:  The  programme  cannot  be  sufficiently  achieved  by  the 
Member States alone and can therefore be better achieved at EU level. The ESP 
addresses  a  number  of  needs  and  challenges  observed  in  the  societal,  economic 
and policy spheres. The regulations follow the principle of proportionality and do 
not go beyond what is necessary to achieve that objective. The annex of the ESP 
Regulation discusses the needs and challenges faced by the ESS. The diagram in 
this  annex presents  a summary version of the needs  and challenges encountered 
by  the  ESS.  Other  factors  to  take  into  consideration  are  the  evolving  nature  of 
complex and multidimensional statistics and the increasing availability of data in 
the  digital  age.  In  addition,  the  ESP  is  challenged  by  the  decentralisation  of 
statistics  production  and  the  coexistence  of  multiple  statistics  producers. 
Rationalising the production and use of statistics – and improving their efficiency 
–  are  key  goals  to  ensure  the  ESP  does  not  create  a  disproportionate 
administrative burden for statistics producers (notably Member States, which are 
facing significant financial constraints). 

  Inputs: EU legislation is an input, a resource provided by the EU. The resources 
needed to meet the legislative requirements (such as management effort and time 
spent on the ESS by the Member States, other statistics producers and Eurostat) 
have also been categorised as inputs. The funding available (through grants and 
procurements,  to:  (i)  help  the  national  statistical  systems  develop  statistics  in 
specific policy domains; and (ii) develop and maintain the general infrastructure) 
is also an input. 

  Activities: The activities resulting from the implementation of the legal acts (such 
as  the  collection,  processing  and  validation  of  data  by  Member  States, 
statistics producers  and Eurostat) are included in  this category. This  category 
also  includes  activities  to  communicate  and  disseminate  statistics.  Lastly,  an 
‘overall  management’  category  includes:  (i)  the  provision  of  methodological 
guidance; (ii) the sharing of good practices and standards (including exchange of 
knowledge  between  regional,  national,  European  and  international  levels);  (iii) 
training;  (iv)  stakeholder  dialogues;  and  (v)  participation  in  international 
activities. 

  Outputs:  The  outputs  of  the  ESP  are  the  immediate  products  of  the  activities 
implemented. The outputs are also called ‘detailed objectives’ in the Regulation, 
and they are categorised into three priority areas: statistical outputs, production 
methods of European statistics, and partnerships. Overall, the three categories 
have  remained  stable  over  time.  Figure  1  includes  a  summary  of  the 
outputs/detailed  objectives,  structured  along  the  three  priority  areas.  Table  1 
provides  a comprehensive  list  of  the  23  detailed  objectives  included  in  the 
Regulation. 

  Outcomes/objectives: In the medium-term, the ESP is expected to achieve four 
‘objectives’  (that  is  the  terminology  used  in  the  Regulations),  also  called 
‘outcomes’  (Article  4(2)  of  the  2013  Regulation).  These  four  outcomes  are 
structured  along  the  three  priority  areas  previously  identified.  The  causality 
and 
within 
outcomes/objectives is described in the annex to the Regulation. 

area  between  outputs/detailed  objectives 

each  priority 

  General  objective:  As  per  the  2013  Regulation  (Article  4.1),  the  ESP  aims  to 
enable the ESS to continue to be the leading provider of high-quality statistics on 
Europe. 

8 

 
  Strategic objectives: Deciding the strategic objectives goes beyond the ESP, its 
functioning  and  governance.  The  strategic  objectives  must  be  chosen  to  suit  the 
different  types of users of the ESP. Through providing high-quality statistics on 
Europe, the ESP is expected to contribute to: (i) improving policy-making and the 
measuring  of  the  performance  and  impact  of  initiatives;  (ii)  strengthening  the 
transparency and accountability of policies; and (iii) enabling EU citizens’ civic 
engagement  and  democratic  participation  in  political  life.  The  ESP  should 
contribute  to  these  strategic  objectives  without  creating  a  disproportionate 
administrative burden – in particular for statistics producers. In Figure 1, we have 
differentiated the result levels to indicate when the results are internal to the ESP 
(outputs, outcomes and impacts) and when they are external to the ESP (strategic 
objectives). 

9 

 
Figure 1: ESP intervention logic 

10 

 
 
 
Table 1: Detailed list of ESP objectives 

Statistical outputs available to end users 

Production methods of 
European statistics 

Partnership 

9. Partnership within ESS and 
beyond 
Objective 9.1: implement the 
enhanced ESS governance 
framework. 
Objective 9.2: improve the 
coordinating role of the Commission 
(Eurostat) as the EU’s statistical 
office. 
Objective 9.3: strengthen 
cooperation with the ESCB and 
European and international 
organisations involved in the 
production of data through common 
projects, ensure consistency 
between standards. 
Objective 9.4: promote and 
implement statistical advisory and 
statistical assistance activities in 
countries outside the EU (including 
awareness-raising activities) in line 
with the priorities of the EU’s foreign 
policy (in particular enlargement and 
ENP). 

1. Indicators 
1.1. Europe 2020 
Objective 1.1.1: provide high-quality statistical 
information, which should be available in a 
timely manner for the European Semester, to 
monitor the implementation of the Europe 2020 
targets. 
1.2. Economic governance 
Objective 1.2.1: develop new and improve 
existing statistical information relevant for EU 
decision-makers and the public at large on: (i) 
the strengthened and integrated economic 
governance of the EU; and (ii) the surveillance 
cycle integrating the stability and growth pact 
and economic policy. 
Objective 1.2.2: provide EU decision makers 
with reliable statistics and indicators for 
administrative and regulatory purposes and for 
monitoring specific EU policy commitments. 
1.3. Economic globalisation 
Objective 1.3.1: improve indicators and 
statistical information on economic 
globalisation and global value chains for EU 
decision makers and the public at large to 
create a better understanding of globalisation. 
2. Accounting frameworks 
2.1. Economic and social performance (incl. 
inequalities) 
Objective 2.1.1: supplement measurement of 
economic performance by different aspects of 
globalisation, quality of life, access to goods 
and services, environmental sustainability, 
health, well-being, social cohesion and social 
inclusion. Develop a framework for analysing 
globalised production. 
Objective 2.1.2: provide key macroeconomic 
and social indicators and principal European 
economic indicators as a coherent set of 
indicators addressing EU’s statistical data 
requirements (and statistical data requirements 
beyond the EU) and adjust principal European 
economic indicators (PEEIs) to meet evolving 
users’ needs. 
2.2. Environmental sustainability 
Objective 2.2.1: provide environmental 
accounts and climate change-related statistics, 
taking into account international developments 
in this area (including environmental footprints). 

3. Data 
3.1. Business 
Objective 3.1.1: increase 
efficiency and effectiveness of 
statistical production 
processes, including in light of 
the better regulation agenda 
and REFIT. 
Provide high-quality statistics 
on key areas where 
enterprises are the centre of 
interest. 
3.2. People’s Europe 
Objective 3.2.1: provide 
statistics on key areas of 
social policy where the citizen 
is the centre of interest (with 
expansion of demographic 
challenges). 
3.3. Geospatial, 
environmental, agricultural 
and other sectoral statistics 
Objective 3.3.1: support 
evidence-based policy making 
by a more flexible and 
increased use of spatial 
information combined with 
social, economic, territorial 
and environmental statistical 
information. 
Objective 3.3.2: provide 
environmental statistics to 
support EU policy making.  
Objective 3.3.3: provide 
energy and transport statistics 
to support the policies of the 
EU, in line with the European 
energy union, and including 
energy dependency and 
security of supply. 
Objective 3.3.4: provide 
agriculture, fisheries and 
forestry statistics to develop 
and monitor the CAP and 
CFP, reflecting key European 
strategic objectives in line 
with the BR agenda and 
REFIT. 

4. ESS quality management 
Objective 4.1: implement a 
quality-management system in the 
ESS based on compliance with 
the Code of Practice. 
5. Priority-setting and 
simplification 
Objective 5.1: implement a 
priority-setting mechanism for the 
ESS to simplify reporting 
requirements and to adjust to new 
needs for statistics. 
6. Multi-purpose statistics and 
efficiency gains in production 
Objective 6.1: (i) put in place an 
ESS business architecture 
allowing more integrated 
production of European statistics; 
(ii) harmonise and standardise 
statistical production methods and 
metadata; (iii) improve the 
horizontal and vertical integration 
of statistical production processes; 
(iv) use and integrate multiple 
data sources; (v) produce multi-
purpose statistics. 
Objective 6.2: ensure the good 
functioning and coherence of the 
ESS through effective 
collaboration and communication. 
7. Dissemination and 
communication 
Objective 7.1: make the ESS the 
first data source on European 
statistics for all users; extend and 
rationalise range of dissemination 
products, improving user 
friendliness, retrievability and 
convertibility. 
8. Training, innovation and 
research 
Objective 8.1: satisfy learning and 
development needs in the ESS 
(including future needs), and 
ensure the transfer of knowledge 
and the sharing and 
implementation of best practices. 

11 

 
 
 
2.2 Baseline and points of comparison 

Start of the programme 

When  the  ESP’s  impact  assessment  was  carried  out  in  2011,  Eurostat  found  that  the  ESS 
faced a number of challenges:  

The  need  for  European  statistics  had  been  constantly  increasing  over  the  preceding 

(1) 
years, and it was unlikely that this trend of increased need would change in the future. 

(2) 
The  nature  of  statistics  was  changing  –  there  was  a  growing  need  for  complex 
multi-dimensional  statistics  of  even  higher  quality  which  had  to  be  provided  within  shorter 
periods of time than before. 

(3) 
Due to new participants appearing on the information market providing new types of 
statistics (including participants providing information almost in real time) the future priority 
for the ESS would be the freshness of data, in particular for short-term economic information. 

(4) 
The  situation  for  statistics  producers  has  become  more  challenging  due  to  budget 
constraints both at national and European level. These constraints had become more pertinent 
with the economic crisis, as had the need to further reduce the burden of compiling statistics 
on businesses and citizens. 

The ESP was designed to face all those challenges. For this reason, its main objectives were 
combining the production of more statistics (including statistics that are more complex and of 
a higher quality) with the modernisation and reengineering of the processes used to produce 
these statistics. 

As a step further in modernising the production of statistics, the ESSC agreed in early 2015 to 
implement the ESS Vision 20207, a strategic vision for how the ESS should develop by 2020. 
The implementation initially involved eight implementation projects for ESS Vision 2020. 

First mid-term evaluation4 

The  ESP’s  state  of  implementation  was  evaluated  for  the  first  time  in  2015,  taking  into 
consideration  the  activities  carried  out  and  the  results  obtained  in  the  first  2  years  of  the 
programme  –  2013  and  2014.  The  first  mid-term  evaluation  concluded 
the 
implementation  of  the  ESP  was  advancing  well,  with  17  of  the  programme’s  23  detailed 
objectives well on track to be completed. This first evaluation provided a detailed summary 
of  the  ESP’s  produced  outputs.  Good  progress  had  also  been  made  on  projects  related  to 
modernising  the  production  systems  for  statistics.  The  first  evaluation  stated  that  the  ESP 
continued  to  be  relevant,  as  its  objectives  still  corresponded  to  the  EU’s  needs  and  it 
continued to provide clear EU added value. The evaluation concluded that Eurostat had made 
efficient use of its resources, both financial and human, with improved productivity at a time 
when  staff  numbers  were  being  reduced.  The  first  evaluation  said  that  the  ESP  was  well 

that 

7   http://ec.europa.eu/eurostat/documents/10186/756730/ESS-Vision-2020.pdf (available in English only). 

12 

 
 
                                                           
coordinated with other EU initiatives in the same field, both other Eurostat programmes and 
initiatives on statistics produced by other Commission Directorate-Generals. 

The first mid-term evaluation also made the following recommendations: 

1.  to give special attention to the objectives where problems have been encountered; 

2.  to try to secure sufficient resources to maintain the necessary level of investment to 

modernise the production of European statistics; and 

3.  to  identify  and  implement  projects  at  EU  level  which  could  maximise  EU  added 

value. 

Second mid-term evaluation5 

A second mid-term evaluation of the ESP was performed in 2018, covering implementation 
in  2015-2017.  The  results  of  the  second  mid-term  evaluation  indicated  that  the  ESP  had 
shown  a  high  degree  of  effectiveness.  The  few  problems  registered  in  the  first  mid-term 
evaluation  had  been  tackled  successfully.  The  percentages  of  planned  outputs  that  were 
achieved or on target in the 3 years under examination were constantly over 90%. 20 of the 
23 detailed objectives of the ESP could be considered on track to be accomplished by the end 
of  the  extended  programme.  The  second  mid-term  evaluation  said  that  the  remaining  three 
objectives  ((i)  objective  2  on  economic  governance;  (ii)  objective  4  on  economic 
globalisation;  and  (iii)  objective  18  on  dissemination  and  communication)  only  showed 
limited  problems,  which  would  have  to  be  taken  care  of  in  the  remaining  period  of  the 
programme. 

The  second  mid-term  evaluation  indicated  that  the  programme:  (i)  was  run  efficiently;  (ii) 
provided  clear  EU-added  value;  and  (iii)  was  coherent  internally  and  consistent  externally 
with  other  initiatives  to  produce  statistics.  The  results  also  showed  that  the  ESP  had 
contributed  –  and  continued  to  contribute  –  to  satisfying  user  needs  and  to  the  design  and 
monitoring  of  policies.  Nevertheless,  the  results  also  showed  that  ESP  needed  to  do  more. 
Eurostat  and  the  ESS  were  modernising  their  statistical  production  processes  to  respond  to 
user needs while taking advantage of new technologies. The second mid-term evaluation said 
that  the  ESP  needed  to  focus  on  implementing  its  plan  to  modernise  statistical  production 
processes, saying that this might require: (i) changes in the way statistical authorities gather 
and process data; and (ii) sufficient resources. The second mid-term evaluation also showed 
that  the  ESP’s  structure  was  fairly  complex  in  general,  with  objectives  and  sub-objectives 
covered by areas and measured by indicators which sometimes were not clearly distinct from 
one another. This made it difficult to understand and monitor the ESP. Therefore, the second 
mid-term evaluation recommended simplifying the structure of the programme for the future 
post-2020 ESP. 

Programme extension 2018-2020 

The  2013  programme  was  extended  in  2017  by  Regulation  (EU)  2017/1951  to  cover  the 
period 2018-2020 (the remaining period of the multiannual financial framework up to 2020). 
The  impact  assessment  accompanying  Regulation  (EU)  2017/1951  noted  limitations  in  the 

13 

 
ESP’s structure relating to: (i) a lack of flexibility to deliver new statistics, while also limiting 
the  associated  cost  and  administrative  burden;  and  (ii)  a  lack  of  investment  to  meet  the 
growing  demand  for  statistics  or  the  need  to  make  statistics  available  more  quickly.  The 
impact  assessment  recommended  an  amended  programme  including:  (i)  new  statistical 
outputs in line with the 10 political priorities for the Commission’s agenda for jobs, growth, 
fairness  and  democratic  change8;  (ii)  investment  in  the  statistical  infrastructure;  and  (iii) 
investment in new sources of statistics (with an increased budget). The extension aimed at: 

  providing  high-quality  statistical  information  and  closing  the  statistical  gaps  that 
needed  to  be  addressed  most  urgently,  focusing  on  a  number  of  priority  areas  that 
reflected the Commission’s 10 political priorities; 

  building the permanent  capacity needed to  respond more quickly to  emerging needs 
and  to  adapt  the  statistical  infrastructure  so  as  to  harness  the  potential  of  new  data 
sources; and 

  strengthening  the  partnership  within  the  ESS  and  beyond  to  further  increase  its 

productivity and secure its leading role in official statistics worldwide. 

The  amendment  of  the  programme  introduced  the  following  three  modifications,  among 
others. 

 

 

 

It  improved  substantially  the  timeliness  of  data  on  inequality,  poverty  and  material 
deprivation  of  people  in  Europe  (for  instance  through  flash  estimates  on  the 
distribution of household income that are fit for purpose for the European Semester). 

It  expanded  the  production  of  statistics  on  energy  (efficiency,  security,  renewables, 
consumption, prices, etc.) to accompany the strategic framework for the energy union 
and improved the timeliness of these statistics (e.g. issuing flash estimates on energy 
balances). 

It  improved  the  quality,  coverage  and  timeliness  of  environmental  data  to  support 
climate  change  policy  and  the  circular  economy  package  and  to  support  the  UN’s 
Sustainable Development Goals (SDGs). 

In the extension of the ESP, the annual budget was increased for the years 2018 to 2020 to a 
total of around EUR 208 million for the 3 years compared to around EUR 281 million for the 
previous 5 years. The reason for this increase was to: (i) enable Member States to cope with 
new demands and to modernise their production systems, as recommended in the first mid-
term evaluation of the programme; and (ii) invest in the statistical infrastructure and in new 
sources of statistics, as recommended in the impact assessment.  

Programme progress report9 

A progress report on the implementation of the ESP, covering 2018 and the first half of 2019, 
was produced in 2019. The report indicated that the implementation of the programme was 
progressing well, producing significant results under the programme’s different objectives. 

8   https://ec.europa.eu/commission/sites/beta-political/files/juncker-political-guidelines-speech_en.pdf 
9   https://ec.europa.eu/eurostat/documents/64157/4375449/ESP_progress_report_2019.pdf 

14 

 
                                                           
The  two  mid-term  evaluations  and  the  progress  report  demonstrated  that:  (i)  the  delivery 
mechanism  had  worked  effectively;  (ii)  the  programme  was  run  efficiently;  and  (iii)  the 
programme  was  reaching  its  objectives.  They  all  showed  that  the  ESP  provided  clear  EU 
added value thanks to the harmonised provision of comparable and high-quality data for the 
EU.  However,  the  evaluations  also  showed  the  need  to  ensure  sufficient  resources  in  the 
future to modernise the statistical production processes. In addition, the evaluations all argued 
that  Eurostat  needed  to  be  able  to:  (i)  respond  to  users’  growing  needs,  especially  on  the 
timeliness and coverage of new data for emerging policy needs; and (ii) become more agile 
and take advantage of new technologies. 

3. 

IMPLEMENTATION/STATE OF PLAY 

3.1 Description of the current situation 

The  ESP  lasted  8  years,  from  2013  to  2020.  The  total  programme  budget  was 
EUR 489 million (including EFTA funds and administrative expenditure in direct support of 
programme  implementation).  Around  41%  was  spent  on  grants  and  59%  on  procurements. 
Almost another EUR 225 million supplemented the ESP’s own budget (i.e. EUR 225 million 
in addition to the EUR 489 million budget to make total support of EUR 714 million) in the 
form  of  credits  sub-delegated  by  other  policy  DGs  to  cover  data  collections  specifically 
requested by these same DGs. 

The  budget  was  spent  in  the  programme’s  three  priority  areas.  The  first  priority  area, 
‘statistical outputs’, deals with the production of European statistics. In this area, the money 
was  mostly  spent  on  grants  to  provide  financial  support  to  Member  States.  Member  States 
used this support to improve their national statistical systems and to implement measures to 
carry  out  new  data  collections.  All  Member  States  benefited  from  these  grants,  which 
ultimately allowed them to produce more and better quality data. As a result, the total number 
of disseminated datasets increased by 809 – or around 18% – from 2013 to 2020, as detailed 
in Section 5.3 on efficiency. 

15 

 
Figure 2: Number of datasets published by Eurostat, 2013-202010 

s
t
e
s
a
t
a
d
f
o
r
e
b
m
u
N

6.000

5.000

4.000

4 596

4 674

4 902

5 065

5 396

5 227

5 239

5 405

3.000

2.000

1.000

0

2013

2014

2015

2016

2017

2018

2019

2020

Source: Eurostat’s dissemination database 

The second priority area, ‘production methods of European statistics’, deals with improving 
the way statistics are produced, their quality, and the way they are disseminated. In this area, 
grants  were  given  to  Member  States  to  strengthen  the  quality  and  efficiency  of  statistical 
production through innovative statistical methods and tools. Member States used this money 
to participate in the modernisation projects set out in the ESS Vision 2020 programme. In this 
same area, procurements were used either: (i) to partly finance ESS Vision 2020 projects; or 
(ii) to finance the modernisation of the statistical infrastructure used to regularly produce and 
disseminate  the  statistics.  Examples  of  results  under  this  priority  area  include:  (i)  the 
modernisation  of  the  IT  infrastructure  to  exchange  data  with  Member  States;  (ii)  the 
renovation  of  Eurostat’s  website;  and  (iii)  the  introduction  of  a  series  of  new  visualisation 
tools and new ways to reach out to users11. More details on the ESS Vision 2020 programme 
can be found in the responses to various evaluation questions (EQs), and in particular under 
EQ 2 in the chapter on relevance. 

The third priority area was ‘partnership’. This priority area aimed to support the production 
and quality of statistics by improving cooperation within the ESS and with other international 
organisations  and  countries  outside  the  EU.  In  this  area,  the  budget  was  mostly  used  to 
support the development – and, to a lesser degree, the production – of statistics in countries 
outside  the  EU  and  EFTA.  The  budget  had  a  particular  emphasis  on  enlargement  and 
European neighbourhood policy. This priority area enabled Eurostat to regularly publish data 
for  non-EU  countries,  especially  to  support  the  enlargement  process  and  negotiations  with 
those non-EU countries. 

10   The  decrease  between  2017  and  2018  was  due  to  the  implementation  of  the  new  datasets  in  national 
accounts in 2017, following the new European system of accounts (version 2010) legislation, while keeping 
those  following  the  previous  European  system  of  accounts  (version  1995).  The  datasets  following  the 
previous legislation were removed at the beginning of 2018. 

11   Further details can be found in the individual  reports of the ESS Vision 2020 (see ESSC 2021/45/13/EN, 

Table 1). 

16 

 
 
                                                           
 
 
In  spending  the  budget,  there  were  no  major  problems  and  there  were  no  infringements. 
Eurostat  put  in  place  an  effective  system  of  anti-fraud  measures,  which  prevented  cases  of 
fraud. 

Different monitoring arrangements were put in place for the ESP. 

Twice a year, the status of all planned Eurostat activities and their outputs are checked. The 
percentage  of  activities  and  outputs  which  were  on  track  or  achieved  has  consistently  been 
greater than 90%. 

A  set  of  key  performance  indicators  (KPIs)  was  monitored  each  year  and  reported  in  the 
programme  statements  of  the  ESP,  which  are  part  of  the  annual  performance  and 
management reports of the European Commission. The values of the indicators, as extracted 
from the last programme statement of the ESP12, are shown in Table 2. 

Table 2: ESP KPIs 

Source: ESP programme statement for 2020 

In 2020, almost all indicators were on track or even surpassing their respective targets. Only 
the value of the last indicator, related to objective 3, saw a slight decrease in 2020, although it 
was  still  performing  well.  Some  of  the  partnership  activities  experienced  problems  in  2020 
caused by the COVID-19 pandemic. This was the case for seminars, missions, assessments of 
statistical systems, and statistics training, all of which were impacted by the restrictions due 
to the pandemic, especially in countries outside the EU. 

More  details  on  the  indicators  can  be  found  under  the  different  EQs,  in  particular  in  the 
chapter on effectiveness. 

Eurostat wished to continue the activities of the ESP  – while adapting them to both: (i) the 
changing environment, with new possibilities for producing statistics using more sources and 

12   https://ec.europa.eu/info/publications/annual-management-and-performance-report-2020_en 

17 

 
 
                                                           
innovative technologies; and (ii) changing user needs. To achieve this, Eurostat prepared the 
statistical annex of the SMP. 

Drawing  on  the  results  of  the  two  mid-term  evaluations  of  the  ESP,  the  2018  impact 
assessment on the SMP13 indicated some important issues to address in the new programme 
such as: 

 

 

 

 

the availability of high-quality statistical information to design and monitor new EU 
policies (e.g. on globalisation, digitalisation of the economy, and security); 

the  need  for  agility  to  address  new  demands  and  provide  both  faster  evidence  on 
emerging  topics  and  deeper  analysis  of:  (i)  the  effects  of  globalisation;  (ii)  new 
technological developments; and (iii) socioeconomic trends; 

the  need  to  reduce  statistical  reporting  requirements  on  businesses  as  well  as  the 
public; 

the  need:  (i)  to  improve  analysis  of  European  statistics  on  today’s  complex  realities 
and interlinkages to better understand them; and (ii) for improved communication and 
engagement with users, especially EU citizens. 

The  SMP  Regulation14  is  applicable  from  1 January 2021.  The  SMP  has  as  a  general 
objective: 

to develop, produce and disseminate high-quality, comparable, timely and reliable European 
statistics  which  underpin  the  design,  monitoring  and  evaluation  of  all  Union  policies  and 
help  citizens,  policymakers,  authorities,  businesses,  academia  and  the  media  to  make 
informed decisions and to actively participate in the democratic process. 

The specific objective to realise the programme’s general objective is the following: 

developing, producing, disseminating and communicating high-quality European statistics in 
line with the quality criteria laid down in Article 12(1) of Regulation (EC) No 223/2009, in a 
timely,  impartial  and  cost-efficient  manner,  through  a  strengthened  European  Statistical 
System, referred to in Article 4 of Regulation (EC) No 223/2009, and enhanced partnerships 
within  that  system  and  with  all  relevant  external  parties,  using  multiple  data  sources, 
advanced  data  analytics  methods,  smart  systems  and  digital  technologies,  and  providing 
a national and, where possible, regional breakdown. 

4.  METHOD 

4.1 Short description of methodology 

The evaluation was performed as indicated in the road map. 

A  contractor,  Tetra  Tech  International  Development,  was  selected  via  an  open  call  for 
tenders.  Having  an  external  contractor  supporting  the  evaluation  process  was  deemed  very 

13   https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52018SC0320 
14   OJ L 153, 3.5.2021, p. 1. 

18 

 
                                                           
important to get an independent opinion on the work carried out by Eurostat. The contractor 
was charged with: 

  designing the evaluation, and in particular the EQs for all criteria and the indicators to 

respond to them; 

  preparing and executing a comprehensive stakeholder consultation; 

  collecting and analysing the necessary information; 

  compiling the stakeholder-consultation report; 

  writing an evaluation report to support the preparation of the Commission evaluation 

staff working document (SWD). 

The  contract  started  on  10 September 2020  and  the  contractor  delivered  its  final  report  on 
23 July 2021. 

The  stakeholder  consultation  took  place  between  December  2020  and  March  2021,  and  it 
included  a  vast  spectrum  of  stakeholders,  who  were  consulted  in  several  ways.  The 
consultation comprised: 

  scoping interviews with representatives of Eurostat and other DGs; 

  a twelve-week public consultation (PC); 

 

targeted surveys of users and producers of statistics; 

  a large number (50) of interviews of different sorts of stakeholders. 

The contractor also carried out 4 thematic case studies and 5 country case studies. It collected 
and analysed many supporting documents. 

The  contractor  triangulated  information  from  the  different  sources  to  reach  robust 
conclusions. 

The  synopsis  report  of  the  stakeholder  consultation  is  included  in  Annex  2  of  the  present 
evaluation. Details of the sources for the contractor’s desk research and the case studies can 
be found in the company’s final report in Annexes 2, 3 and 4. 

An interservice steering group (ISG) was set up for the project, including members from all 
directorates  of  Eurostat  and  from  many  other  DGs.  The  first  meeting  of  the  ISG  coincided 
with  the  kick-off  meeting  of  the  contract  on  21 September 2020.  The  ISG  then  met  on 
10 November 2020,  6 May 2021  and  6 July 2021.  The  ISG  was  able  to  comment  on  the 
different deliverables of the contract, including: (i) the design of the project; (ii) the design of 
the  stakeholder  consultation.  The  ISG  was  also  able  to  comment  on  the  Commission 
evaluation report and the SWD. 

More information on the ISG, including its composition, can be found in Annex 1. 

19 

 
4.2 Limitations and robustness of findings 

The COVID-19 crisis prevented the contractor’s  team from collecting data face-to-face. All 
activities  took  place  remotely  instead.  This  meant  that  the  team  made  stronger  and  more 
targeted efforts to ensure good response rates for the interviews to be conducted for: (i) the 
overall  programme  assessment;  (ii)  the  country  case  studies;  and  (iii)  the  thematic  case 
studies. Overall the level of stakeholder engagement was high. 

5.  ANALYSIS AND ANSWERS TO THE EQS 

This section presents the analysis and answers to the EQs for the five evaluation criteria. The 
contractor  drew  up  evaluation  sub-questions,  judgement  criteria  (JC)  and  corresponding 
indicators to help respondents to reply to the questions, as shown in the evaluation matrix in 
Annex  3.  Using  the  information  collected  on  the  five  criteria,  the  evaluation  then  also 
answers the four questions explicitly asked in Article 15 of the ESP Regulation. 

A more extensive analysis, that is more detailed level than what is possible in this document, 
can be found in the final report of the contractor (Annex 4). 

5.1 Relevance 

The  ESP  provided  the  framework  for  developing,  producing,  and  disseminating  European 
statistics, setting out the objectives of the activities envisaged for 2013-2020. It financed the 
development  and  maintenance  of  Eurostat’s  statistical  infrastructure  and  of  the  ESS.  The 
objective of the ESP was therefore to both: (i) fulfil the needs of Eurostat, the NSIs, and other 
national authorities (ONAs) as producers of European statistics; and (ii) serve the needs of a 
wide range of users of these statistics. 

The EQs are presented below. They explored the relation  between:  (i) the objectives  of the 
programme; (ii) the activities of the programme; (iii) the needs of the ESS; and (iv) the needs 
of stakeholders more generally. The following sections provide an evidence-based judgement 
of both: (i) the extent to which the ESP was originally relevant; and (ii) the suitability of the 
activities  to  achieve  the  objectives  of  the  programme.  The  first  EQ  focuses  on  ESS 
stakeholders, namely producers of statistics, while the second EQ also covers users. 

Evaluation questions 

Sub-questions 

EQ1: To what extent did 

ESP objectives 
reflect the needs of 
the ESS?  

 To what extent did ESP objectives correspond to the needs of statistical organisations within 
the ESS when it was set up in 2013? 
 To  what  extent  did  ESP  objectives  continue  to  meet  the  needs  of  statistical  organisations 
within the ESS after its extension and redesign in 2018? 
 To what extent did reprioritisation exercises meet the identified needs at the time, especially 
regarding the 2018 extension? 
 What  mechanisms  are  used  to  ensure  the  ESP  objectives  continue  to  meet  statistical 
organisations’ needs? 

EQ2: To what extent were 
ESP activities 
appropriate to 
deliver the set 
objectives of the 
ESP?  

 To what extent were ESP activities appropriate to deliver the objectives set in 2013? 
 To what extent do ESP activities continue to be appropriate to deliver the objectives revised 
in 2018 and to date? 
 What  mechanisms  were  used  to  gather  feedback  from  users?  To  what  extent  were  these 
adequate and allowed for adjustment? 
 To what extent has the ESP flexibly adapted to technological advances (such as big data)? 

20 

 
The replies to the two questions indicate that the ESP was appropriately designed to satisfy 
stakeholders’ needs. The replies also  indicate that, throughout  its  entire duration, the ESP’s 
activities  were  carried  out  to  continue  to  satisfy  these  changing  needs.  At  the  same  time, 
although the ESP made it possible to implement appropriate activities to meet its objectives, 
the replies showed that these activities were not enough to deliver all the statistics that users 
wished for. However, some of the thematic areas that users identified as missing go beyond 
the scope of the ESP – so non-delivery in these areas cannot be considered as failures in the 
implementation of the programme. 

There  was  some  tension  between  continuously  increasing  user  needs  and  limitations  from 
producers  in  keeping  up  with  new  demands  under  an  already  ambitious  programme.  This 
meant  that  a  balance  needed  to  be  struck  between  producer  capacity  and  user  requests.  In 
addition, needs related to adaptation to technological advances such as big data became more 
prominent during the programme, as reflected in the extension of the ESP. These increasingly 
prominent needs  were translated into specific  activities,  mainly under the ESS Vision 2020 
programme. Similar activities will be continued in the next programme, and their results have 
the  potential  to  provide  solutions  to  some  of  the  problems  the  ESS  still  faces  in  terms  of 
resources and costs. 

EQ1: To what extent did ESP objectives reflect the needs of the ESS? 

JC  1.1  ESP  objectives  and  priorities  corresponded  to  the  identified  needs  and  gaps  of 
the European statistics sector and to their evolution (i) in 2013; (ii) in 2018 when 
the  ESP  was  extended  and  revised;  and  (iii)  at  the  time  of  the  evaluation  in 
2020/2021 

Stakeholder feedback  for this evaluation  generally  confirmed that the specific objectives of 
the  ESP  addressed  the  needs  of  the  ESS  in  2013.  Most  producers  who  responded  to  the 
targeted survey assessed the four specific objectives of the ESP as relevant to either a great 
extent  or  a moderate  extent.  Most  stakeholders  interviewed  highlighted  that  the  objectives 
and priorities of the ESP were all relevant and well designed. They also  confirmed that the 
ESP’s  specific  objectives  were  aligned  with  the  needs  of  the  ESS  and  that  Member  States’ 
statistical strategies were often based on the ESP’s objectives. 

The  stakeholder  consultation  did  not  highlight  any  mismatch  between  the  objectives  and 
priorities of the ESP and the needs and gaps of the European statistics sector in 2018 when 
the programme was extended and revised. Producers indicated that the objectives of the ESP 
had  continued  to  meet  their  needs.  The  extension  also  made  it  possible  to  fully  align  the 
objectives of the ESP with the 10 priorities of the Juncker Commission, as detailed under JC 
2.1. 

More than half of the producers who responded to the targeted producer survey (55% of 43 
overall responses) indicated that they did not see any gaps that the objectives of the ESP did 
not  cover.  Nevertheless,  a considerable  share  of  the  producers  who  responded  (31%  of  43 
overall responses) did indeed see such gaps. 46% of the 43 that responded also indicated they 

21 

 
did not see gaps in terms of meeting future needs, compared to 27% who did. Perceived gaps 
included the provision of statistics in new or rapidly changing policy areas. 

The targeted producer survey also revealed limitations in the ESP, including time constraints, 
challenges  in  keeping  up  with  Eurostat’s  demands,  and  requests  for  new  statistics.  The 
surveyed producers said  these limitations came in addition to the already demanding nature 
of  the  ESP.  Respondents  to  the  targeted  producer  survey  also  felt  there  was  a  need  to 
strengthen partnerships in the ESS, and said that Eurostat could coordinate NSIs’ work more 
closely, as it did during the COVID-19 crisis. Insufficient coverage of access to privately held 
data was another limitation identified in the targeted producer survey. 

In addition, respondents to the targeted producer survey and producers interviewed said that 
an effective system for priority setting (especially for identifying statistics that have become 
less relevant over time) had been missing until the end of the programme. Eurostat had been 
conducting  an  annual  priority-setting  exercise,  in  consultation  with  users  and  producers  of 
statistics since the start of the ESP in 2013. For the new programme 2021-2027, this priority-
setting exercise included a review of statistical requirements in existing domains of European 
statistics with the intention of reducing costs for Member States. In February 2020, the annual 
priority-setting  exercise  was  replaced  by  a  new  priority-setting  mechanism  adopted  by  the 
ESSC, and a pilot review for climate change related statistics was also put in place15. The new 
mechanism  is  based  on  four  parts:  a  multiannual  action  plan,  reviews,  user  dialogue,  and 
resources/funding. 

JC  1.2  Mechanisms  for  feedback  and  input  from  ESS  stakeholders  existed  and  were 

perceived as appropriate 

The  ESP  relied  on  a  ‘feedback  loop’  and  input  from  ESS stakeholders.  This  feedback  loop 
consisted of the ‘partnership approach’, namely an ongoing discussion between the EU and 
Member States,  which  ensured the consideration of different  perspectives. As a result, ESS 
stakeholders consulted by the contractor for this evaluation said that they found there was a 
good  balance  in  the  steering  and  development  of  the  ESP.  Consultation  mechanisms  were 
therefore  part  of  the  ESP  and  the  ESS  stakeholders  that  were  consulted  broadly  perceived 
these mechanisms as appropriate. 

EQ2: To what extent were the ESP’s  activities appropriate to  deliver the set  objectives of 
the ESP? 

JC  2.1  The  ESP’s  activities  were  the  appropriate  tools  in  2013  to  deliver  the  set 
objectives, and they continued to be the appropriate tools in both 2018, when the 
ESP  was  extended  and  revised,  and  again  at  the  time  of  the  evaluation  in 
2020/2021 

80%  of  the  producers  who  responded  to  the  targeted  survey  (to  which  43  responded) 
confirmed that the ESP’s activities were appropriate to achieve the four specific objectives – 
either to a great or to a moderate extent. Given that the objectives of the ESP were formulated 

15   ESSC  2020/42/3/EN,  Priority-setting  in  the  ESS  (42nd  Meeting  of  the  European  Statistical  System 

Committee, February 2020). 

22 

 
                                                           
at a general level, all NSI activities related to the production of European statistics fell within 
the  scope  of  the  programme.  To  support  the  implementation  of  the  ESP,  annual  work 
programmes  set  detailed  objectives  and  outputs  for  each  year.  Interviewees  mentioned  that 
the  annual  work  programmes  made  it  possible  to  adapt  and  adjust  the  ESP.  However,  the 
interviewees  also  said  that  the  links  between  the  activities  of  the  programme,  the  ESP’s 
objectives, and the related indicators were hard to follow, even though Eurostat continuously 
registered  and  improved  such  links.  The  new  ESP  multiannual  action  plan  2021-2027  was 
designed  to  tackle  this  issue.  It  is  intended  as  a  bridge  between  the  SMP,  which  covers 
European statistics for 2021-2027, and the annual work programmes. 

In  2016,  the  impact  assessment  for  the  extension  of  the  programme16  concluded  that  there 
was a need at the time to address several statistical gaps in the ESP’s activities to support EU 
policies. In addition, the impact assessment said that the relevance of the ESP’s activities was 
being  challenged  by:  (i)  the  need  for  statistics  arising  from  the  Juncker  Commission’s  10 
political  priorities17;  and  (ii)  the  increasing  complexity  of  European  society.  The  impact 
assessment said that the ESP should therefore be amended and that there also needed to be 
investment in the statistical infrastructure and new sources of statistics through an increased 
budget. 

In addition, the extension of the programme was an opportunity to: (i) make adaptations and 
reflect the new policy directions, in line with the ESS Vision 2020; and (ii) complement the 
ongoing prioritisation in the ESP. 

To  assess  whether  the  ESP’s  activities  were  still  the  appropriate  tools  to  deliver  the  set 
objectives  at  the  time  of  the  evaluation  in  2020/2021,  the  contractor  proposed  in  the 
evaluation’s methodology to look at their statistical outputs and see whether they served the 
needs of the wide range of users. 

On  statistical  gaps  at  the  time  of  the  evaluation  in  2020/2021,  users  who  responded  to  the 
targeted survey (to which 45 responded) generally indicated that they would like Eurostat to 
collect and publish more statistics within its existing statistical themes (64% agreed with this 
statement  while  20%  of  the  respondents  were  unsure  and  16%  found  the  collection  and 
publication  of  more  statistics  unnecessary).  Respondents  highlighted  the  need  for  a  more 
detailed  and  granular  breakdown  of  different  types  of  statistics  at  regional  level.  They  also 
highlighted the need for data and statistics subdivided into more detailed sociodemographic 
attributes,  such  as  gender.  Institutional  users  that  were  interviewed  also  identified  similar 
needs  that  they  said  the  ESP’s  activities  did  not  meet.  However,  producers  (of  which  43 
responded to the survey), were less eager to extend the datasets. 43% of producers indicated 
that they would not like Eurostat to collect and publish more statistics, while 26% said they 
would. Producers pointed out the need for Eurostat to keep a balance between: (i) the demand 

16   European Commission (2016), Commission Staff Working Document  – Executive summary of the Impact 
Assessment Accompanying the document Proposal for a Regulation of the European Parliament and of the 
Council  amending  Regulation  (EU)  No  99/2013  on  the  European  statistical  programme  2013-2017,  by 
extending it to 2018-2020, (SWD(2016) 287 final, 7.9.2016). 

17   European Parliament, The Juncker Commission’s ten priorities – An end-of-term assessment (PE 637.943, 

05.2019). 

23 

 
                                                           
for  statistics;  (ii)  the  producers’  (limited)  capacity;  and  (iii)  the  increased  burden  new 
statistics could create for producers. 

Nevertheless,  all 45 users contributing  to  the targeted survey found that  European statistics 
were  appropriate  for  their  purpose,  either  to  a  great  extent  (89%)  or  to  a  moderate  extent 
(11%). This indirectly confirms the relevance of the ESP’s activities. The PC (to which 424 
responded) also confirmed this, with more than 80% of respondents indicating that European 
statistics  were  relevant  to  their  role  or  sector,  either  to  a  great  extent  (53%  agreed)  or  to  a 
moderate extent (31% agreed). In specific policy areas (assessed through the case studies on 
business statistics and managed-migration statistics), the evaluation team also confirmed this 
assessment, as both sets of statistics fitted within the objectives of the ESP and reflected the 
needs of the ESS. 

In  addition,  interviewees  from  other  European  Commission  DGs  confirmed  that  the  ESP’s 
original  activities  were  appropriate  to  meet  their  needs,  and  said  that  these  activities  had 
continued to be appropriate to meet their needs until that day. 

Nevertheless, some respondents from the European Commission were of the opinion that the 
ESP lacked flexibility. They said that any change to the scope of the programme required a 
change in EU legislation, which is a lengthy process. For example, stakeholder feedback, in 
the case study on the response to the COVID-19 outbreak, suggested that the outbreak did not 
activate  any  specific  mechanisms  embedded  in  the  ESP  to  address  the  challenges  the 
pandemic presented for European statistics. These stakeholders said that the ESP’s activities 
were neither referred to nor used to shape the response to COVID-19. They said that that the 
response to COVID-19 was based on informal solutions to unforeseen problems and needs, 
which formal ESP procedures did not support. However, these stakeholders also said that it 
was  impressive  how  the  ESS  was  kept  running  during  the  pandemic  in  Eurostat  and  the 
Member States, including by shifting to new ways of working. In addition, these stakeholders 
agreed  that  Eurostat  was  agile  in  leading  the  work  on  assessments  of  how  COVID-19 
impacted statistics18. Also,  these stakeholders said  that much work took  place even if it did 
not lead to the production of new statistics, such as Eurostat advising colleagues responsible 
for  policy  developments  on  the  statistical  implications  of  anti-COVID-19  measures.  The 
stakeholders agreed that this shows that the ESP was set up in a flexible enough way to allow 
for rapid innovations at a lower level than the ESSC.  

The continuous and increasing relevance of the ESP is confirmed by the growing number of 
users accessing European statistics on the Eurostat website. The number of users is estimated 
via the number of distinct IP addresses that were used to access the database.  

18   https://ec.europa.eu/eurostat/web/covid-19/overview 

24 

 
                                                           
Figure 3: Total number of users (distinct IP addresses) accessing Eurostat’s website 

14.000.000

12.000.000

10.000.000

8.000.000

6.000.000

4.000.000

2.000.000

0

11 731 274

8 835 878

6 134 923

3 441 157

3 771 733

3 674 145

4 070 776

2014

2015

2016

2017

2018

2019

2020

Source: Eurostat’s website monitoring 

Another  phenomenon  that  shows  the  increasing  relevance  of  the  statistics  produced  is  the 
number of data extractions made by users, which increased substantially during the ESP, as 
detailed under JC 3.2 in the chapter on effectiveness.  

Yet another indication of the increasing relevance of the statistics produced under the ESP is 
the continuous growth in the numbers of people following Eurostat on social media. This can 
be seen in the figures from social-media monitoring, which are presented in Table 3 below. 
Eurostat  opened  its  Facebook  account  on  10 January 2017  and  its  Instagram  account  on 
4 May 2020. 

Table 3: Eurostat’s followers on social media at the end of the year 
Social media 
2012 
Twitter 
10 930 
Facebook 
- 
Instagram 
- 
Source: Eurostat’s social-media monitoring 

2017 
109 365 
10 400 
- 

2014 
47 450 
- 
- 

2015 
73 000 
- 
- 

2016 
86 775 
- 
- 

2013 
25 080 
- 
- 

2018 
127 600 
26 200 
- 

2019 
144 000 
42 000 
- 

2020 
164 000 
62 000 
9 500 

Following the demands  of the scientific  community, Eurostat  has increased the numbers of 
microdata  datasets  that  are  available  for  research  purposes.  As  these  datasets  contain 
confidential data, researchers must follow a specific procedure to access them, according to 
Commission  Regulation  (EU)  No  557/2013  on  access  to  confidential  data  for  scientific 
purposes. Table 4 below shows the list of datasets and the number of requests for the years 
2014-2020.  Three  new  datasets  have  been  published  over  this  period  and  the  number  of 
requests has increased every year until 2019, diminishing only in 2020 due to the COVID-19 
pandemic.  This  is  a  sign  of  growing  interest  in  the  data  contained  within  these  datasets.  A 
single request can be made for more than one dataset. 

25 

 
 
Table 4: Requests for access to microdata, 2014-2020 
CIS 

EU-
SILC 
164 

EU-
LFS 
134 

EC
HP 
41 

206 

187 

219 

252 

276 

229 

152 

131 

173 

179 

183 

183 

40 

38 

48 

48 

45 

39 

2014 

2015 

2016 

2017 

2018 

2019 

2020 

32 

37 

36 

36 

42 

47 

43 

SES  AES  EHI
S 
13 

32 

21 

CV
TS 
8 

CSI
S 

MM
D 

ER
FT 

HB
S 

2  N/A 

1  N/A 

TUS  Requests 
number 
310 

N/A 

28 

28 

50 

48 

46 

44 

14 

9 

20 

11 

30 

27 

15 

24 

29 

29 

43 

31 

6 

6 

14 

6 

15 

18 

7 

8 

17 

16 

16 

18 

4 

6 

7 

3 

8 

5 

4  N/A 

9 

5 

3 

9 

5 

9 

36 

28 

49 

41 

N/A 

N/A 

N/A 

N/A 

N/A 

22 

352 

356 

400 

433 

462 

404 

Source: Eurostat 

JC  2.2  Mechanisms  for  feedback  and  input  by  users  existed  and  were  perceived  as 

appropriate 

Bilateral  hearings  between  Eurostat  and  European  Commission  policy  DGs  were  organised 
regularly. These bilateral hearings gave DGs the possibility to give feedback on their needs, 
including new statistics requests, to Eurostat. They were also an opportunity for the different 
parties  to  discuss  the  overall  balance  within  the  programme.  The  large  majority  of  the 
requests made by the DGs in these bilateral hearings could be taken into account by Eurostat, 
which  ensured  that  the  ESP’s  activities  remained  relevant  for  the  needs  of  the  main  users. 
These hearings also enabled Eurostat to maintain an inventory of the statistical data produced 
by  other  DGs  and  services,  thus  helping  to  avoid  possible  duplication  of  statistics.  More 
details on the hearings can be found in JC 10.2 in the coherence chapter. 

Throughout  the  period  evaluated,  Eurostat  regularly  carried  out  general  user-satisfaction 
surveys  (USSs)19.  These  surveys:  (i)  measured  the  degree  to  which  Eurostat  met  its 
obligations  towards  its  users;  and  (ii)  helped  Eurostat  ensure  that  the  design  of  the 
programme  was  appropriate,  by  implementing  improvement  actions  as  a  follow-up  on  the 
feedback received. In addition to the USSs, Eurostat carried out specific usability testing and 
used  feedback  gathered  through:  (i)  daily  contacts;  (ii)  questions  coming  into  their  user 
support services; and (iii) comments left on social media. 

In addition, as part of the ESS Vision 2020, Eurostat engaged in a regular dialogue with users 
to better understand their needs. Eurostat recognised that different user groups have different 
needs  and  planned  to  address  this  diversity  by  offering  the  right  information  in  the  most 
appropriate way. 

Eurostat  also  co-organised  the  2014,  2016  and  2018  conferences  of  European  statistics 
stakeholders to bring together producers and users of statistics to discuss user needs. 

Even  though  the  documentary  analysis  showed  that  mechanisms  for  feedback  and  input  by 
users  existed,  most  of  the  users  interviewed  were  not  familiar  with  these  mechanisms  and 
therefore  could  not  assess  them.  However,  this  is  based  on  a  limited  sample  of  users  with 
different relations to Eurostat, and is therefore not representative. 

19 The USS reports are available at Evaluation - Eurostat (europa.eu). 

26 

 
 
                                                           
JC  2.3  New  technologies  and  statistical  methods  have been  considered  in  programme-

planning exercises 

The 43 producers of statistics who responded to the targeted survey confirmed the flexibility 
of the ESP in adapting to technological advances. Most of these producers indicated that the 
programme was effective in adapting to: (i) increasing data-security risks (55% said the ESP 
was  effective  or  very  effective  in  this);  (ii)  innovation  in  technology  and  methods  for 
statistical production (77% agreed the ESP has been effective or very effective in doing so); 
(iii)  statistic  variances  across  countries  (79%  agreed);  (iv)  big  data  (58%  agreed);  and  (v) 
advances  in  data  visualisation  (72%  agreed).  However,  national  producers  that  were 
interviewed pointed out the lack of a strategy to access and integrate privately held data. 

The ESS Vision 2020 programme mentioned several challenges the ESS faced in addressing: 
(i) the new opportunities presented by new technologies and new statistical methods; and (ii) 
emerging  needs  (for  instance  to  capture  global  phenomena  and  develop  statistics  with 
increasing geographical detail). 

The  extension  of  the  programme  in  2018  recognised  several  of  these  challenges.  It  also 
recognised that there were opportunities to  improve the timeliness and relevance of official 
statistics and to reduce response burden. 

To  face  these  challenges  and  the  continued  constraints  on  resources,  the  ESS  decided  to 
gradually  implement  the  strategic  goals  set  out  in  the  ESS  Vision  2020,  building  upon  a 
holistic approach to deliver quality and efficiency gains. It would implement these goals by: 

  using  both  traditional  surveys  and  other  sources,  including  administrative  data, 
geospatial  data  and,  where  possible,  big  data  to  deliver  statistical  products  and 
services; 

  getting access to new data sources, creating methods, and finding suitable technology 

to use such data sources to produce European statistics in a reliable way; 

 

 

improving the efficiency of statistical production by further intensifying the sharing of 
knowledge, experiences, and methodologies, but also by sharing tools, data, services, 
and resources where appropriate and duly justified; 

implementing a dissemination and communication strategy for European statistics that 
is flexible enough to adapt to emerging technologies, giving guidance in a world that 
is undergoing a data revolution, and serving as a reliable pillar of democracy. 

The ESP made it possible to invest in projects launched in response to new challenges faced 
by  European  statistics,  including  through  the  ESS  Vision  2020.  This  led  to  several  new 
projects, including: 

  proofs-of-concept for both: (i) the generation of outputs (based on new data sources) 
in  response  to  user  needs;  and  (ii)  addressing  horizontal  topics  related  to  the  use  of 
‘big data’ for European statistics20; 

20   See the final report of the project on big data for official statistics (BIGD); https://europa.eu/!bf94rQ. 

27 

 
                                                           
  preparing  the  IT  infrastructure  of  the  ESS  so  it  can  exchange  large  volumes  of 

sensitive data21, which in turn supported developments towards the  

  secure exchange of intra-EU trade microdata between all ESS members22;23, making it 
possible to both substantially reduce the response burden for businesses and increase 
quality. 

Under  the  ESP,  Eurostat  developed  a  variety  of  data-visualisation  tools  to  better  meet  user 
needs. These tools presented data from different statistical themes in a way that was attractive 
and as understandable as possible so everyone could explore them. Eurostat’s tools used for 
the dissemination of statistics included: 

  visualisation tools, such as the European statistical recovery dashboard, themes in the 

spotlight, etc.24; 

 

the My Region mobile app; 

  extraction tools, such as the data browser and explorer; 

 

the  ESS  experimental  statistics  hub25,  which  provided  a  way  to  disseminate 
experimental  statistics  on  different  topics,  including  COVID-19  (examples  of  these 
experimental  statistics  include:  (i)  COVID-19  labour  effects  across  the  income 
distribution26  specifically;  or  (ii)  quarterly  data  on  registrations  and  bankruptcies, 
which  can  provide  critical  information  on  the  economic  effects  of  the  COVID-19 
pandemic27). 

Mechanisms  to  adapt  to  new  technologies  and  statistical  methods  existed  under  the  ESP  in 
the  form  of  pilot  projects  for  example.  This  clearly  shows  that  the  ESP’s  programme-
planning exercises considered new technologies and statistical methods. 

5.2 Effectiveness 

The  analysis  of  effectiveness  considers  how  successful  the  ESP’s  activities  have  been  in 
achieving  or  progressing  towards  its  objectives.  This  analysis  gives  an  opinion  on  the 
progress made to date and the role of the ESP in delivering the observed changes. It looks for 
evidence of whether or how these changes are linked to the programme. 

The EQs, presented below, explore the relation between the objectives  and activities of the 
programme. The answers to these questions provide an evidence-based judgement of: (i) the 

21   See  the  final  report  of  the  project  on  the  European  Statistical  Data  Exchange  Network  (ESDEN); 

https://europa.eu/!CN43cG. 

22   See the final report of the project on single market statistics (SIMSTAT); https://europa.eu/!hw88Qk. 
23   See the final report of the project on the re-design of Intrastat (REDESIGN); https://europa.eu/!Wb73kK. 
24   For the full list, see: https://ec.europa.eu/eurostat/help/first-visit/tools. 
25   Eurostat,  ESS 

https://ec.europa.eu/eurostat/web/experimental-

–  Experimental 

statistics 

hub: 

statistics/overview/ess. 

26   Eurostat, COVID-19 labour effects across the income distribution: https://europa.eu/!Mn9wnq. 
27   Eurostat, Experimental statistics > Quarterly registrations and bankruptcies:   

https://ec.europa.eu/eurostat/web/experimental-statistics/quarterly-registrations-and-bankruptcies. 

28 

 
                                                           
extent  to  which  the  ESP’s  objectives  were  successfully  achieved;  or  (ii)  what  factors 
influenced why an activity/objective was (partially) unsuccessful or not yet achieved. 

Evaluation questions 

EQ 3: To what extent were the 

objectives of the ESP 
2013-2020 fulfilled?  

EQ 4: To what extent did the ESP 
make progress on rendering 
access to official statistics 
easier and more user-friendly?  

in  providing 

timely statistical 

the  ESP  successful 

Sub-questions 
 To  what  extent  was 
information (Obj. 1)? Was this delivery consistent (Obj. 4)? 
 To  what  extent  was  ESP  data  used  in  the  development,  monitoring  and 
evaluation of EU policies? Is there evidence that the ESP contributed to improving 
policy  making  (at  EU  /  MS  level)?  Is  there  evidence  that  the  ESP  contributed  to 
other purposes? 
 To  what  extent  did  the  ESP  increase  the  availability  of  data,  including  social 
economy activities and on the Europe 2020 indicators? 
 To  what  extent  was  the  ESP  successful  in  introducing  efficiency  gains  in  the 
production  of  European  statistics  (Obj.  2)  and  avoiding  duplication  of  effort  (Obj. 
1)? 
 To  what  extent  was  the  ESP  successful  in  introducing  quality  improvements  in 
the production of European statistics (Obj. 2)? 
 How effective was the ESP in strengthening partnerships within and beyond the 
ESS (Obj.3)? 
 Which  factors prevented or  reduced  the  impact  of  ESP  activities?  How  could 
these be overcome?  
 Were effective feedback mechanisms in place to identify accessibility issues and 
improvements? 
 Did  users  find  the Eurostat  dissemination channels  (including the website)  easy 
to use? 
 Were users provided with sufficient information about key aspects of the data?  

EQ 5: Did ESP activity result in wider 

economic, social or 
environmental impacts?  

 Is  there  evidence  that  the  ESP  contributed  to  wider  social,  economic  or 
environmental impacts? 

The  evaluation  findings  show  that  the  ESP  was  effective  in  delivering  on  its  objectives, 
despite  remaining  weaknesses  in  the  timeliness  and  completeness  of  European  statistics. 
Between 2013 and 2020, Eurostat provided high-quality statistics which were used by a wide 
range  of  users  for  a  wide  range  of  purposes.  These  purposes  include  supporting  the 
development,  monitoring  and  evaluation  of  policies  at  the  EU  and  Member-State  level.  In 
that period, the ESP also increased the availability of its data and statistics. To a more limited 
extent, it also introduced efficiency gains in the production of its statistics. 

In  addition,  Eurostat  effectively  developed  and  strengthened  partnerships  with  its  members 
within  the  ESS,  as  well  as  with  partners  beyond  the  ESS  –  in  particular  international 
organisations.  However,  more  needs  to  be  done,  in  particular  to:  (i)  get  access  to  privately 
held data; and (ii) create partnerships with private organisations. Work towards both of these 
goals started in 2020 and is being addressed in the next programme. 

In addition to achieving its objectives, the ESP was also effective in making access to official 
statistics easier and more user-friendly between 2013 and 2020. 

EQ3: To what extent were the objectives of the ESP 2013-2020 fulfilled? 

JC 3.1 Users expressed satisfaction as to the timeliness and completeness of statistics 

Objective  1  of  the  ESP  was  to  provide  statistical  information  in  a  timely  manner.  This 
reflected a particular concern about timely data. This concern was also highlighted in the ESS 
Vision 2020 programme and repeated in both the 2016 Commission impact assessment on the 
proposal to extend the ESP, and the 2019 Commission SWD on progress in implementing the 
ESP. 

29 

 
Despite improvement in a number of statistical domains, timeliness remained an area where 
users  said  that  work  was  ongoing  and  needed.  The  continued  issue  of  timeliness  is  also 
reflected in Eurostat’s USSs19 conducted between 2013 and 2020. In that time, around half of 
users (results fluctuated between 51% and 54%, and respondent numbers fluctuated between 
1 009 and 4 839) rated the timeliness of European statistics as either ‘very  good’ or ‘good’ 
for their purposes. A higher satisfaction rate (63%) was registered in the last USS in 2020 (to 
which  1 842  responded).  Nonetheless,  as  in  previous  years,  respondents  to  the  2020  USS 
continued to ask Eurostat to further improve the quality of statistics by improving timeliness. 
Similarly, interviews conducted with users and producers of statistics, including stakeholders 
at  the  Commission,  highlighted  timeliness  as  an  issue  needing  to  be  addressed.  However, 
interviewees  also  acknowledged  that  quality  official  statistics  required  more  time  to  be 
produced,  and  confirmed  that  there  was  therefore  a  trade-off  between  timeliness  and  other 
fluctuated  between 
aspects  of  quality.  Lastly, 
domains/statistical themes. 

timeliness  of  statistics  also 

the 

However, while timeliness was identified as a challenge, improvements  in  that regard were 
also  highlighted  by  users.  For  instance,  51%  of  respondents  to  the  PC  (to  which  424 
responded)  agreed  that  ‘the  ESP  had  been  successful  in  improving  the  production  of 
European statistics in regard to timeliness and punctuality’ to a great extent, while 28% said 
this was true to a moderate extent, and only 5% said it was true to a lesser extent. Similarly, 
82% of respondents to the targeted user survey (to which there were 45 replies) thought that 
the  ESP  was  successful  in  improving  the  timeliness  and  punctuality  of  European  statistics. 
Progress  reported  against  the  ESP  objectives  confirmed  these  perceptions.  Indeed,  the 
percentage  of  outputs  achieved  or  on  target  for  strategic  objective  1  has  consistently 
increased since 2016, reaching a peak in 2019, as detailed under JC 3.6 (see Figure 5). 

On  the  completeness  of  statistics,  based  on  the  USS,  half  of  users  between  2013  and  2019 
(with results fluctuating between 49% and 52%, and respondent numbers fluctuating between 
1 009 and 4 839) rated the overall completeness of European statistics as either ‘very good’ 
or ‘good’ for their purposes, with a peak at 62% in 2020. The issue of completeness was also 
raised during interviews with users who highlighted gaps, particularly when data for certain 
countries  were  not  available.  These  users  also  noted  inconsistencies  in  how  data  was 
collected  for  different  countries.  Respondents  to  the  targeted  producer  survey  judged  the 
availability of statistical data as sufficient, pointing out limitations due to confidentiality. In 
contrast, the producers of statistics that were interviewed assessed the availability of data as 
very good, with the right balance between timeliness and availability of data. Similarly, users 
at  the  European  Commission  stressed  that  the  statistics  provided  under  the  ESP  were 
comprehensive, comparable, and reliable. 

JC  3.2  European  statistics  and  data  were  downloaded  and  used  by  a  range  of 

stakeholders for a range of purposes 

Serving the needs of a wide range of users remained a specific objective (objective 1) of the 
programme throughout the period under evaluation. 

30 

 
According  to  the  2020  USS  (to  which  1 842  responded),  39%  of  respondents  identified 
themselves  as  advanced  users,  35%  as  intermediate  users  and  26%  as  light  users. 
Respondents  belonged  to  all  categories,  from  ‘students,  academics  and  private  users’  to 
‘business’,  ‘government’  and  ‘EU  and  international  organisations’.  This  suggests  that 
European statistics were indeed downloaded and used by a range of stakeholders. 

Web  analytics  confirmed  that  European  statistics  are  downloaded  at  an  increasing  rate. 
Between 2014 and 2020, the number of data extractions made by external users from Eurostat 
reference databases greatly increased from 6 813 000 extractions to 21 480 000 (+315%) as 
shown  in  Figure  4  below.  Similarly,  between  2014  and  2020,  the  number  of  distinct  IP 
addresses  consulting  European  statistics  increased  from  3  441  157  visits  to  11  731  274 
(+240%).  The  number  of  requests  submitted  for  microdata  also  increased  until  2019,  while 
diminishing in 2020 due to the COVID-19 pandemic. 

Figure 4: Number of data extractions from Eurostat’s reference databases 2014-2020 

Source: Eurostat’s website monitoring 

Users  of  European  statistics  were  asked  in  the  2019  USS  (to  which 1 009  responded)  to 
indicate  the  reason  for  their  interest  in  using  European  statistics.  Among  the  responses 
available,  ‘monitoring  or  formulating  policy’  (chosen  by  32%)  and  ‘general  background 
information’  (chosen  by  23%)  were  the  most  commonly  reported  purposes.  Looking  at  the 
importance  of  European  statistics,  more  than  three  quarters  of  participants  (77%)  indicated 
them to be either ‘essential’ or ‘important’ for their work. 

Building on the USS, a targeted user survey (to which there were 45 responses) was carried 
out, in which respondents were asked to indicate the usefulness of European statistics for nine 
areas of practice. Six of these areas (development,  monitoring and evaluation of policies at 
the EU level; development, monitoring and evaluation of policies at the Member-State level; 
academic  research;  scientific  research;  media  reporting;  and  giving  information  on  Europe) 
received a positive assessment (more than 60% said  European statistics in  these areas were 
either ‘useful’ or ‘very useful’). 

31 

 
 
JC 3.3 Eurostat data were used in journalistic and research activity 

European  statistics  were  used  in  journalistic  and  research  activities,  even  if  these  activities 
were  not  the  main  use  of  these  statistics.  According  to  the  targeted  user  survey  (to  which 
there were 45 responses), 9% of respondents indicated that they used European statistics for 
‘academic/research’  purposes.  Similarly,  12%  of  respondents  to  the  2019  USS  (to  which 
there  were  1 009  respondents)  said  they  used  European  statistics  for  ‘research’  purposes, 
which  was  the  fourth  most  commonly  selected  main  purpose,  while  1%  indicated  that  they 
used the statistics for ‘media use’. As counted by the Commission’s ‘My news’ tool, Eurostat 
was mentioned in the media more than 6 000 times per month on average in the last months 
of 2020. 

JC 3.4 Statistics were used by EU policy makers to inform policy making and decisions 

The reformulation of objective 1 in the programme extension in 2017 stressed that decision-
makers  at  EU  level  (but  also  in  Member  States,  local  governments  and  business)  needed 
European statistics ‘to prepare, apply, monitor and assess all EU policies’. 

The  findings  from  the  USS,  the  targeted  user  survey,  and  the  PC  suggested  that  European 
statistics were indeed used by policy makers to  inform  policy making and decisions.  In the 
2019  USS  (to  which  there  were  1 009  responses),  the  most  common  purpose  identified  by 
users  of  European  statistics  was  ‘monitoring  or  formulating  policy’  (selected  by  32%). 
Additionally,  ‘preparing  legislation’  and  ‘monitoring  or  formulating  policy’  both  got  a 
combined share of 87% of participants who indicated them to be ‘essential’ or ‘important’ for 
their  work.  Similarly,  64%  of  respondents  to  the  PC  (to  which  there  were  424  responses) 
considered European statistics useful to a great extent in informing and monitoring policy and 
decision-making in Europe. 

Interviews  with  users  from  the  Commission  confirmed  these  findings.  Commission  users 
systematically  mentioned  the  comprehensiveness,  comparability  and  reliability  of  the  data 
provided  under  the  ESP  and  the  value  of  this  data  for  EU  policy  work.  For  instance,  the 
European  statistical  recovery  dashboard  for  tracking  the  recovery  from  COVID-19  was 
highlighted  by  policy  makers  in  several  DGs  as  important  to  ‘inform  and  support  policy 
makers’ and to ‘respond quickly to emerging needs’. In the case studies, regional data were 
also  reported  as  being  heavily  used  for  policy  planning,  especially  in  the  policy  area  of 
regional  development.  Similarly,  migration  statistics  were  reported  by  DG  Migration  and 
Home Affairs as important for informing EU migration policy, policy briefs, and legislative 
proposals. 

JC 3.5 Data were being used by Member State policy makers to inform policy making 

and decisions 

The findings outlined in this JC suggested that European statistics were also used by policy 
makers at Member-State level, albeit: (i) to a lesser extent than by EU policy makers; and (ii) 
to a greater extent at the national level than at the regional and local level. Further breakdown 
of the data by contributors showed that among the respondents to the PC (to which there were 

32 

 
424  responses),  81%  of  respondents  from  public  authorities  with  a  national  scope  of  work 
considered European statistics relevant to inform and monitor public policy. 

When asked about the usefulness of European statistics in the targeted user survey (to which 
there  were  45  responses),  30%  of  respondents  found  them  to  be  ‘very  effective’  and  52% 
‘effective’  in  the  ‘development,  monitoring  and  evaluation  of  policies  at  the  Member  State 
level’.  European  statistics  were  found  to  be  less  effective  at  the  regional  level  (where  only 
31% found them effective) and at the local level (where only 10% found them effective). This 
could be linked to the insufficient disaggregation of statistics at local level. 

JC 3.6 ESP activities resulted in a net increase of available data; users considered newly 

available data sources to be useful 

When  asked  in  the  targeted  producer  survey  (to  which  there  were  43  responses)  about  the 
availability of European statistics, more than three quarters of respondents indicated there had 
been an increase in  the availability of data to  a  great  extent (37% agreed) or to  a moderate 
extent  (44%  agreed).  Similarly,  in  the  targeted  user  survey  (to  which  there  were  45 
responses),  respondents  found  that  there  had  been  an  increase  in  quantity  for  general  and 
regional  statistics  (44%  agreed)  and  for  economy  and  finance  (45%  agreed).  A  noticeable 
share (43%) of respondents to the PC (to which there were 424 responses) also reported an 
increase  in  the  availability  of  data  on  population  and  social  conditions.  This  was  further 
corroborated by stakeholders interviewed as part of the asylum and managed-migration case 
study.  Interviews  conducted  with  users  of  European  statistics,  producers  of  European 
statistics, and people from EFTA and candidate countries confirmed this point. 

Table 5 below shows the quantity of statistics and long-time series28 published by Eurostat. 
Statistical coverage is calculated as the number of statistical indicators, sub-indicators and all 
their breakdowns included in Eurobase (the Eurostat dissemination database) not taking into 
account those differing only because of the time dimension29. The table thus demonstrates the 
increasing amount of data made available by Eurostat over the years. 

Table 5: Statistical coverage in millions of statistics 
Year 
2018 
Statistical coverage 
Long time-series 
Source: Eurostat reference database 

328 
35 

2017 

2019 

2020 

366 
42 

423 
50 

446 
55 

The  ESP’s  great  effectiveness  in  producing  statistics  is  further  demonstrated  by  the 
percentage  of  planned  outputs  which  were  achieved  under  ESP  objectives  1  and  4,  which 
were constantly above 92% (and above 95% in the second half of the programme), with only 
3% or less of outputs cancelled or not fully achieved.  

28   Long time series are defined in this case as those covering 10 or more consecutive years. 
29   The derived datasets are not counted, but the variables included in more than one primary dataset (e.g. GDP 

and population) are counted as different statistics. 

33 

 
                                                           
Figure  5:  Eurostat’s  activities  contributing  to  the  management  plan’s  outputs  by  year 
(2014-2020) for objective 130 

3%

5%

4%

4%

3%

4%

2%

2%

2%

2%

3%

1%

2%

92%

94%

93%

95%

96%

97%

95%

100%

3%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

2014

2015

2016

2017

2018

2019

2020

Outputs achieved or on target

Outputs revised

Outputs canceled or not fully achieved

Source: Data from Eurostat Annual Activity Reports 2014, 2015, 2016, 2017, 2018, 2019 and 
2020. (*) for the year 2014 and 2015, objective 1 and 4 are combined 

Furthermore, the few limited problems encountered in three detailed objectives in the second 
mid-term evaluation of the programme were all solved by 2019 The actions implemented to 
solve  these  problems  included:  (i)  a  pilot  to  test  the  remote-access  system  connecting  safe 
centres  in  national  statistical  institutes  (NSIs)  to  the  central  node;  (ii)  an  SWD  covering 
progress on the European public-sector accounting-standards project; and (iii) carrying out a 
survey on international sourcing. 

JC  3.7  ESS  stakeholders  and/or  users  considered  that  ESP  activity  increased  the 

efficiency of European statistics production 

Specific objective 2 of the ESP was to implement ‘new methods of production of European 
statistics aiming at efficiency gains’. As this objective relates to efficiency, a detailed answer 
to this JC has been provided under the chapter on efficiency. Overall, both the qualitative and 
quantitative evidence collected suggest that ESP activities have, to a limited extent, resulted 
in  efficiency  gains  by  increasing  the  use  of  new,  more  efficient  methods  of  production  of 
European  statistics  while  reducing  administrative  burden.  However,  although  the  consulted 
stakeholders  acknowledged  efficiency  gains  implemented  by  the  ESP,  they  also  said  that 
more innovative solutions and sources should be used to further increase efficiency. 

Eurostat  launched  a  set  of  projects  to  increase  the  efficiency  and  quality  of  the  process  for 
producing  European  statistics.  These  projects  included:  (i)  implementing  the  ESS  Vision 
2020  programme;  (ii)  strengthening  the  European  statistical  infrastructure;  and  (iii)  making 
better  use  of  big  data  and  shared  infrastructure,  as  detailed  under  JC  2.3  in  the  chapter  on 
relevance. 

30   Data from Eurostat AARs 2014, 2015, 2016, 2017, 2018, 2019 and 2020. 

34 

 
 
                                                           
JC  3.8  ESS  stakeholders  considered  that  ESP  activity  had  avoided  or  prevented 

duplication of effort on the part of stakeholders 

As  laid  down  in  its  2016-2020  strategic  plan,  Eurostat  sought  to  provide  high-quality 
statistics  in  a  ‘cost  effective  manner  without  unnecessary  duplication  of  effort’.  This  was 
achieved  through:  (i)  sharing  knowledge  and  best  practices  across  Member  States;  and  (ii) 
developing  new  technologies,  common  tools  and  collaborative  networks  in  the  ESS  for 
‘taking advantage of possible synergies and avoiding duplication of effort’. Eurostat therefore 
produces European statistics in partnership with the Member States based on shared statistical 
standards, methods, procedures, practices, and tools. 

Respondents  to  the  targeted  producer  survey  (to  which  there  were  43  responses)  indicated 
that to a moderate extent (51% agreed) and to a great extent (19% agreed) the ESP had been 
successful in reducing duplication of effort. 

Additionally, stakeholders within the Commission highlighted the ongoing dialogue between 
Eurostat and policy DGs on the ‘inventory of other statistics’ as being effective at preventing 
overlap  and  duplication  of  efforts.  Eurostat’s  interviewees  also  highlighted  the  European 
Commission’s  participation  in  discussions  in  international  partnerships.  Some  of  these 
discussions  focused  on  efforts  to  limit  the  response  burden  for  Member  States  that  are 
statistics  producers  for  different  international  organisations  by  avoiding  the  duplication  of 
statistics production. 

JC  3.9  ESS  stakeholders  and/or  users  considered  that  ESP  activity  had  increased  the 

quality of European statistics production 

As  mentioned  above,  specific  objective  2  of  the  ESP  was  to  implement  ‘new  methods  of 
production  of  European  statistics  aiming  at ...  quality  improvements’.  The  indicator  on 
timeliness already covered this objective. Additionally, the percentage of users that rated as 
‘very  good’  or  ‘good’  the  overall  quality  of  European  statistics  remained  constant  between 
2013  and  2019  (fluctuating  between  57%  and  60%,  with  respondent  numbers  fluctuating 
between  1 009  and  4 839)  and  increased  considerably  in  2020  (when  there  were  1 842 
responses to the USS) to 72%. The increase in 2020 will have to be checked in future surveys 
to confirm if the overall quality of statistics has improved. Nevertheless, it should be noted 
that overall quality is rated more favourably than individual quality dimensions. 

Similar findings were reported in the targeted user survey (to which there were 45 responses) 
where  respondents  were  asked  to  judge  the  overall  quality  of  nine  of  Eurostat’s  statistical 
themes.  Overall,  respondents  said  that  all  themes  were  of  either  high  or  moderate  quality. 
However, when asked whether there had been  changes to  data that improved the quality of 
the  statistics  available,  for  most  themes,  respondents  were  unsure  if  there  had  been  such 
changes. The only themes in which changes were noted were: (i) economy and finance where 
58% of respondents reported having noticed changes to the data that improved the quality of 
the  statistics  available;  and  (ii)  population  and  social  conditions  where  39%  of  respondents 
noted changes. 

35 

 
Producers  of  statistics,  users  of  statistics,  European  Commission  DGs,  EU  bodies,  and 
stakeholders from  EFTA and candidate countries all emphasised in  their interview the high 
quality  of  the  statistics  produced  by  Eurostat.  Additionally,  and  in  contrast  to  the  surveys’ 
findings, interviewees were positive about  the progress  which had been  made in  improving 
the quality of European statistics production. 

Overall, the quality was assessed by all stakeholders as being high, which would mean that 
there  was  little  room  for  improvement  and  would  therefore  suggest  that  the  small  increase 
witnessed still represents an improvement. 

JC 3.10 ESP activity strengthened existing partnerships or developed new ones 

The  ESS  is  a  partnership  between  Eurostat  and  the  NSIs  and  ONAs  responsible  in  each 
Member  State  for  developing,  producing  and  disseminating  European  statistics.  This 
partnership also includes the EEA and EFTA countries. Specific objective 3 of the ESP was 
to  ‘strengthen  the  partnership  within  the  ESS  and  beyond  in  order  to  further  enhance  its 
productivity and its leading role in official statistics worldwide’. 

Progress  made  against  this  objective  has  been  reported  on  in  Eurostat’s  annual  activity 
reports (AARs). An analysis of the AARs in recent years shows that the percentage of outputs 
achieved or on target for strategic objective 3 has been high since 2014, suggesting that ESP 
activities have been successful in strengthening existing partnerships. 

Figure  6:  Eurostat’s  activities  contributing  to  the  management  plan’s  outputs  by  year 
(2014-2020) for objective 3 

7%

1%

5%

3%

4%

3%

11%

4%

5%

93%

94%

93%

100%

86%

96%

95%

100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

2014

2015

2016

2017

2018

2019

2020

Outputs achieved or on target

Outputs revised

Outputs canceled or not fully achieved

Source: Data from Eurostat AARs 2014, 2015, 2016, 2017, 2018, 2019 and 2020. 

Additionally,  the  comparability  of  European  statistics  was  one  of  the  important  expected 
results  of  the  partnership  within  the  ESS.  According  to  the  USSs,  the  percentage  of 
respondents viewing the comparability of European statistics as either ‘very good’ or ‘good’ 
increased between 2014 (when it 50% assessed comparability as either ‘very good’ or ‘good’, 
from a survey to which 4 839 replied) and 2019 (when it was 53%, from a survey to which 
1 009 replied) with a jump in 2020 (when 1 842 responded to the survey) to 58%. This was 

36 

 
 
confirmed  in  the  interviews  with  users,  who  highlighted  the  close  and  intense  cooperation 
with Eurostat. 

Producers of statistics in the Member States reported an increase in – and strengthening of – 
partnerships. They also reported a good overall level of cooperation across the ESS. On the 
COVID-19 crisis, these producers of statistics said that meetings and exchanges to share good 
practices and new methods had increased since the outbreak of the pandemic. Examples of 
aspects that worked well in the partnership and which were highlighted by interviewees were: 
(i) the working groups, taskforces and meetings between Eurostat’s Director and the directors 
of the NSIs; and (ii) the annual hearings organised bilaterally between Eurostat and the policy 
DGs. In interviews, EFTA and candidate-country stakeholders also noted the strengthening of 
partnerships  taking  place  within  the  ESS.  The  evaluation  found  that  Eurostat  cooperated 
closely  with  international  organisations  to:  (i)  align  their  methodologies;  (ii)  agree  on 
definitions for individual statistical indicators; and (iii) validate and exchange data. Eurostat 
currently  has  memoranda  of  understanding  (MoUs)  signed  with  four  organisations:  (i)  the 
Organisation  for  Economic  Cooperation  and  Development  (OECD);  (ii)  the  ECB;  (iii)  the 
UN International Civil Service Commission (UN ICSC); and (iv) the International Service for 
Remunerations  and  Pensions  (ISRP).  Eurostat  has  also  signed  administrative  arrangements 
with four organisations: (i) EFTA; (ii) the UN Statistics Division (UNSD); (iii) the Food and 
Agriculture  Organisation  (FAO);  and  (iv)  the  OECD.  Stakeholders  interviewed  from  these 
organisations  were  overwhelmingly  positive  about  their  partnerships  with  Eurostat, 
highlighting a close and fruitful collaboration. 

Additionally,  according  to  the  2019  final  audit  report  on  the  effectiveness  of  Eurostat’s 
cooperation with external stakeholders, this cooperation has had ‘a significant impact on the 
implementation of the ESP’. However, despite an overall positive assessment, this final audit 
report also highlighted some challenges. To address these challenges, corrective actions were 
undertaken  in  2019  and  2020,  including:  (i)  the  development  of  a  strategy  for  international 
cooperation;  (ii)  the  revision  of  internal  guidance  on  administrative  arrangements;  and  (iii) 
the  preparation  of  formal  agreements  to  be  signed  with  external  stakeholders.  The  Internal 
Audit Service (IAS) concluded in its follow-up report in 2021 that the recommendations had 
been adequately and effectively implemented. 

Eurostat also cooperates with non-EU countries based on regional and thematic priorities to 
exchange information and practices that help to address global challenges. 

As part of its work to create a quality framework for big data, as first stipulated by ESGAB 
Recommendation 2017/4, Eurostat has, as part of the ESS Vision 2020 BIGD project, worked 
on  developing  a  methodology  and  practices  for  dealing  with  privately  held  data  and  their 
quality  aspects.  However,  Eurostat  considered  that  access  to  privately  held  data  was  still 
insufficiently  covered  and  more  needed  to  be  done  in  this  area.  In  its  2020-2024  strategic 
plan,  Eurostat  is  planning  a  stronger  partnership  with  the  private  sector  to  improve  data-
production processes. 

37 

 
JC  3.11  Stakeholders  identified  factors  that  prevented  or  reduced  the  impact  of  the 

ESP’s activity 

Producers of statistics stressed the need for more innovative solutions and sources to be used, 
such  as  big  data  based  on  privately  held  data.  However,  using  such  data  requires  specific 
skills traditionally not available in statistical offices. It also poses complex problems, such as: 

 

 

 

the  lack  of  incentives  and  enablers,  including  in  harmonised  legal  provisions,  for 
private  data  holders  to  contribute  to  the  production  of  official  statistics  for  public 
benefit; 
the  perceived  risks  for  compliance  with  data-protection  legislation  and  reputational 
risks linked to privacy protection on the part of private data holders; 
the high initial investment needed to re-purpose data and to develop methodological 
and quality frameworks related to the use of new data sources. 

Eurostat is aware of these problems and will work together with the ESS to solve them. Work 
is already planned under the SMP and included in the 2021-2027 multiannual action plan. 

Problems  with  timeliness  reduced  the  impact  and  use  of  the  statistics  being  produced  by 
Eurostat.  However,  respondents  and  interviewees  acknowledged  that  timely  statistics  were 
often dependent on external factors, namely Member States’ capacity to provide statistics in 
a timely manner. Interviewees also acknowledged that quality official statistics needed time 
to  be  produced  and  verified.  Interviewees  also  agreed  that  an  additional  challenge  to 
timeliness came from the fact that the EU’s legislative cycle is relatively long. 

Stakeholders  from  Eurostat  said  that  modernisation  required  an  upfront  investment  which 
Member States may be reluctant to pay. They said that this had impacted the ESP’s ability to 
deliver efficiency gains to reduce burden in the medium-term. The stakeholders said that the 
impact  of  COVID-19  was  likely  to  produce  financial  restraints  on  the  EU  and  on  Member 
State  budgets  dedicated  to  producing  statistics.  They  said  that,  as  a  result,  there  was  an 
increased risk of delays in achieving the objectives of modernising statistics production. 

An additional factor that limited the impact of the ESP was the increasing risk of third-party 
dissemination of non-validated or fake information.  In response, the ESGAB recommended 
in its 2019 report that Eurostat and the NSIs design appropriate actions of communication and 
outreach to highlight the trustworthiness of official statistics. To address this problem, ESAC 
also  recommended promoting statistical  literacy  at  all educational  levels  and improving the 
public’s statistical literacy to promote the value of official statistics. 

EQ4: To what extent did the ESP make progress on rendering access to official statistics 
easier and more user-friendly? 

JC  4.1  Effective  processes  were  in  place  to  monitor  and  receive  feedback  on  data 

accessibility 

User feedback is the best way to assess the accessibility and clarity of European statistics. To 
monitor  feedback  from  its  users,  Eurostat  carried  out  several  general  USSs  throughout  the 
period under evaluation. The objective of these surveys was to measure the degree to which 
Eurostat  met  its  obligation  towards  its  users.  The  surveys  were  designed  to  obtain  better 

38 

 
knowledge  about  users,  their  needs  and  their  level  of  satisfaction  with  the  services  and 
statistics provided by Eurostat, including in the area of data accessibility. USSs were held in 
2013,  2014,  2015,  2016,  2017,  2019  and  202019.  In  addition,  input  and  feedback  were  also 
received through daily contacts from: (i) questions coming into the user-support service; (ii) 
comments  left  on  social  media;  and  (iii)  a  staff  feedback  mechanism.  These  forms  of 
feedback  fed  into  the  revision  and  creation  of  new  and  existing  communication  materials. 
When asked in the PC (to which there were 424 responses) about Eurostat’s communication 
channels,  more  than  half  of  the  respondents  (53%)  said  they  found  that  these  channels  had 
been effective to a great extent in responding to their feedback and input, while 35% found 
them effective to a moderate extent. The same question was asked in the targeted user survey 
with similar results. 

Both users and producers of statistics were satisfied with the processes in place to guarantee 
effectiveness.  However,  many users may not  be  familiar with  the feedback mechanisms,  as 
indicated in the interviews. 

JC 4.2 Relevant information (e.g. new releases, data-quality standards) was available in 

a clear and accessible manner 

The  main  way  to  access  European  statistics  was  through  the  Eurostat  website,  which  has 
provided  users  with  free  access  to  its  databases  and  all  its  communication  and  electronic 
publications. 

The Eurostat website was updated daily throughout the period of the ESP, and gave visitors 
access to the latest and most comprehensive statistical information available on the EU, EU 
Member States, EFTA countries, and candidate countries. The Eurostat website also provided 
access to a narrower range of statistics covering non-members of the EU, including potential 
candidate  countries,  neighbouring  countries, and  other major  economies,  such as Japan and 
the  United  States.  People  who  registered  (free  of  charge)  were  able  to  receive  tailor-made 
email  alerts  providing  information  on  new  publications  as  soon  as  they  were  online.  They 
also had access to enhanced functionalities within databases (for example, the ability to save 
data  queries  and  make  bulk  downloads).  Journalists  were  able  to  obtain,  on  the  day  of 
publication,  news  releases  or  a  weekly  release  calendar  by  email  in  German,  English,  or 
French. Data were made available for download from the website’s data explorer in various 
formats  (XLS,  CSV,  HTML,  PC  AXIS,  SPSS,  TSV  and  PDF).  Similarly,  Eurostat’s 
dissemination unit also created different profiles for users to ensure that they receive tailored 
communication packages. 

Eurostat’s website was completely redeveloped at the beginning of the programme. The new 
website was launched in December 2014 and it has been continuously improved ever since. 
Since the beginning of the ESP, Eurostat has developed and added many new: (i) electronic 
and interactive publications; (ii) visualisation tools; (iii) mobile apps; and (iv) tools for data 
extraction. These tools present data from different statistical themes in an attractive and easy-
to-understand way for everyone to explore and can be easily accessed on Eurostat’s website. 
More details can be found under JC 2.3 in the chapter on relevance. 

39 

 
Adding modern ways of communication, Eurostat has also been active on social media, with 
its three corporate social-media accounts on Twitter, Facebook and Instagram. 

Users that were interviewed provided positive feedback on  Eurostat’s publications, with the 
publication ‘Statistics explained’ being  referred to  several  times  as useful and  well written. 
Different aspects of the publications (such as detailed metadata with descriptions, sources and 
methods) were also deemed useful and easily accessible via links. 

However, some limitations in accessibility and clarity were revealed in the USS. On average, 
in  2019  (when  there  were  1 009  responses  to  the  USS),  only  45%  of  respondents  found 
metadata  sufficient  for  their  purposes  (48%  found  them  partially  sufficient,  and  7%  not 
sufficient).  Additionally,  although  some  users  reaffirmed  in  their  comments  that  metadata 
were clear, complete, and better than those of other data providers, others found the metadata 
difficult  to  access,  not  clear  enough,  too  long,  or  too  technical.  The  main  improvements 
suggested  by  users  included:  (i)  providing  some  more  basic  metadata  that  was  easy  to 
understand  and  written  in  plain  language  for  non-specialists;  (ii)  providing  metadata  at  a 
more  detailed  level  and  for  all  indicators;  and  (iii)  giving  clearer  and  more  complete 
definitions of all codes. Other respondents wished to get more information on the production 
of statistics and the methodology used, and to understand more easily the differences between 
countries. They also  said that metadata should be consistent over time and among different 
statistics, and always updated if there were changes in the methodology. 

In both the targeted producer surveys and the targeted user surveys, respondents were asked 
to rate the extent to which Eurostat was providing sufficient information about the sources of 
the data, the data-quality standards, and the methods of data collection. Answers were mostly 
positive in all cases, suggesting that both producers and users found relevant information on 
the  data  produced  by  Eurostat  to  be  available  in  a  clear  and  accessible  manner,  with 
producers being slightly more positive in their feedback than users. 

Overall,  and  despite  the  weaknesses  identified  (notably  with  metadata),  the  findings 
suggested  that  relevant  information  was  provided  by  Eurostat  in  a  clear  and  accessible 
manner. 

EQ5: Did ESP activity result in wider economic, social, or environmental impacts? 

JC 5.1 Examples of wider social, economic, or environmental impact were identified by 

stakeholders and attributed in part or in full to ESP activity 

Identifying the direct social, economic, or environmental impact of statistics is difficult. This 
is because statistics ordinarily only have an indirect impact through the policies that are based 
on them. However, stakeholders at Eurostat highlighted that the ESP had had a wider direct 
social impact, thanks to training, skills, and the diffusion of innovation. One example of this 
is  the  creation  of  the  European  Master’s  in  Official  Statistics  (EMOS)  to  strengthen 
collaboration  among  academics  and  producers  of  official  statistics,  and  to  help  develop  a 
professional  network  of  statisticians.  Similarly,  interviewees  mentioned  the  development  of 
the ‘trusted smart statistics’ hub on web intelligence. At this hub, all NSIs are provided with 
the opportunity to explore the web as a data source, and to use algorithms and coding for free 

40 

 
to  illustrate  the  wider  social  impact  of  Eurostat  activities.  Additionally,  Eurostat  has  an 
‘education  corner’  page  on  its  website  where  teachers  can  find  material  to  use  in  the 
classroom when teaching statistics, geography, social science, etc. The education corner can 
also be used directly by students to gain an easier understanding of statistics. 

The case studies also showed the social impact  of European statistics. For instance, asylum 
and  managed-migration  statistics  were  reported  as  having  a  social  impact  linked  to  the 
integration  of  refugees  into  their  host  communities.  This  integration  affected  issues  beyond 
migration,  ranging  from  health  and  housing  to  employment.  Similarly,  the  case  studies 
showed  that  the  wide  range  of  statistics  and  data  published  by  Eurostat  in  its  European 
statistical  recovery  dashboard  as  part  of  its  response  to  the  COVID-19  crisis  had  a  wider 
impact by providing a baseline against which the impact of the crisis could be measured. The 
dashboard  thus  supports  analysis  of  how  the  pandemic  is  developing,  and  of  how  EU 
recovery policies are decided on and coordinated. More broadly, European statistics have a 
crucial impact on the long-term economy of the EU as Eurostat is responsible for providing 
the data used by DG Economic and Financial Affairs for surveillance as part of the Stability 
and Growth Pact31. 

European  statistics  also  have  an  indirect  environmental  impact.  For  example,  the  European 
environmental economic accounts set out the share of the overall economy occupied by the 
environmental  goods  and  services  sector.  It  also  detailed  the  environmental  goods  and 
services sector’s production and consumption of natural resources and energy. 

5.3 Efficiency 

Under  the  efficiency  criterion,  the  evaluation  focused  on  the  costs  of  producing  European 
statistics, trends in these costs, and the burden of producing these statistics faced by the NSIs 
and  ONAs.  Three  general  issues  were  assessed  to  find  out  whether  the  ESP  was  being  run 
efficiently. The first issue related to whether the programme ensured the best use of available 
resources, both financial and human. The second issue concentrated on the costs and burden 
involved in  producing European statistics  in  the  ESS and how these costs and burden have 
evolved. The third issue was to assess how efficient the ESP process was for reporting and 
monitoring. 

Evaluation questions 

EQ 6: To what extent were ESP 

resources used efficiently to 
achieve the desired results? 

Sub-questions 
 What  were  the  processes  in  place  to  collect  information  on  costs  and  benefits 
across ESP activities and to what extent did they inform programme decisions and 
operations? 
 What were the systems in place to monitor and optimise the use of resources? 
 How  effective  were  anti-fraud  measures  and  processes  in  place  to  prevent 
misallocation of ESP funds? 
 To  what  extend  was  the  ESP  successful  in  introducing  efficiency  gains  in 
statistics production? 

31 

https://ec.europa.eu/info/business-economy-euro/economic-and-fiscal-policy-coordination/eu-economic-

governance-monitoring-prevention-correction/stability-and-growth-pact_en 

41 

 
                                                           
Evaluation questions 

EQ 7: To what extent were ESP 

activities successful in limiting 
the administrative burdens for 
ESS stakeholders, including 
Member States and data 
providers (respondents)? 

EQ 8: Was the management / 

organisation of the ESP as a 
whole conductive to supporting 
efficient delivery? 

Sub-questions 
 What  steps  were  taken  to  analyse  the  administrative  burdens  for  NSIs  and 
statistics producers? 
 What  steps  were  taken  to  reduce  the  administrative  burdens  for  NSIs  and 
statistics producers? How effective were they? 
 To what extent did the ESP provide benefits for Member States and other data 
providers relative to the costs of delivering these results? 
 What national factors affected this balance?  
 How effective were systems in place to review the efficiency and performance of 
the ESP? 
 How effective were Eurostat governance mechanisms at monitoring the efficient 
use of resources?  

The  analysis  of  the  evidence  suggests  that  the  programme  has  been  efficient.  The  ESP 
demonstrated  efficient  use  of  both  financial  and  human  resources  while  delivering  high-
quality  European  statistics.  Moreover,  the  productivity  of  statistics  production  increased 
during 
these  developments  were  good 
governance, good management, and effective monitoring mechanisms. 

the  period  evaluated.  Factors  underpinning 

Stakeholders  expressed  concerns  about  the  costs  for  Member  States  and  the  administrative 
burden  placed  on  data  providers.  These  concerns  arose  because  of  the  need  to  produce  an 
increasing volume of statistics, which made it necessary to increase the efficiency of statistics 
production. Several initiatives were undertaken to respond to this challenge including: (i) the 
activities  within  the  ESS  Vision  2020  programme;  (ii)  an  initiative  to  modernise  business 
statistics  resulting  in  the  European  Business  Statistics  Regulation32;  (iii)  an  initiative  to 
improve  social  statistics33  with  the  Integrated  European  Social  Statistics Regulation;  (iv)  an 
initiative  to  improve  agriculture  statistics  with  the  Integrated  Farm  Statistics  Regulation34; 
and (iv) work to strengthen partnerships within the ESS. 

EQ6: To what extent were ESP resources used efficiently to achieve the desired results? 

JC  6.1  Sufficient  information  about  costs  and  benefits  across  ESP  activities  was 

available and used to inform programme decisions and operations 

Producers  were  generally  positive  in  assessing  the  cost-benefit  ratio  of  the  ESP.  In  the 
targeted  producer  survey  (to  which  there  were  43  respondents),  most  respondents  (55%) 
assessed  the  cost-benefit  ratio  as  proportionate  and  a  further  7%  assessed  it  as  very 
proportionate. However, 33% of respondents were not able to answer this question. 

Producers  were  also  asked  to  indicate  the  extent  to  which  the  ESP  had  been  successful  in 
introducing  efficiency  gains  in  the  production  of  statistics  by:  (i)  reducing  duplication  of 
effort; and (ii) reducing administrative burden. Most respondents indicated that the ESP was 
more  successful  in  reducing  duplication  (70%  of  respondents  agreed  it  was  successful  to  a 
moderate or a great extent) than administrative burden (48% agreed to a moderate or a great 
extent). 

The respondents to the targeted user survey (to which there were 45 respondents) were asked 
to assess the cost-benefit ratio of the ESP, but more than half of respondents (57%) said that 

32   Regulation (EU) 2019/2152. 
33   Regulation (EU) 2019/1700. 
34   Regulation (EU) 2018/1091. 

42 

 
                                                           
they  could  not  give  an  answer.  In  addition,  11%  of  respondents  described  the  cost-benefit 
ratio as very proportionate and 29% as proportionate. 

In  the  open-ended  questions  in  the  targeted  producer  survey,  some  respondents  felt  it  was 
necessary  to  get  rid  of  unused  data  and  to  stop  collecting  irrelevant  data.  However,  when 
asked to indicate less important statistical fields that could be deleted from the programme, 
respondents did not consider any fields as less important. 

JC  6.2  Measures  and  processes  in  place  to  monitor  and  optimise  the  efficient  use  of 

resources were effective 

To measure costs, Eurostat launched three projects in the ESS: (i) a cost-assessment survey 
on  producing  official  statistics  in  the  ESS  (including  NSIs,  regional  offices,  ONAs,  and 
national central banks (NCBs)); (ii) a survey on the cost of European statistics (by product) in 
the ESS (including NSIs, regional offices, ONAs, and NCBs); and (iii) sharing best practices 
in the area of cost accounting within the ESS. 

Information from the survey on the costs of producing official statistics in the ESS (project 1) 
is presented in Table 8 below (under JC 6.4). The cost  fell in real terms between 2013 and 
2020. A similar conclusion was derived from the analysis of the cost of European statistics 
(by product) in the ESS (project 2). The analysis covered 26 products in the first phase and 27 
products  in  the  second  phase  and  it  concentrated  on  costs  related  to  the  ESP  only.  Sharing 
best  practices  in  cost  accounting  was  aimed  to  apply  common  principles  for  the  cost-
collection surveys for the two projects. The initiative on sharing best practices in the area of 
cost accounting sought to both improve adherence to 23 common guiding principles for the 
cost-collection surveys and identify common items. 

The evaluation also found that an important way to optimise the efficient use of resources in 
the  production  of  statistics  is  to  draw  up  framework  regulations.  The  European  Business 
Statistics  Regulation  (EBS)  adopted  in  November  2019  brought  statistics  in  the  business 
sector under a common legal framework. The EBS Regulation showed that the principle of 
‘collect once, use many times’ has been followed, and it therefore significantly reduced the 
risk  of  double  counting  or  inconsistency.  The  Integrated  Farm  Statistics  Regulation  (IFS 
Regulation) adopted in 2018 modernised farm statistics by: (i) setting up the data-collection 
schedule for 2019-2029; (ii) introducing more flexibility and efficiency; and (iii) providing a 
harmonised  basis  not  only  for  farm  statistics  but  also  for  the  whole  agricultural  statistics 
system.  The  IFS  Regulation  also  strengthened  the  possibility  of  using  administrative  data 
sources,  reducing  the  burden  on  respondents  and  data  providers.  The  Integrated  European 
Social  Statistics  Regulation  (IESS  Regulation)  adopted  in  October  2019:  (i)  brought  seven 
social surveys under a common legal framework; (ii) rationalised the planning of the different 
surveys;  (iii)  significantly  reduced  inconsistency  between  the  different  surveys  by  using 
common standardised variables; and (iv) improved overall relevance. In particular, the IESS 
Regulation  makes  possible  a  better  use  of  resources  and  makes  European  Social  Statistics 
more responsive to user needs by making the core and modules of the surveys more flexible 
and by making the data more timely and more precise. 

43 

 
JC 6.3 Anti-fraud measures and processes in place were effective 

In  2013,  Eurostat  adopted  an  anti-fraud  strategy  valid  for  2014-2017  and  implemented  an 
updated strategy for 2018-2020, following the updated methodology and guidance issued by 
the  European  Anti-Fraud  Office  (OLAF).  The  overall  objective  of  Eurostat’s  anti-fraud 
strategy was to provide assurance that risks related to managing financial transactions and the 
control environment were adequately managed, taking into account the cost-effectiveness of 
managing  those  risks.  The  updated  strategy  included  an  action  plan  with  10  actions,  with 
accompanying key monitoring indicators for the actions. The strategy was evaluated in 2019, 
showing that all 10 actions had been effectively implemented in an efficient way. As a result 
of  these  good  strategies,  no  investigations  were  opened  by  OLAF,  nor  was  any  potential 
fraud-related case reported to OLAF during the ESP. 

JC 6.4 ESP operations spending was efficient 

To  assess  the  efficiency  of  spending  by  the  ESP  operations,  the  evaluation  examined  the 
following elements: 

 

the ESP budget for 2013-2020 and its execution rate; 

  Eurostat’s staff productivity; 

  expenditure on other contributions; 

 

the costs faced by Member States in producing official statistics. 

The  first  element  examined  was  the  budget  for  the  ESP.  The  budget  consisted  of:  (i)  the 
regular  budget  made  available  by  European  Commission  decisions;  and  (ii)  additional 
sources, mainly a contribution from EFTA. The execution rate of the budget showed whether 
available  financial  resources  were  spent  as  planned  to  provide  an  indication  of  whether  the 
ESP  achieved  the  desired  results.  The  execution  rate  was  extremely  high  during  the  entire 
programme, at more than 99% since 2015 and reaching 100% in 2020. 

Table 6: ESP budget execution rates 2013-2020 

Year 
Execution rate 
Sources: Eurostat Unit A.4 

2013 
92.42% 

2014 
98.35% 

2015 
99.59% 

2016 
99.61% 

2017 
99.16% 

2018 
99.85% 

2019 
99.96% 

2020 

100% 

Staff numbers at Eurostat fell by 7% (52 people) in full-time equivalents (FTEs) during the 
implementation of the ESP. In the same period, the number of annual datasets published by 
Eurostat increased by 18%. This means an increase in statistics productivity of Eurostat staff 
by more than 20% during this time. 

44 

 
Table 7: Number of Eurostat staff and published datasets at the end of years 2013-2020 
2014 
Year 
789 
Eurostat staff (FTEs) as of 31 December 
4 674 
Number of published datasets 
5.9 
Number of datasets per Eurostat staff 
Source: Eurostat 

2018 
741 
5 22735 
7.1 

2017 
751 
5 396 
7.2 

2015 
796 
4 902 
6.2 

2019 
734 
5 239 
7.1 

2016 
778 
5 065 
6.5 

2013 
791 
4 596 
5.8 

2020 
739 
5 405 
7.3 

The  staffing  policy  meant  that  there  was  pressure  on  existing  staff  and  no  flexibility  to 
manage new features of the programme (the only  possibility was to stop dealing with other 
parts of the programme so as to be able to reallocate staff to any new feature). Eurostat also 
faced recruitment challenges to attract and retain skilled staff. These staffing challenges also 
existed at Member-State level, with interviewees emphasising that some NSIs were critically 
understaffed. 

The efficiency of the ESP was strengthened by access to additional money. There was – and 
there still is – very good cooperation with thematic DGs in the European Commission, and 
this led to the mobilisation of specific budget lines. For instance, a DG was sometimes able to 
fund a particular dataset through ‘sub-delegated credits’ to the ESP (however, this raised the 
question  of  how  this  dataset  would  be  managed  in  the  absence  of  additional  staff).  The 
funding of the partnerships with candidate countries and non-EU countries provided another 
example. These partnerships fell under EU development cooperation, and were covered by a 
contribution  from  the  IPA  (Instrument  for  Pre-accession  Assistance).  Partnerships  with  the 
European  neighbourhood-policy  east  countries  were 
the  European 
Neighbourhood  Instrument,  and  partnerships  with  the  Pan-African  programme  were  funded 
under the Development Cooperation Instrument. 

funded 

from 

Eurostat  launched  a  survey  to  assess  the  cost  of  producing  official  statistics  in  the  ESS 
including  NSIs,  regional  offices,  ONAs  and  NCBs.  The  information  collected  showed  that 
costs fell by around 4% in 2012-2016. The decline in FTEs was reported at around 3% over 
the same period. Cost estimations for 2017 and 2020 showed a slight increase over the period 
but remaining practically stable in real terms. At the same time, the numbers of staff working 
in official statistics across the ESS decreased by 6 700 FTEs between 2013 and 2020. These 
estimated  figures  show  that  the  burden  on  the  resources  of  European  statistics  fell  at  an 
annual pace of 3-4%. 

Table 8: Costs estimates and FTEs in the ESS 
Year 
Costs (billion EUR, current prices) 
Staff working in official statistics in the ESS (1000s of FTEs) 
Source: Surveys on the cost of statistics in the ESS 

2013 

2017 

2020 

3.1 
49.2 

2.8 
43.1 

3.0 
42.5 

At  Member-State  level,  interviewees  commented  on  the  appropriateness  of  the  budget 
available for producing statistics. This has remained a concern since the adoption of the ESP 

35   The  decrease  between  2017  and  2018  was  due  to  the  implementation  of  the  new  datasets  in  national 
accounts in 2017, following the new European system of accounts (version 2010) legislation, while keeping 
those  following  the  previous  European  system  of  accounts  (version  1995).  The  datasets  following  the 
previous legislation were removed at the beginning of 2018. 

45 

 
                                                           
in  2013.  Back  then,  the  EU  had  just  gone  through  the  2008  economic  and  financial  crisis 
which  put  a  lot  of  pressure  on  national  budgets  and  even  more  pressure  on  funding  for 
statistical  production.  Interviewees  all  agreed  that  a  similar  situation  of  reduced  funding 
would  likely  happen  soon  because  of  the  COVID-19  crisis.  Interviewees  also  said  that 
Member States often lacked capacity and staff numbers to deal with the increasing number of 
statistics requested. 

Overall, and even if it was difficult to assess production costs, these costs remained a concern 
for the ESP. They also remained a variable that should be monitored to ensure the continued 
acceptability of the programme. The continued pressure on costs was another reason why the 
work done on harmonising methods and modernisation under the programme was critical for 
the ESP. This work helped make the ESP more efficient. 

JC 6.5 The ESP was successful in introducing efficiency gains in statistics production 

Specific objective 2 of the ESP was to implement ‘new methods of production of European 
statistics aiming at efficiency gains’. Progress made towards this objective was reported on in 
several Eurostat AARs. An analysis of these AARs over the lifetime of the ESP shows that 
the  percentage  of  outputs  achieved  or  on  target  for  strategic  objective  2  has  significantly 
increased since 2014. This suggests that the ESP activities dedicated to finding more efficient 
ways to produce European statistics have increasingly been met. As stated in Annex 2 of the 
2020  Eurostat  AAR,  the  relatively  lower  percentage  of  outputs  achieved  or  on  target 
registered  in  2020  (89%)  was  because  a  few  of  the  new  projects  encountered  some  delays, 
mainly due to COVID-19. 

Figure  7:  Eurostat’s  activities  contributing  to  the  management  plan’s  outputs  by  year 
(2014-2020) for objective 2 

14%

6%
5%

9%

4%
7%

5%

3%

1%

5%
6%

86%

89%

91%

89%

95%

96%

89%

100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%

2014

2015

2016

2017

Outputs achieved or on target

Outputs revised

2018

2020
2019
Outputs canceled or not fully achieved

Source: Data from Eurostat AARs 2014, 2015, 2016, 2017, 2018, 2019 and 2020 

To further assess the extent to which this objective was achieved, respondents to the targeted 
producer survey (to which there were 43 responses) were asked to rate the extent to which the 
ESP  had  been  successful  in  introducing  efficiency  gains  in  the  production  of  statistics  by 
reducing  administrative  burden.  48%  of  respondents  indicated  that  the  ESP  had  been 

46 

 
 
successful  to  a  moderate  or  great  extent,  while  43%  said  it  had  been  successful  to  a  small 
extent or not  at  all. This  suggests that while ESP activities may have  resulted  in  efficiency 
gains  in  the  production  of  European  statistics  by  reducing  administrative  burdens, 
stakeholders had a mixed view of the effect of these measures. 

Efficiency  gains  in  the  production  of  statistics  were  also  mentioned  during  the  interviews 
with producers of statistics. In particular, these efficiency gains were mentioned with regard 
to: (i) the increased use of administrative and register data; (ii) the increased use of estimation 
methods; and (iii) the application of microdata exchanges. Stakeholders also noted significant 
improvements  in  the  provision  of  European  statistics,  for  example  due  to  the  adoption  of 
framework  regulations.  These  efficiency  gains  were  closely  related  to  the  strengthening  of 
partnerships  within  and  beyond  the  ESS.  However,  some  interviewees  said  that  these 
efficiency gains in recent  years had tended to focus on the provision of European statistics, 
whereas  data  collection  tended  to  be  the  costliest  part  of  producing  statistics  for  Member 
States.  Additionally,  interviewees  highlighted  that  more  innovative  solutions  and  sources 
should be used, such as big data based on privately produced data. 

Finally, in 2019 the  IAS completed an audit of Eurostat’s quality management of statistical 
processes36. The  IAS  concluded that although the  primary quality  controls  embedded in  the 
production processes were adequately implemented, the quality review function was not yet 
effective.  Therefore,  IAS  made  two  recommendations,  one  to  improve  the  design  and 
implementation  of quality  reviews  and one to  strengthen the quality  controls  by production 
units.  Eurostat  implemented  an  action  plan  to  address  these  recommendations  and  the  IAS 
concluded  in  its  follow-up  on  18 January 2021  that  the  first  recommendation  had  been 
adequately  and  effectively 
therefore  closed.  The  second 
implemented  and  was 
recommendation will be implemented as set out in the action plan by 31 December 2021. 

EQ7: To what extent were ESP activities successful in limiting the administrative burdens 
for ESS stakeholders, including Member States and data providers (respondents)? 

JC  7.1  Measures  and  processes  in  place  to  analyse  administrative  burdens  for  NSIs, 

statistics producers and data providers were effective 

There  was  a  concern  expressed  by  statistics  producers  that  the  pressure  for  better  quality, 
consistency, timeliness and availability often led to a higher burden of statistical production. 
Statistics producers said that these increasing requirements may exceed producers’ resources. 

Interviewees highlighted that the Member States varied in size, and therefore the NSIs varied 
significantly  in  capacity.  However,  all  NSIs  were  obliged  to  produce  the  same  mandatory 
statistics.  Interviewees  said  that  treating  all  countries  equally  created  a  greater  burden  on 
smaller Member States. Nevertheless, they acknowledged that smaller countries may receive 
derogations and sometimes do not need to produce data with the same breakdowns or level of 
detail. 

Progress  made  with  the  framework  regulations  and  the  ESS  Vision  2020  programme  only 
partly  helped  to  reduce  costs.  Interviewees  said  that  a  really  effective  system  for  priority 

36 Final audit report on Eurostat’s quality management of statistical processes 

47 

 
                                                           
setting  was  needed,  especially  for  identifying  statistics  that  had  become  less  relevant  over 
time. Although Eurostat had been conducting an annual priority-setting exercise, interviewees 
said  this  exercise  was  not  very  effective  in  reducing  costs  for  Member  States.  In  February 
2020, the annual priority-setting exercise was replaced by a new priority-setting mechanism 
adopted by the ESSC, and a pilot review for climate change-related statistics was also put in 
place.  The  new  mechanism  is  based  on  four  elements:  a  multiannual  action  plan,  reviews, 
user dialogue, and resources/funding. 

Interviewees  from  candidate  countries  also  said  that  they  felt  a  somewhat  heavy 
administrative burden. They mentioned as valuable the support received through the financial 
support  of  the  IPA  multi-beneficiary  programme  and  the  European  statistical  training 
programme. 

In  conclusion,  smaller  Member  States  experienced  more  challenges  in  dealing  with 
administrative  burden  than  larger  countries.  Funds  available  for  pilot  statistics  partly 
contributed to limiting the burden. Processes to analyse administrative burdens for NSIs were 
partly effective from the stakeholders’ point of view. 

JC  7.2  Measures  undertaken  to  reduce  administrative  burdens  for  NSIs,  statistics 
producers and data providers resulted in an improved cost-benefit balance 

Most statistics producers consulted within the evaluation tended to provide much more input 
on production costs than on administrative burden, which is the focus of JC 7.2. Nonetheless, 
the stakeholders consulted stressed that there was a constantly increasing demand for timely 
and high-quality statistics, including at EU level. They also said that the mechanisms within 
the  ESP  did  not  help  to  address  these  problems.  Some  interviewees  highlighted  that 
institutional  users,  both  national  and  at  EU  level,  should  be  made  more  aware  of  the 
production costs of statistics. 

Stakeholders referred to several approaches that had been implemented to reduce this burden. 
The  most  effective  approaches  were  the  introduction  of  new  methods  and  tools  of  data 
collection  such  as:  (i)  more  intensive  use  of  administrative  data  and  registers;  (ii)  more 
sample  surveys;  (iii)  online  data  transmission  or  web  scraping.  Another  approach  taken  by 
Eurostat  was  the  development  of  experimental  statistics,  which  they  believed  held  great 
promise – for example the use of big data  as an additional data source.  Eurostat also  made 
efforts  to  increase  the  interoperability  of  data  and  metadata  exchange  via:  (i)  the  use  of 
standards (SDMX, SIMS); and (ii) improvements to the IT infrastructure for data exchange. 
Stakeholders also highlighted several solutions that had been developed to reduce the burden 
of producing business statistics, such as microdata exchange in intra-EU trade statistics and in 
the European system of interoperable business registers. 

The  management  and  governance  mechanisms  of  the  ESP  were  generally  considered 
efficient.  Statistics  producers  considered  that  there  was  good  coordination  with  –  and  good 
communication from – Eurostat. 

Several  interviewees  mentioned  that  the  ESS  should  adopt  a  ‘one  in,  one  out’  approach  to 
statistical surveys, because there had been no critical review of new tasks and the impact of 

48 

 
new  tasks  on  additional  resources  were  not  sufficiently  explored.  However,  some 
interviewees also acknowledged the difficulty of introducing the ‘one in, one out’ approach in 
practice,  saying  that  this  approach  had  also  been  considered  at  country  level  but  never 
successfully implemented. 

JC  7.3  NSIs,  Member  States  and  other  data  providers  considered  the  benefits  of  the 

ESP to outweigh the administrative burden they face 

All  interviewees  appreciated  the  benefits  of  participation  in  the  programme.  Closer 
partnership  and  the  exchange  of  experience  and  best  practices  between  NSIs  helped  the 
members  of  the  ESS  to  find  better  ways  to  produce  statistics.  Therefore,  the  benefits  of 
participating in the ESP outweighed the costs and burdens. The benefits and added value of 
the ESP are presented and discussed in Section 5.5. 

EQ8:  Was  the  management/organisation  of  the  ESP  as  a  whole  conducive  to  supporting 
efficient delivery? 

JC  8.1  Effective  systems  were  in  place  to  review  the  ESP’s  performance  and  identify 

risks to the efficient use of resources 

Eurostat  prepared  the  annual  work  programmes  to  ensure  that  European  statistics  could  be 
produced  with  the  resources  made  available  at  the  national  and  EU  level.  By  prioritising, 
Eurostat aimed to help reduce costs and burdens for new statistical requirements by reducing 
statistical requirements in existing domains of European statistics. 

The ESS Vision 2020 programme was founded on a clear need – agreed by all the members 
of  the  ESS  –  to  modernise  the  production  of  European  statistics  to  ensure  that  the  ESS 
remains  competitive  in  the  future.  One  of  the  five  outlined  priorities  was  the  promotion  of 
efficient and robust statistical processes. 

Staffing  challenges  existed  at  Eurostat  and  at  Member-State  level,  with  interviewees 
emphasising that some NSIs were critically understaffed (saying that this in turn might have 
consequences  for  their  capacity  to  deliver  against  the  set  objectives).  The  efficiency  of  the 
ESP  was  also  strengthened  by  the  financial  leverage  existing  at  different  levels.  There  was 
very good cooperation with thematic DGs of the European Commission, which had led to the 
mobilisation of specific budget lines. 

The financial management of the ESP was led confidently and did not create risks. 

JC 8.2 Stakeholders involved in the governance of the ESP and Eurostat had sufficient 

access to information to monitor the efficiency of resource use 

The  information  included  in  the  AARs  proved  that  Eurostat’s  operational  budget 
implemented  the  ESP  in  line  with  the  multiannual  framework  and  in  accordance  with  the 
Financial Regulation. An appropriate set of checks, both internal and external, were applied 
to  Eurostat’s  financial  transactions,  and  this  resulted  in  a  positive  assessment  on  the 
efficiency of resource use. Audit observations on budget implementation were also positive. 

49 

 
Table 9: Overview of costs and benefits identified in the evaluation 

Overview of costs – benefits identified in the evaluation 

Citizens/Consumers  

Businesses 

Administrations 

[Other…] 

ESP budget 

Recurring 
administrative costs 
(production costs for 
producers of European 
statistics (NSIs, 
ONAs, NCBs) and 
administrative burden 
for the providers of 
information, 
households and 
businesses) 

Direct and indirect 
benefits from having 
European statistics 
available for free 

Expected prediction 
from IA of ESP 
extension 

Quantitative  
/ monetary 

Burden is not 
quantified. 

Benefits 
cannot be 
quantified in 
money 
because 
European 
statistics are 
not sold and 
they do not 
have a direct 
impact on the 
economy, 
society or 
environment.  

Qualitative 

Burden is variable. 
Response burden on 
citizens is very difficult 
to measure and varies 
each year because it 
depends on: (i) which 
sources are used; (ii) 
how the data are 
collected in each 
Member State for each 
survey; (iii) the sample 
sizes compared to the 
population; (iv) the 
frequency of the surveys 
etc. 
Citizens have difficulties 
in estimating the costs. 

As a direct benefit, 
official European 
statistics are available for 
free to all EU citizens. 
As an indirect benefit, 
EU policies for citizens 
are supported by – and 

Direct 
costs / 
Direct and 
indirect 
benefits 

Qualitative 

Quantitative / 
monetary  

Qualitative  Quantitative 
/ monetary  

Qualitative  Quantitative 
/ monetary 

Not 
applicable 

As a direct 
benefit, official 
European 
statistics are 
available for 
free to all EU 
businesses. 
Businesses can 
compare their 
situation with 
similar 
businesses all 
over the EU. 
As an indirect 
benefit, EU 
policies for 
businesses are 
supported by – 
and based on – 
the statistics. 

EUR 689 m in 
2016 (latest 
available 
estimate) for all 
business 
statistics37. This 
represents less 
than 1% of the 
total 
administrative 
costs for 
businesses. It is 
not possible to 
distinguish 
costs of 
national and 
European 
statistics, as 
they sometimes 
overlap. Only 
direct costs are 
applicable. 

The direct 
benefit for the 
statistical 
administration
s (NSIs, 
ONAs, 
NCBs) is that 
they are able 
to fulfil their 
mission by 
producing 
official 
statistics. The 
direct benefits 
for the other 
administration
s are that the 
official 
statistics that 
they need are 
available for 
free. 

Benefits cannot 
be quantified in 

In the targeted 
producer 

EUR 489 
million for 
the ESP 
budget. 

Production 
costs can be 
quantified at 
EUR 3.0 
billion and 
42 500 
FTEs in 
2020 (latest 
available 
estimate) for 
all official 
statistics. 
Production 
costs fell at 
an annual 
rate of 3-4% 
during the 
ESP. It is 
not possible 
to 

37   From the impact assessment of the framework regulation integrating business statistics, now the Regulation on European Business Statistics. 

50 

 
 
 
 
 
 
 
 
 
 
 
 
 
                                                           
based on – the statistics. 

Respondents to the 
targeted user survey (to 
which there were 45 
responses) (including 
citizens and businesses) 
were asked to assess the 
cost-benefit ratio of the 
ESP, and more than half 
of respondents (57%) 
said that they did not 
know. In addition, 11% 
assessed the cost-benefit 
ratio as very 
proportionate and 29% as 
proportionate. 

money because 
European 
statistics are not 
sold and they 
do not have a 
direct impact on 
the economy, 
society or 
environment. 

distinguish 
costs of 
national and 
European 
statistics, as 
they 
sometimes 
overlap. 
Only direct 
costs are 
applicable. 

survey (to 
which 43 
responded), 
most of the 
respondents 
assessed the 
cost-benefit 
ratio of the 
ESP as 
proportionate 
(55%) or very 
proportionate 
(7%). Fewer 
respondents 
(5%) assessed 
the ratio as 
disproportion
ate, and one 
third of 
respondents 
were not able 
to assess the 
cost-benefit 
ratio. 

51 

 
 
5.4 Coherence 

This  section  focuses  on  the  internal  and  external  coherence  of  the  ESP.  The  internal 
coherence of a programme is determined by looking at how the various components of that 
programme operate together to achieve the intended objectives. The external coherence of a 
programme relates to  the extent to which the programme’s activities are aligned with  other 
EU and international partner activities. 

The  answers  to  the  EQs  analyse  where  and  how  the  programme’s  objectives  and  activities 
worked  internally  and  externally  in  a  coherent  manner,  and  points  to  areas  where  there  are 
tensions. 

Evaluation questions 
EQ9: To what extent did ESP 

activities and objectives 
contribute to the internal 
coherence of the ESS?  

EQ10: To what extent did ESP 

activities complement / 
contradict / overlap with wider 
EU activity?  

EQ11: To what extent were ESP 

activities coherent with the 
activities of international 
statistics organisations?  

Sub-questions 
 Were  the  activities  and  objectives  of  the  ESP  set  out  in  the  Regulations  and 
programme planning internally coherent? 
 At the national and international level, were processes to ensure the coherence 
of statistical data identifiable and deemed fit-for purpose?  

 Were  ESP  activities  and  data  aligned  with  the  needs  of  overarching  EU 
strategies and objectives? Were there opportunities for further alignment? 
 Were  the  statistics  delivered  throughout  the  ESP  flexible  to  respond  to  new 
strategic priorities? 
 How effectively did Eurostat coordinate with other EU bodies?  

 How effectively did Eurostat coordinate with international partners (e.g. OECD) 
on the development of international concepts, classifications, methods and other 
standards?  

No overlaps or inconsistencies were identified at the programme level between the objectives 
and  among  the  activities  listed  within  the  regulations  and  planning  documents  of  the  ESP 
(internal  coherence).  The  evaluation  identified  various  governance  bodies  and  advisory 
boards that had a coherent mandate. The work of these bodies and boards contributed to the 
internal coherence of the ESP by ensuring that its activities were in line with its objectives. 
Furthermore,  interviewees  involved  in  the  production  of  European  statistics  highlighted  the 
internal coherence of the ESP. 

Eurostat  effectively  cooperated  with  EU  bodies  and  agencies,  as  well  as  international 
organisations.  This  cooperation  resulted  in  the  external  coherence  of  the  ESP  by  ensuring 
synergies with: (i) wider EU objectives and needs; and (ii) international statistical activities. 
This  cooperation  also  ensured  the  development  of  comparable  and  harmonised  European 
statistics  at  the  regional  and  international  level.  However,  some  weaknesses  which  could 
potentially affect  the external  coherence of the ESP were identified, namely:  (i) the lack of 
flexibility in European statistics to respond to emerging needs (because this lack of flexibility 
might  cause a misalignment with  other EU strategies  should the needs not  be covered); (ii) 
the burden on Member  States when replying to  requests from  individual DGs for statistics; 
and (iii) the necessity of getting access to privately held data and creating partnerships with 
private organisations (for which activities began in 2020). Eurostat has begun to tackle these 
weaknesses and will continue to address them in the next programme. 

Overall, the findings suggested that the ESP was internally and externally coherent. 

52 

 
 
EQ9: To what extent did ESP activities and objectives contribute to the internal coherence 
of the ESS? 

JC  9.1  Activities  and  objectives  set  out  in  the  Regulations  and  internal  planning 

documents were internally coherent 

The second mid-term evaluation of the ESP indicated that the ESP was internally coherent. 
Similarly, the desk review conducted as part of this final evaluation identified no coherence 
issues between the ESP’s objectives and its activities. 

In 2013 and 2014, another spending programme was running in Eurostat: the programme for 
the modernisation of European enterprise and trade statistics38. The first mid-term evaluation 
concluded  that  the  ESP  was  well  coordinated  with  this  programme.  In  the  following  years, 
the  ESP  was  Eurostat’s  only  spending  programme  and  the  objectives  of  Eurostat  were 
identical to the objectives of the ESP, and therefore automatically coherent with it. 

The  ESP  had  three  priority  areas  set  out  in  the  regulations.  The  first  priority  area  was 
statistical  outputs,  which  dealt  with  statistics  production.  This  priority  area  defined  the 
European statistics that would be produced or developed by the ESS. The second priority area 
was production methods for European statistics, which supported the production of statistics 
by  improving  the  way  statistics  were  produced,  improving  their  quality,  and  improving  the 
way  they  were  disseminated.  The  third  priority  area  was  partnership,  which  supported  the 
production  and  quality  of  the  statistics  by  improving  cooperation  within  the  ESS  and  with 
other  international  organisations  and  non-EU  countries.  The  detailed  structure  of  the  ESP 
ensured  that  there  was  no  overlap  between  the  different  activities  undertaken  under  the 
programme.  It  therefore  ensured  coherence  between  the  programme’s  activities  and  related 
objectives. 

Drawing up the catalogue of statistical products (priority area 1) was essential for the ESP to 
be relevant for EU policies and coherent with the Commission’s objectives. 

Overall,  the  activities  and  objectives  set  out  in  the  Regulations  and  internal  planning 
documents were internally coherent,  and this was further  confirmed by interviewees during 
the stakeholder consultations. 

JC 9.2 Effective processes were in place to monitor and enable the coherence of internal 

ESP activities 

The cornerstone of the quality  framework of European statistics was the  code of practice39, 
which set the standards for developing, producing and disseminating European statistics. The 
implementation  of  the  quality  framework  in  the  ESS  was  monitored  through  peer  reviews 
which covered all the Member States, all the EFTA countries, and Eurostat itself. 

In  addition,  a  number  of  governance  bodies  provided  strategic  direction  to  the  ESS  and 
monitored its activities. These bodies included a wide range of relevant stakeholders, meeting 

38   Decision  No  1297/2008/EC  of  the  European  Parliament  and  of  the  Council  of  16  December  2008  on  a 
Programme  for  the  Modernisation  of  European  Enterprise  and  Trade  Statistics  (MEETS)  (OJ  L  340, 
19.12.2008, p. 76). 

39   https://ec.europa.eu/eurostat/documents/4031688/8971242/KS-02-18-142-EN-N.pdf. 

53 

 
                                                           
in  different  formats  to  discuss  various  issues  related  to  the  activities  of  the  ESS.  In  their 
meetings  and  discussions,  and  in  the  position  papers  and  reports  they  published,  these 
governance bodies were able to contribute to the internal coherence of the ESP by ensuring 
coherence between its activities and its objectives. These bodies were40: 

 

 

 

 

 

the European Statistical System Committee (ESSC); 

the Directors General of the National Statistical Institutes (DGINS) Conference; 

the Partnership Group; 

the European Statistical Governance Advisory Board (ESGAB); 

the European Statistical Advisory Committee (ESAC). 

JC  9.3  Stakeholders  involved  in  producing  European  statistics  considered  that  the 
ESP’s  activities  promoted  internal  coherence  and/or  did  not  negatively  impact 
internal coherence 

To confirm  the potential for internal  coherence in  the design of the ESP (see previous JC), 
stakeholders involved in producing European statistics were asked whether: (i) the objectives 
of the ESP as set out in  the Regulations and programme planning were internally coherent; 
(ii)  the  activities  of  the  ESP  as  set  out  in  the  Regulations  and  programme  planning  were 
internally coherent; and (iii) there were effective mechanisms in place to coordinate activities 
within  the  ESP.  Overall,  the  largest  share  of  respondents  agreed  with  these  statements  to  a 
great or moderate extent in relation to the programme’s objectives (85% agreed the objectives 
were coherent), the programme’s activities (86% agreed they were internally coherent), and 
the existence of coordination mechanisms (74% agreed they were effective). Noticeably, no 
respondents disagreed with any of the statements. 

Respondents  were  also  asked  to  provide  examples  of  synergies  between  the  objectives  and 
activities  of  the  ESP.  For  example,  respondents  said  that  strengthening  the  partnerships 
within  the  ESS  had  enabled  exchanges  of  knowledge  and  experience  that  led  to  the  joint 
development  and  implementation  of  new  methods.  In  turn,  new  methods  potentially  led  to 
improved timeliness and a wider scope of European statistics. 

EQ10:  To  what  extent  did  ESP  activities  complement/contradict/overlap  with  wider  EU 
activity? 

JC 10.1 ESP activities and data were actively  aligned and/or not in contradiction with 

overarching EU strategies and objectives 

Strong external coherence characterised the ESP throughout the period under evaluation. This 
was first reflected in the alignment between the ESP and overarching EU political objectives. 
Both mid-term evaluations of the ESP in 2015 and 2018 concluded that there was a high level 
of demand for official European statistics, and that the ESP’s objectives corresponded to the 
EU’s needs. Similarly, the Eurostat strategic plan for 2016-2020 highlighted the importance 

40   Please  refer  to  https://ec.europa.eu/eurostat/web/european-statistical-system/overview  for  details  on  these 

bodies. 

54 

 
                                                           
of Eurostat’s role in helping to achieve the overall objectives of the Commission by providing 
high-quality  statistics.  The  strategic  plan  also  provided  extensive  examples  of  relevant 
European 
the  European  Commission’s  10  general 
objectives/priorities. In addition, in 2015 an overview of the 14 priority areas informing the 
strategic objectives of the ESP highlighted the correspondence between the Juncker priority 
areas and the sub-priority areas of the ESP. 

statistics  corresponding 

to 

Stakeholders  at  European  Commission  DGs  and  EU  bodies  interviewed  as  part  of  the 
programme assessment supported this conclusion, as did the case studies. They said that the 
ESP  was  indeed  aligned  with  the  European  Commission’s  policy  priorities.  However,  the 
stakeholders and case studies also argued that there was potential for improvement, as ESS’s 
flexibility and capacity to react to sudden changes and emerging priorities could be enhanced. 

JC  10.2  Effective  mechanisms  were  in  place  to  monitor  and  respond  to  wider  EU 

strategies and priorities 

In  addition  to  the  question  of  alignment  with  wider  EU  objectives,  external  coherence  also 
required a certain level of coordination between the ESP and the work done by other DGs to 
meet statistical needs. Since 2013, Eurostat has maintained an inventory of the statistical data 
produced  by  other  DGs  and  services.  Throughout  the  period  evaluated,  the  updating  and 
validation  of  this  inventory  was  performed  by  statistical  correspondents  every  year.  This 
updating  and  validation:  (i)  included  the  statistical  data  collected  by  the  different  DGs;  (ii) 
included  plans  for  future  collections  of  statistical  data;  and  (iii)  indicated  whether  the  data 
could  be  reused  by  other  DGs.  Eurostat  checked  the  completeness  of  the  information 
provided by the DGs and, based on a conversation with the DGs (in bilateral hearings), it was 
decided what kind of coordination or cooperation was needed. 

In  2017,  the  Commission’s  IAS  completed  an  audit41  on  the  production  process  and  the 
quality of statistics not produced by Eurostat. The overall objective of the audit was to assess 
whether the Commission had in place an effective process to ensure the quality of statistics 
not produced by Eurostat (i.e. produced or acquired and used by DGs and services) to support 
its key policies. The audit concluded that, within the limits of the current framework and the 
respective  responsibilities  of  those  involved,  some  steps  had  already  been  taken  both  in 
Eurostat and in other DGs to coordinate and manage the production process of statistics by 
DGs to improve their quality.  

However, the audit also found that the current framework was not robust enough to ensure: 
(i)  that  the  quality  of  the  statistics  produced  by  other  DGs  were  of  a  satisfactory  standard 
overall; and (ii) that the various processes currently used were sustainable in the longer term. 
Therefore,  the  IAS  recommended  that  Eurostat  and  the  other  DGs  strengthen  the  overall 
production  process  and  improve  the  production  and  quality  of  other  statistics.  The 
recommendations for Eurostat concerned in particular: (i) the management of the inventory of 
statistics  produced  by  other  DGs;  and  (ii)  guidance  to  DGs  on  methodological  and  quality 
issues. Eurostat implemented an action plan to address these recommendations, and the IAS 

41   https://webgate.ec.testa.eu/Ares/documentInfoDetails.do?documentId=080166e5b824794f&_f=ext. 

55 

 
                                                           
concluded in its follow-up report in 2020 that the recommendations had been adequately and 
effectively implemented and were therefore closed. During the bilateral hearings between the 
DGs  and  Eurostat,  DGs  were  able  to  ask  for  additional  statistics  needed  for  upcoming  EU 
policy proposals. Eurostat then integrated the information from these hearings into its annual 
planning processes and was able to collect voluntary data from Member States to address the 
DGs’ needs. Thus, through its cooperative mechanisms with other DGs, the ESP was aligned 
with – and able to complement – upcoming and existing EU strategies and priorities. During 
2018-2020,  47  bilateral  hearings  took  place  between  Eurostat  and  the  other  DGs.  The  ratio 
between requests approved, requests to investigate, and requests refused was approximately 
3:1:1. Eurostat was not able to meet some requests due to confidentiality restrictions, a lack 
of legal basis, a lack of resources or the expected negative impact of a request on data quality. 

Interviews  conducted  with  stakeholders  from  DGs  spoke  positively  about  the  usefulness  of 
these hearings. However, as previously highlighted, these stakeholders  also  raised issues of 
timeliness  and  data  gaps  due  to  the  lack  of  flexibility  or  long  reaction  time  of  the  ESP  in 
responding to emerging priority areas. This lack of flexibility and long reaction time was also 
raised by Member State stakeholders during interviews. 

JC 10.3 EU stakeholders considered that Eurostat coordinated effectively with other EU 

bodies 

Eurostat  set  up  a  network  of  statistical  correspondents  in  the  DGs.  It  also  created  and 
coordinated  an  inventory  catalogue  on  the  statistics  being  produced  across  the  European 
Commission. Interviewees from all DGs that were sampled highlighted Eurostat’s important 
coordination work on statistics to ensure that, within the European Commission, there was no 
duplication of the data produced. Despite this overall positive assessment, respondents to the 
targeted producer survey felt that Eurostat should have carried out more exchanges with other 
EU  bodies.  These  respondents  felt  that  more  exchanges  with  other  EU  bodies  were 
particularly  needed  to  coordinate  the  ESS  when  the  other  DGs  prepared  non-statistical 
regulations  that  required  additional  data  and  therefore  increased  the  burden  on  NSIs. 
Similarly, producers of statistics interviewed at Member-State level  also said  that there had 
been increasing demand for statistics from different DGs, with DGs not always recognising 
the cost and burden of producing statistics. This would suggest there is potential for Eurostat 
to take on an increased role in coordinating data requests among EU bodies. 

The  producers  of  European  statistics  interviewed  considered  Eurostat’s  coordination  with 
Member States and other EU bodies to be very effective. Similarly, most respondents to the 
targeted producer survey (to which 43 responded) agreed – to a moderate extent (37%) and to 
a great extent (21%) – that there were effective mechanisms in place to coordinate with other 
EU bodies. 

Beyond the European Commission, Eurostat also cooperated and coordinated with other EU 
bodies  and  agencies.  For  instance,  in  the  area  of  asylum  and  managed-migration  statistics, 
Eurostat cooperated with the European Asylum Support Office and the European Border and 
Coast  Guard  Agency  (Frontex).  This  cooperation  focused  on  developing  a  common 
methodology  and  classification  methods  to  harmonise  European  statistics  and  reduce  the 

56 

 
burden  of  statistical  production  on  Member  States.  Stakeholders  interviewed  were  very 
positive  about  both  the  level  and  the  nature  of  the  cooperation.  Similarly,  Eurostat  also 
cooperated  with  other  EU  institutions  such  as  the  European  Committee  of  the  Regions  and 
the  European  Central  Bank  (ECB).  The  ESS  and  the  European  System  of  Central  Banks 
(ESCB) also cooperated closely through the Committee on Monetary, Financial and Balance 
of Payments Statistics (CMFB)42, as well as through the European Statistical Forum (ESF). 

Overall,  the  findings  suggested  that  EU  stakeholders  considered  that  Eurostat  coordinated 
effectively with other EU bodies. However, the issue of additional burden on Member States 
was raised as a weakness due to the DGs’ individual requests for statistics. 

EQ11:  To  what  extent  were  ESP  activities  coherent  with  the  activities  of  international 
statistics organisations? 

JC 11.1 Effective mechanisms were in place to coordinate with international partners 

Eurostat  participated  in  task  forces,  expert  groups,  and  decision-making  bodies  of 
international organisations. It participated in these through both bilateral (e.g. the UNSD and 
United Nations Economic Commission for Europe (UNECE)) and multilateral settings (e.g. 
UN  Statistical  Commission).  Eurostat  also  sponsored  initiatives  such  as  the  initiative  on 
statistical data and metadata exchange (SDMX) which develops standards for the exchange of 
statistical  information.  Eurostat’s  cooperation  strategy  with  international  partners  was 
outlined in its document on international cooperation strategy, which provided thematic and 
regional  direction  and  outlined  the  mechanisms  in  place  to  coordinate  with  international 
partners.  At  working  level,  units  within  Eurostat  also  cooperated  with  units  in  relevant 
international organisations. 

Eurostat  and  its  international  partners  cooperated  to  align  their  methodologies,  agree  on 
definitions  for  individual  statistical  indicators,  and  validate  and  exchange  data.  The  aim  of 
this cooperation was to reduce the burden on Member States and to ensure the comparability 
of  the  statistics  produced.  Eurostat  currently  has  MoUs  signed  with  4  organisations  (the 
OECD,  the  ECB,  the  UN-ICSC  and  the  ISRP)  and  administrative  arrangements  with  4 
organisations (EFTA, the UNSD, the FAO and the OECD). 

In 2019, the IAS issued an audit report43 on the effectiveness of Eurostat’s cooperation with 
external  stakeholders.  In  that  report,  the  IAS  concluded  that  Eurostat  had  set  up  effective 
cooperation  arrangements  with  a  number  of  external  stakeholders.  However,  the  IAS  also 
said that the management and control systems in place lacked a clear overall policy at DG-
level, and that Eurostat needed to more effectively coordinate activities and the exchange of 
information  within  Eurostat.  The  IAS  also  made  three  recommendations  on:  (i)  further 
develop  and  formalise  cooperation  arrangements  with  Eurostat’s  external  stakeholders;  (ii) 
clearly  define  internal  roles,  responsibilities  and  information  sharing;  and  (iii)  improve  the 

42   The CMFB was established by Council Decision in 1991. The original Council Decision 91/115/EEC was 

replaced on 13 November 2006 by Council Decision 2006/856/EC (OJ, L 332, 30.11.2006, p. 21). 

43 

https://webgate.ec.testa.eu/Ares/document/show.do?documentId=080166e5c09be320&timestamp=1631267
315629. 

57 

 
                                                           
performance-management framework. Eurostat implemented an action plan to address these 
recommendations and the IAS concluded in its follow-up that the recommendations had been 
adequately and effectively implemented and were therefore closed. 

In addition to the desk review, Eurostat’s external coherence with international partners was 
also  investigated  through  the  targeted  producer  survey.  In  the  targeted  producer  survey, 
participants were asked to rate: (i) the extent to which they agreed that effective mechanisms 
were in place to coordinate with international partners; and (ii) the extent to which Eurostat 
activities  complemented  the  statistics-production  and  statistics-dissemination  activity  of 
international partners. Respondents (of which there were 43) agreed to a moderate (42%) or 
great  (30%)  extent  that  there  were  effective  mechanisms  in  place  to  coordinate  with 
international  partners.  A  similar  share  agreed  on  the  complementarity  between  Eurostat’s 
activities  and  international  partners’  statistics  production  and  dissemination  activities  (39% 
agreed to a moderate extent and 30% agreed to a great extent). 

Similarly,  producers  of  European  statistics  that  were  interviewed  considered  Eurostat’s 
coordination with international partners to be generally coherent and effective. They also said 
that this coordination had improved in recent years. Although there were still some overlaps 
between  the  requirements  of  Eurostat  and  international  organisations,  producers  said  that 
there were many initiatives aiming to prevent the duplication of work, in particular between 
Eurostat the OECD and the UNSD. 

JC 11.2 International partners considered that Eurostat activities complemented and/or 
were  not  in  contradiction  with  their  own  statistics-production  and  statistics-
dissemination activities 

International partners that were interviewed found the programme to be externally coherent 
but were generally unable to assess these questions in detail. 

Interviews  conducted  with  international  organisations  in  case  studies  provided  similar 
positive feedback. In the area of migration, the UNSD highlighted its very long-standing and 
good  cooperation with  Eurostat  which:  (i) involved the  exchange of data  and staff; and  (ii) 
contributed  to  the  overall  external  coherence  of  their  activities  by  ensuring  their 
complementarity. 

Similarly, for statistics on education, health, and transport, Eurostat, together with the OECD, 
WHO  and  UNESCO  developed  a  system  for  joint  data  collections  for  statistical  indicators. 
This enabled Eurostat and its international partners to: (i) optimise the collection of data from 
the Member States; (ii) apply common concepts and definitions; and (iii) avoid duplication of 
efforts.  Eurostat  was  also  seen  as  a  valued  member  by  international  organisations  in 
international forums, where its collaborative attitude and activities received positive feedback 
from interviewees. In addition, the statistics produced by Eurostat were seen as reliable and 
were  widely  used  to  complement  and  even  validate  the  statistics  collected  by  other 
organisations.  This  is  further  evidence  of  the  complementarity  between  the  activities  and 
statistics of Eurostat and its international partners. 

58 

 
5.5 EU added value 

The EU added value criterion looks at the value resulting from the ESP that is additional to 
the value that would have resulted from similar activities carried out only at national level or 
by  individual  Member  States.  The  EU  added  value  criterion  is  linked  to  the  principle  of 
subsidiarity, which states that the EU should only act when it can achieve better results than 
the Member States acting alone. The analysis of the criterion focused on the added value of 
the ESP to both users and producers of European statistics. 

Evaluation questions 

EQ 12: What was the EU 
added value of the 
ESP?  

Sub-questions 
 Did the ESP contribute to increased comparability of national statistics? 
 Were European statistics preferred to other competing international statistical datasets? 
 Did the ESP contribute to reduced time lag between reference period and publication data of 
statistics? 
 Did the ESP enhance the pool of resources available for production and development of new 
statistics at MS level? 
 Could the production of European statistics be successfully achieved at the MS level?  

The  evidence  collected  confirmed  the  EU  added  value  of  the  ESP.  It  suggested  that  the 
programme’s main added value was: (i) its contribution to harmonising European statistics so 
they deliver comparable statistics; (ii) providing these statistics in a single location; and (iii) 
supporting evidence-based policy making at EU and national level. The ESP produced high-
quality, comprehensive, comparable and reliable statistics across the Member States. 

In the absence of the ESP, evidence-based policy making at EU level and policy making at 
country  level  (based  on  comparative  analysis  of  statistics  across  countries)  would  be  very 
difficult (due to potential inconsistencies in the evidence). 

Users from EU institutions, Member States, and candidate countries confirmed the EU added 
value of providing European statistics in a single online location (a ‘one-stop shop’). 

The  programme  also  drew  up  a  clear  roadmap  for  statistics  production  for  producers  of 
statistics,  including  in  candidate  countries.  The  ESP  also  helped  to  strengthen  the 
international statistical community and contributed to efforts to ensure the quality of statistics 
at international level through cooperation between Eurostat and international organisations. 

EQ12: What was the EU added value of the ESP? 

JC 12.1 Users of statistics considered the ESP to be the preferred source of information 

when they compared indicators across Member States 

The findings suggested that the comparability of statistics across Member States was one of 
the  key  reasons  for  using  European  statistics,  and  that  no  other  source  could  provide  these 
statistics in a similar manner and scale. 

Users from European Commission DGs highlighted that the ESP made statistics available and 
contributed to the transparency objective on a wide range of issues. Without the ESP, users 
said  that  it  would  not  have  been  possible  to  have  comprehensive,  high-quality  and  reliable 
statistics  across  Member  States  to  support  the  European  Commission’s  policy  making 
objectives.  Even  when  users  said  that  they  also  used  other  statistics,  they  emphasised  that 
European statistics  remained their preferred source of information  because the scope of the 

59 

 
statistics  matched  exactly  the  Commission’s  (strategic)  objectives  and  covered  all  Member 
States (as the primary constituency of EU policy making). 

Individual users at Member-State level agreed with the statement of this JC. They indicated 
that  there  was  clear  EU  added  value  in  Eurostat’s  work,  and  that  this  mainly  related  to 
ensuring the comparability of statistics from the Member States and providing these statistics 
in  a  single  online  location.  Producers  of  statistics  also  shared  this  opinion.  However,  more 
advanced users pointed out that there was still more to do to further improve comparability, 
mainly by harmonising methods of collecting data across Member States. 

Eurostat  is  also  considered  a  trusted  source  of  information,  as  confirmed  by  the  USS.  For 
instance, in 2020, as in previous years, responses were overwhelmingly positive, with 96% of 
users stating that they trusted European statistics greatly or tended to trust them. 

JC  12.2  Individual  and  institutional  users’  stated  preference  for  European  statistics 

over other international statistics/datasets 

Users from the EU institutional level confirmed their preference for European statistics over 
other international statistics/datasets. There was a consensus that European statistics were the 
most relevant and of the highest quality for EU policy-making needs. Users from the Member 
States  provided  slightly  different  feedback.  European  statistics  (and  national  statistics  from 
their respective countries) were the main source of statistical information for these Member 
State  users,  but  they  indicated  that  they  also  used  other  international  sources  for  other 
purposes, including comparisons with non-EU countries. 

Most respondents to the targeted survey for users (77% of 45 responses) and to the PC (62% 
of  424  responses)  indicated  that  they  also  used  other  international  sources  of  statistics. 
Through  the  online  consultation  activities,  respondents  assessed  the  quality  of  European 
statistics in comparison to other international sources according to five principles. As shown 
in  Figure  8  below,  almost  half  of  the  respondents  reported  that  the  quality  of  European 
statistics and other international sources were similar. The proportion of respondents saying 
that  the  quality  of  European  statistics  was  higher  or  much  higher  than  that  of  other  data 
sources ranged from 29% to 51% across the five principles. 

60 

 
Figure  8:  In  your  opinion,  how  does  the  quality  of  European  statistics  compare  to  other 
international sources of data that you use 

Accessibility and clarity

16%

24%

47%

13%

Coherence and comparability

22%

29%

47%

2%

4%

Timeliness and punctuality

7%

22%

47%

18%

2%

Accuracy and reliability

16%

33%

49%

2%

Relevance

11%

29%

49%

11%

0%

Much higher

Higher

20%
Similar

40%

60%

80%

100%

Lower

Much lower

Don't know

Source: Targeted user survey (45 responses received) 

In the PC (to which there were 424 responses), respondents provided slightly more positive 
feedback,  with  the  proportion  judging  the  quality  of  European  statistics  higher  or  much 
higher  than  other  international  sources  of  data  ranging  from  34%  to  50%  across  the  five 
principles. 

The analysis suggested that there was complementarity rather than competition between the 
ESP and other international sources. 

JC 12.3 There was a recognised advantage of having a ‘one-stop shop’ for harmonised 

and validated European statistics 

This JC complements the first JC related to the ESP as the preferred source of information for 
comparisons  across  Member  States.  Throughout  the  period  evaluated,  the  main  perceived 
advantage  of  the  ESP  was  to  enable  cross-country  comparative  analyses,  which  the 
harmonisation and comparability of statistics made possible. 

In the targeted producer survey  (to which there  were 43 responses), most  producers agreed 
that  the  ESP  contributed  to  a  great  (63%  agreed)  or  moderate  (26%  agreed)  extent  to  the 
increased comparability of national statistics. 

Feedback from the European Commission DGs suggested that there was EU added value in 
having  a ‘one-stop  shop’  for  harmonised  and  validated  European  statistics.  The  ESP  was 
a central  mechanism  for  all  things  related  to  statistics  in  the  EU.  It  therefore  decreased 
transaction costs for its stakeholders while making available all the necessary data and tools. 

Finally, the existence of a ‘one-stop shop’ was also important for candidate countries. For this 
group of countries, increasing comparability and harmonisation with European statistics had 
been  one  of  the  main  goals  in  their  strategies.  Collaboration  through  the  ESP  was 
instrumental in modernising the statistical systems of these candidate countries. 

61 

 
 
JC 12.4 What would happen in the absence of the ESP? 

The EU added value of the programme is shown by the harmonised provision of comparable 
and  high-quality  data  for  EU  countries.  The  ESP  as  a  harmonised  system  with  common 
quality  standards  for  producing  statistics  is  unparalleled  in  the  world.  A  similar  level  of 
harmonisation, comparability and quality could not be achieved at Member-State level alone 
to make essential contributions to EU activities. Only a coordinated approach to developing, 
producing  and  disseminating  European  statistics,  as  ensured  through  the  programme,  could 
guarantee the required coherence and comparability of the statistics relevant for EU activities. 
At  the  same  time,  the  programme’s  financial  support  to  Member  States  helped  it  to  drive 
development,  modernisation  and  innovation  in  the  ESS.  The  programme  has  succeeded  in 
implementing  modernisation  projects  that  are  beneficial  to  all  Member  States,  which  could 
not have been achieved through uncoordinated national spending. 

In  the  absence  of  the  ESP,  evidence-based  policy  making  at  EU  level  and  country  level, 
based on the comparative analysis of statistics across countries, would be very difficult (due 
to  potential  inconsistencies  in  the  evidence).  EU  statistical  production  requires  the 
implementation of a harmonised methodology and the definition of common outputs, which 
can only be fully achieved by EU action. Delivering the same results through Member States’ 
individual  action,  without  the  overarching  and  coordinating  role  of  the  ESP  and  Eurostat, 
would have been extremely challenging, if not impossible. 

Producers  also  highlighted the usefulness  of Eurostat’s grant  scheme.  In  the absence of the 
ESP,  access  to  these  resources  would  not  be  possible.  NSIs  would  not  have  access  to 
methodological support or financial support for projects from Eurostat. Sharing best practices 
and mutual learning would be more challenging without the coordinating role of Eurostat. 

The  thematic  case  studies  confirmed  these  findings.  Recently,  the  actions  at  EU  level 
supported the response of the statistical community to the COVID-19 pandemic by providing 
methodological  support  to  statisticians  and  creating  a  European  ‘dashboard’  with  recovery 
indicators.  New  methodological  guidelines  were  issued  which  aimed  to  ensure  the 
consistency of data-collection processes related to reporting on the pandemic. 

Lastly,  and  from  a  forward-looking  perspective,  new  challenges  have  come  with  new  data 
sources,  like  privately  held  data.  These  data  sources  must  be  correctly  framed  to  ensure 
quality,  reliability  and  the  protection  of  personal  data.  This  requires  setting  up  the  proper 
legal  framework  and  addressing  any  possible  legal  barriers.  Stakeholders  expected  that  this 
would  be  more  effectively  done  at  EU  level  than  at  national  level,  especially  for  sources 
which were in essence pan-European. 

5.6 Evaluation of the points raised in Article 15 of the ESP 

The  answers  to  the  four  specific  points  referred  to  in  Article  15  of  the  ESP  Regulation  are 
based on the evidence collected and the analysis conducted for the five evaluation criteria. No 
additional EQs had to be drawn up for the first three specific topics. The fourth topic is based 
on an ad hoc sub-question, which can be linked to the effectiveness criterion. Therefore, it is 

62 

 
treated separately in this section. All the answers, with a summary of the evidence supporting 
them, are given in this section. 

5.6.1 The outcome of the reprioritisation and cost evaluation of statistical products 

Despite improvements, prioritisation  remained  a  challenge throughout  the programme. This 
resulted in increased pressure on Member-State resources. This was due in large part to the 
increase  in  data  needs  expressed  by  users  of  statistics  and  the  difficulties  in  identifying 
statistics that have become less relevant over time. To solve the issue, a review of statistical 
requirements in existing domains of European statistics and a new prioritisation mechanism 
are being implemented in the new statistical programme for 2021-2027. The new mechanism 
is  based  on  four  elements:  a  multiannual  action  plan,  reviews,  user  dialogue,  and 
resources/funding. 

The  assessment  of  costs  for  the  ESS  showed  that  the  total  costs  of  producing  statistics 
decreased slightly between 2013 and 2020, as did the number of FTEs – although producers 
met an increasing number of statistics requests. 

The  detailed  information  leading  to  the  conclusions  above  can  be  found  in  Section  5.3  on 
efficiency, in the analysis of EQs 6 and 7 and in particular in JC 6.1, 6.2, 6.4, 6.5, 7.1 and 7.2. 

5.6.2 Actions taken by the ESS to reduce the production costs for Member States and to 
limit the overall burden stemming from the statistical projects and fields covered by the 
programme 

The  quantitative  data  show  that  both  staff  numbers  and  the  cost  of  producing  statistics  fell 
while  the  number  of  datasets  being  produced  increased.  This  is  evidence  that  the  actions 
taken by the ESS were effective in reducing production costs for Member States and limiting 
the  overall  burden  stemming  from  the  statistical  projects  and  fields  covered  by  the 
programme.  Among  the  most  important  actions  taken  were:  (i)  the  implementation  of 
framework  statistics  regulations;  (ii)  increased  use  of  administrative  and  register  data;  (iii) 
increased use of estimation methods; and (iv) the application of microdata exchanges. 

However, qualitative feedback provided a more nuanced perspective, highlighting limitations 
in  both  priority  setting  and  the  identification  of  statistics  which  have  become  less  relevant 
over time. In addition, concerns about shrinking budgets and staff numbers, coupled with the 
reported increase in requests for new statistics, suggested that issues of production cost and 
burden may emerge in the future. 

The  detailed  information  leading  to  the  conclusions  above  can  be  found  in  Section  5.3  on 
efficiency, in the analysis of EQs 6 and 7, and in particular in JC 6.4, 6.5, 7.1 and 7.2. 

5.6.3  Progress  on  rendering  access  to  official  statistics  easier  and  more  user-friendly, 
including the provision of data on the Eurostat website 

The ESP helped make access to official statistics easier and more user-friendly, but there was 
still scope for further improvement. 

Eurostat’s website was completely redeveloped at the beginning of the programme. The new 
website was launched in December 2014 and it has been continuously improved since, with 

63 

 
another  new  update  that  will  go  live  soon.  Since  the  beginning  of  the  ESP,  Eurostat  has 
developed  and  added  many  new  electronic  and  interactive  publications,  visualisation  tools, 
mobile apps and tools  for data extraction.  These tools  present  data from  different  statistical 
themes in an attractive and easy-to-understand way for everyone to explore and can be easily 
accessed on Eurostat’s website. 

Adding modern ways of communication, Eurostat has also been active on social media, with 
its three corporate social media accounts on Twitter, Facebook and Instagram. 

The provision of data on the Eurostat website was prioritised during the redevelopment of the 
website.  However,  opinions  about  the  user-friendliness  and  ease  of  access  to  these  data 
differed  depending  on  the  category  of  users.  Although  data  and  statistics  were  easily 
accessible  to  proficient  users  from  the  Commission,  access  was  more  challenging  for  less 
experienced users from the general public. Progress was made in making statistical data easy 
to  retrieve  and  convert  for  practical  use,  including  through  graphs  and  maps.  However,  the 
main weakness of the Eurostat website was its complexity, which led to difficulties in finding 
the right data and statistics. Nevertheless, dedicated Eurostat publications provided data in a 
clear and accessible manner. 

The  detailed  information  leading  to  the  conclusions  above  can  be  found  in  Section  5.2  on 
effectiveness, in the analysis of EQ 4 and in particular in JC 4.2. 

5.6.4  Progress  on  improving  data  availability,  including  social-economy  activities,  and 
on the Europe 2020 indicators 

increased 

From 2013 to 2020, the ESP included activities to develop new statistics and contributed to 
improving data availability, including on both social-economy activities and the Europe 2020 
indicators.  On  social-economy  activities,  data  and  statistics  became  more  available  in  the 
the 
areas  of  population,  social  conditions  and  migration.  Eurostat  also 
disaggregation of its data by migratory status. Despite these improvements, gaps remained in 
the  territorial  disaggregation  of  social  statistics.  On  the  Europe  2020  indicators,  Eurostat 
continuously provided comparable, reliable, and timely statistical information. Eurostat also 
continued  its  work  to  develop  and  produce  these  indicators  throughout  the  programme. 
Europe 2020 indicators are only available at the EU and country levels, because there is no 
territorial disaggregation of the targets that have been set. Nevertheless, some of the data used 
for  these  indicators  are  sometimes  available  at  regional  level.  The  set  of  Europe  2020 
indicators – used to monitor the Europe 2020 strategy for growth and jobs for 2010-2020 – 
had  been  updated  yearly  and  was  available  until  2021.  The  Europe  2020  strategy  has  now 
reached  the  end  of  its  life  cycle.  Moreover,  the  EU  aggregate  (28  countries  including  the 
United  Kingdom),  for  which  the  Europe  2020  targets  were  set,  no  longer  exists  in  official 
statistics. For these two reasons, the Europe 2020 indicators were removed from the Eurostat 
reference  database  and  archived.  However,  all  five  areas  covered  by  the  Europe  2020 
indicators remain part of the EU indicator set on the SDGs and will continue to be updated 
under the heading of ‘sustainable development indicators’. 

64 

 
The findings suggest that ESP activities resulted in an improvement of data availability for all 
themes.  Users  and  producers  of  European  statistics  that  were  interviewed  confirmed  this 
point. They assessed data availability positively and noticed an improvement in that respect.  

Social-economy activities 

43% of the respondents to the PC (to which there were 424 responses) specifically reported 
an  improvement  in  data  availability  on  population  and  social  conditions.  Stakeholders 
interviewed  as  part  of  the  asylum  and  managed-migration  case  study  further  corroborated 
this.  These  stakeholders  were  satisfied  with  the  improvement  in  data  availability  on  social 
(migration-related) activities and the increased disaggregation of data by migratory status. 

Nevertheless,  stakeholders interviewed as part of the territorial  case study  said  that, despite 
improvements in the availability of data that were disaggregated by territory, there remained 
gaps in social statistics. 

Europe 2020 indicators 

An  audit  was  carried  out  in  2015  on  the  support  given  by  Eurostat  to  the  Europe  2020 
strategy and the new Commission priorities. The audit aimed to assess whether Eurostat had 
put in place an efficient and effective process to provide up-to-date statistical data, including 
in  the  areas  covered  by  the  Europe  2020  strategy,  to  help  to  monitor  progress  towards  the 
related  targets44.  The  audit’s  main  recommendation  was  on  other  statistics  (not  Eurostat’s) 
used  by  the  DGs  to  demonstrate  progress  made  in  achieving  the  Europe  2020  targets.  This 
suggests that progress on improving data availability on the Europe 2020 indicators was not 
an issue. 

The 2013-2019 AARs all indicated that Eurostat continuously provided comparable, reliable, 
and timely statistical information supporting the monitoring of progress towards an evidence-
based implementation of the Europe 2020 strategy45. In the same period, these data were also 
available in the flagship publication Smarter, greener, more inclusive? Indicators to support 
the Europe 2020 strategy46. Throughout the ESP, Eurostat made progress on improving data 
availability  as  work  continued  on  developing  and  producing  high-quality  indicators  for 
Europe 2020. Some of these improvements in data availability are set out in the bullet points 
below. 

 

 

In 2014, major achievements concerned, among other things, the further development 
of the Europe 2020 indicators for education and training. 

In 2015, Eurostat accomplished major achievements on the Europe 2020 targets in the 
area  of  sectoral  and  regional  statistics.  Eurostat  also  made  available  timely  Europe 
2020 indicators on education and life-long learning. 

44   Commission  Staff  Working  Document  –  Annual  Internal  Audit  Report  for  2015:  summary  of  the 
engagements finalised by the IAS within the 2015 audit plan accompanying the document ‘Report from the 
Commission to the European Parliament and Council Annual Report on internal audits carried out in 2015’ 
(Article 99(5) of the Financial Regulation) (SWD/2016/0322 final, 30.9.2016). 

45   European Commission, 2013/2014/2015/2016/2017/2018/2019 Annual Activity Report – Eurostat. 
46   Eurostat,  Smarter,  greener,  more  inclusive?  Indicators  to  support  the  Europe  2020  Strategy  – 

2013/20142015/2016/2017/2018/2019 edition. 

65 

 
                                                           
 

 

In 2016, Eurostat updated the headline target indicators for Europe 2020 on sectoral 
statistics  with  new  data  and  disseminated  them  through  its  online  database.  The 
flexible  dashboard  and  other  interactive  visual  tools  in  the  section  on  Eurostat’s 
website dedicated to the Europe 2020 targets also included these new data. 

In 2019, the provision of additional breakdowns and the revision of the methodology 
and  compilation  techniques  further  improved  the  quality  of  the  Europe  2020 
indicators: (i) a new child deprivation indicator was produced; (ii) revised inequality 
indicators were approved; (iii) the ‘low work intensity’ indicator was revised; (iv) the 
timeliness of social data was improved; and (v) a news release was published on the 
early results of indicators on material deprivation. 

The case studies carried out as part of this evaluation confirmed the data availability of the 
Europe  2020  indicators.  Stakeholders  interviewed  on  business  statistics  stated  that  Eurostat 
provided extensive indicator sets to contribute to assessing the Europe 2020 targets. Asylum 
and managed-migration data also included migrant-integration statistics, which were part of 
the regional labour-market indicator. These statistics were broken down by region and degree 
of  urbanisation  and  looked  at  activity  rate,  employment  and  unemployment  rate.  They 
supported  monitoring  of  the  headline  target  indicator  on  the  employment  rate  for  Europe 
2020.  Eurostat  regularly  published  some  of  the  data  used  for  Europe  2020  indicators  at 
regional level in its regional yearbook. Statistics with a territorial dimension supported a wide 
range  of  policy  measures  set  out  in  the  Europe  2020  strategy,  such  as  urban  renewal, 
increasing  education  opportunities,  crime  prevention,  social  inclusion,  or  environmental 
protection47. At the same time, the composition of some indicators made it difficult to make 
them  available  at  regional  level  (for  example,  indicators  focusing  on  narrow  age  groups  or 
measuring  the  consumption  of  renewable  energy  at  the  regional  or  territorial  level  are  not 
feasible).  

6.  CONCLUSIONS 

The  European  statistical  programme  2013-2020  was  created  by  Regulation  (EU)  99/201348 
for  the  years  2013-2017.  It  was  extended  by  Regulation  (EU)  2017/195149  to  cover  the 
remaining period of the previous EU multiannual financial framework, i.e. 2018-2020. 

The general objective of the ESP was for the ESS to continue to be the leading provider of 
high-quality statistics on Europe. As a spending programme, the ESP constituted the overall 
framework  for  developing,  producing  and  disseminating  European  statistics.  The  ESP 
therefore  provided  the  financing  for  developing  and  maintaining  Eurostat’s  statistical 
infrastructure and the ESS as managed by Eurostat. The ESP also provided financial support 
to Member States. 

47   Eurostat, Eurostat regional yearbook 2018 edition, available at: 

https://ec.europa.eu/eurostat/documents/3217494/9210140/KS-HA-18-001-EN-N.pdf. 

48   OJ L 39, 9.2.2013, p. 12. 
49   OJ L 284, 31.10.2017, p. 1. 

66 

 
                                                           
The requirement to carry out a final evaluation and present a report on the ESP is enshrined 
in  Article  15  of  the  amended  Regulation,  which  also  mentions  four  specific  aspects  to 
evaluate. This evaluation also serves to fulfil the requirements of the Commission’s push for 
better regulation and so it has to follow the better regulation guidelines for evaluations. The 
evaluation therefore covers the five compulsory evaluation criteria (effectiveness, efficiency, 
relevance, coherence and EU added value). 

Two  previous  mid-term  evaluations  of  the  ESP4;5  covered  the  implementation  of  the 
programme in  2013-2017, while a progress  report9 assessed the progress  made in  2018 and 
the first half of 2019. The present evaluation takes into account the findings and conclusions 
of  those  other  evaluations,  but  covers  the  implementation  period  of  the  programme  in  its 
entirety (2013-2020). 

A  contractor,  Tetra  Tech  International  Development,  was  selected  (via  an  open  call  for 
tenders)  to  support  the  evaluation  process.  Having  an  independent  opinion  on  the  work 
carried  out  by  Eurostat  was  deemed  very  important.  The  contractor’s  work  included:  (i) 
designing the evaluation; (ii) preparing and executing the stakeholder consultation; and (iii) 
writing an evaluation report. The contractors’ report was used as a base to prepare the SWD. 

The COVID-19 crisis prevented the contractor’s  team from collecting data face-to-face. All 
activities took place remotely instead. However, this did not influence the quality of the work 
as the overall the level of stakeholder engagement achieved was high. 

An ISG was set up for the project, including members from all directorates of Eurostat and 
from many other DGs. The ISG followed and steered the entire evaluation. 

6.1.  Relevance 

The  ESP  provided  the  framework  for  developing,  producing,  and  disseminating  European 
statistics,  setting  out  the  objectives  of  the  activities  planned  for  2013-2020.  It  financed  the 
development  and  maintenance  of  both  Eurostat’s  statistical  infrastructure  and  the  ESS.  The 
objective of the ESP was therefore to both: (i) fulfil the needs of Eurostat, NSIs and ONAs 
(as producers of European statistics); and (ii) serve the needs of a wide range of users of these 
statistics. 

The  replies  to  the  EQs  indicate  that  the  ESP  was  appropriately  designed  to  satisfy  the 
stakeholders’ needs. The replies also  indicate that, throughout  its entire duration, the ESP’s 
activities  were  carried  out  to  continue  to  satisfy  these  changing  needs.  At  the  same  time, 
although the ESP made it possible to implement appropriate activities to meet its objectives, 
the analysis showed that these activities were not enough to deliver all the statistics that users 
would wish. However, some of the thematic areas that users identified as missing go beyond 
the scope of the ESP – so non-delivery in these areas cannot be considered as failures in the 
implementation of the programme. 

The  high  relevance  of  the  ESP  is  clearly  demonstrated  by  the  number  of  data  extractions 
made by external users from Eurostat reference databases, which greatly increased since 2014 
from 6 813 000 extractions to 21 480 000 in 2020 (+315%). That is further confirmed by the 
increase in the number of users accessing the Eurostat website, estimated by the number of 

67 

 
distinct  IP  addresses  visiting  the  website,  which  went  from  3  441  157  visits  in  2014  to 
11 731 274 in 2020 (+240%).  

During the lifetime of the programme, Eurostat increased its presence on social media and its 
numbers  of  followers  grew  constantly.  Eurostat  opened  its  Facebook  account  on 
10 January 2017  and  its  Instagram  account  on  4 May 2020.  Eurostat’s  number  of  followers 
on  Twitter  grew  from  25  000  in  2013  to  164  000  in  2020,  by  which  time  it  had  reached 
62 000 followers on Facebook and 9 500 on Instagram. 

There was some tension between continuously increasing user needs and limitations faced by 
producers  in  keeping  up  with  new  demands  under  an  already  ambitious  programme.  This 
meant  that  a  balance  needed  to  be  struck  between  producer  capacity  and  user  requests.  In 
addition, needs related to adaptation to technological advances such as big data became more 
prominent during the programme, as reflected in the extension of the ESP. These increasingly 
prominent  needs  were  translated  into  specific  activities,  mainly  as  part  of  the  ESS  Vision 
2020 implementation projects. Similar activities will be continued in the next programme, the 
results of which have the potential to provide solutions to some of the problems the ESS still 
faces in terms of resources and costs. 

6.2.  Effectiveness 

The  evaluation  findings  show  that  the  ESP  was  effective  in  delivering  on  its  objectives 
despite remaining weaknesses in the timeliness and completeness of European statistics. The 
percentage of planned outputs that were achieved under the ESP objectives 1 and 4 for the 
production of statistics was constantly  above 92%. They were constantly  above 95% in  the 
second  half  of  the  programme,  with  only  3%  or  fewer  of  outputs  cancelled  or  not  fully 
achieved. 

Between 2013 and 2020, Eurostat provided high-quality statistics which were used by a wide 
range  of  users  for  a  wide  range  of  purposes.  These  purposes  include  supporting  the 
development,  monitoring  and  evaluation  of  policies  at  the  EU  and  Member-State  level. 
During  that  period,  the  ESP  also  increased  the  availability  of  its  data  and  statistics  and,  to 
some  extent,  introduced  efficiency  gains  in  the  production  of  its  statistics.  The  quantity  of 
statistics  and  long-time  series50  published  by  Eurostat  increased  from  328  million  and  35 
million respectively to 446 million and 55 million. 

Several  projects  were  launched  in  response  to  new  challenges  faced  by  European  statistics. 
These  challenges  included  implementing  the  ESS  Vision  2020,  strengthening  the  European 
statistical infrastructure, and making better use of big data and shared infrastructure.  

Eurostat  also  effectively  developed  and  strengthened  partnerships  with  its  members  within 
the ESS as well as with partners beyond the ESS – in particular international organisations. 
However, more needs to be done, in particular to: (i) get access to privately held data; and (ii) 
create  partnerships  with  private  organisations.  Work  towards  both  of  these  goals  started  in 
2020 and is being addressed in the next programme.  

50   Long time series are defined in this case as those covering 10 or more consecutive years. 

68 

 
                                                           
Beyond  the  achievements  of  its  objectives,  the  ESP  was  also  effective  in  making  access  to 
official statistics easier and more user-friendly between 2013 and 2020. 

6.3.  Efficiency 

Under the efficiency criterion, the evaluation focused on: (i) the costs of producing European 
statistics; (ii) the trends in these costs; and (iii) the burden of producing these statistics faced 
by the NSIs and ONAs. Three general issues were assessed to find out whether the ESP was 
being run efficiently. The first issue related to whether the programme ensured the best use of 
available  financial  and  human  resources.  The  second  concentrated  on  the  costs  and  burden 
involved in producing European statistics in the ESS and how these costs and burdens have 
developed. The third issue was to assess how efficient the ESP process was for reporting and 
monitoring. 

The  analysis  of  the  evidence  suggests  that  the  programme  has  been  efficient.  The  ESP 
demonstrated  efficient  use  of  both  financial  and  human  resources  while  delivering 
high-quality European statistics. Moreover, the productivity of statistics production increased 
during the period evaluated. Staff numbers at Eurostat decreased by 7% (52 persons) in FTEs 
during  the  implementation  of  the  ESP.  In  the  same  period,  the  number  of  annual  datasets 
published by Eurostat increased by 18%. This means an increase in statistics productivity of 
Eurostat staff by more than 20% during this time. Staff numbers also fell across the ESS as a 
whole,  going  from  49  200  FTEs  in  2013  to  42  500  in  2020.  Factors  underpinning  these 
developments  in  productivity  were  good  governance,  good  management,  and  effective 
monitoring mechanisms. 

The IAS carried out an audit on Eurostat’s quality management of statistical processes. The 
audit  found  aspects  to  improve,  and  the  IAS  made  two  recommendations.  Eurostat 
implemented  an  action  plan  to  address  these  recommendations.  The  IAS  concluded  in  its 
follow-up  on  18 January 2021  that  the  first  recommendation  had  been  adequately  and 
effectively  implemented  and  was  therefore  closed.  The  second  recommendation  will  be 
implemented by 31 December 2021as set out in the action plan. 

There were concerns about the costs for Member States and the administrative burden placed 
on data providers. These concerns arose because of the need to produce an increasing volume 
of  statistics,  which  made  it  necessary  to  increase  the  efficiency  of  statistics  production. 
Several  initiatives were  undertaken to  respond to this challenge, including:  (i) the  activities 
within the ESS Vision 2020; (ii) an initiative to modernise business statistics resulting in the 
European Business Statistics Regulation; (iii) an initiative to improve social statistics with the 
Integrated  European  Social  Statistics  Regulation;  (iv)  an  initiative  to  improve  agriculture 
statistics  with  the  Integrated  Farm  Statistics  Regulation;  and  (v)  work  to  strengthen 
partnerships within the ESS. 

6.4.  Coherence 

The evaluation examined both the internal and external coherence of the ESP and the findings 
suggested that the ESP was internally and externally coherent. 

69 

 
No overlaps  or inconsistencies were identified at the programme  level  – either between the 
objectives or among the activities listed within the regulations and planning documents of the 
ESP (internal coherence). The evaluation identified various governance bodies and advisory 
boards that had a coherent mandate. The work of these bodies and boards contributed to the 
internal coherence of the ESP by ensuring that its activities were in line with its objectives. 
Furthermore, interviewees involved in producing European statistics highlighted the internal 
coherence of the ESP.  

The work performed under the ESP was in agreement with the priorities of the Commission. 
Relevant  European  statistics  corresponded  to  the  European  Commission’s  10  general 
objectives/priorities, and the sub-priority areas of the ESP were in line with the priority areas 
of the Juncker Commission. 

Eurostat  effectively  cooperated  with  EU  bodies  and  agencies,  as  well  as  international 
organisations.  This  cooperation  resulted  in  the  external  coherence  of  the  ESP  by  ensuring 
synergies with: (i) wider EU objectives and needs; and (ii) international statistical activities. 
This  cooperation  also  ensured  the  development  of  comparable  and  harmonised  European 
statistics at the regional and international level.  

The IAS carried out two audits, one on the production process and the quality of statistics not 
produced  by  Eurostat,  and  one  on  the  effectiveness  of  Eurostat’s  cooperation  with  external 
stakeholders.  Both  audits  found  aspects  to  improve,  and  the  IAS  therefore  made  some 
recommendations.  Eurostat 
to  address  both  sets  of 
implemented  an  action  plan 
recommendations, and the IAS concluded in their follow-ups that the recommendations had 
been adequately and effectively implemented and were therefore closed. 

However, some weaknesses which could potentially affect the external coherence of the ESP 
were identified in the evaluation, namely: (i) the lack of flexibility of European statistics to 
respond to emerging needs (because this lack of flexibility might cause a misalignment with 
other EU strategies should the needs not be covered ); (ii) the burden on Member States when 
replying  to  individual  DG  requests  for  statistics;  and  (iii)  the  necessity  of  getting  access  to 
privately held data and creating partnerships with private organisations (for which activities 
began in 2020). Eurostat has begun to tackle these weaknesses and they will continue to be 
addressed in the next programme. 

6.5.  EU added value 

The  evidence  collected  confirmed  the  EU  added  value  of  the  ESP.  It  suggested  that  the 
programme’s main added value was: (i) its contribution to harmonising European statistics so 
Member  States  deliver  comparable  statistics;  (ii)  providing  these  statistics  in  a  single 
location;  and  (iii)  supporting  evidence-based  policy  making  at  EU  and  national  level.  The 
ESP  produced  high-quality,  comprehensive,  comparable  and  reliable  statistics  across  the 
Member States. 

In the absence of the ESP, evidence-based policy making at EU level and policy making at 
country  level  (based  on  comparative  analysis  of  statistics  across  countries)  would  be  very 
difficult (due to potential inconsistencies in the evidence). 

70 

 
Users from EU institutions, Member States and candidate countries confirmed the EU added 
value of providing European statistics in a single online location (a ‘one-stop shop’). 

The  programme  also  drew  up  a  clear  roadmap  for  statistics  production  for  producers  of 
statistics,  including  in  candidate  countries.  Through  cooperation  between  Eurostat  and 
international  organisations,  the  ESP  also  helped  to  strengthen  the  international  statistical 
community and helped efforts to ensure the quality of statistics at the international level. 

6.6. 

Specific points of Article 15 

The conclusions of the evaluation for the four specific aspects mentioned in Article 15 of the 
amended ESP Regulation are summarised here below. 

a)  The outcome of the reprioritisation and cost evaluation of statistical products 

Despite  improvements,  prioritisation  remained  a  challenge  throughout  the  programme. 
This resulted in increased pressure on Member State resources. This was due in large part 
to  the  increase  in  data  needs  expressed  by  users  of  statistics  and  the  difficulties  in 
identifying  statistics  that  have  become  less  relevant  over  time.  To  solve  the  issue,  a 
review  of  statistical  requirements  in  existing  domains  of  European  statistics  and  a  new 
prioritisation  mechanism  are  being  implemented  in  the  statistical  programme  for  2021-
2027 set out by the SMP. The new mechanism is based on four elements: a multiannual 
action plan, reviews, user dialogue, and resources/funding. 

As detailed in b) below, the assessment of costs for the ESS showed that the total costs of 
producing statistics decreased slightly between 2013 and 2020 as did the number of FTEs 
– although producers faced an increasing number of requests for statistics. 

b)  Actions  taken  by  the  ESS  to  reduce  implementation  and  production  costs  for 
Member  States  and  to  limit  the  overall  burden  stemming  from  the  statistical 
projects and fields covered by the programme 

Staff numbers and the cost of producing statistics fell while the number of datasets being 
produced increased. This is evidence that the actions taken by the ESS were effective in 
reducing  production  costs  for  Member  States  and  limiting  the  overall  burden  stemming 
from  the  statistical  projects  and  fields  covered  by  the  programme.  Among  the  most 
important actions taken were: (i) the implementation of framework statistics regulations; 
(ii)  increased  use  of  administrative  and  register  data;  (iii)  increased  use  of  estimation 
methods; and (iv) the application of microdata exchanges. 

However,  qualitative  feedback  highlighted  limitations  in  priority  setting  and  the 
identification  of  statistics  which  have  become  less  relevant  over  time.  In  addition, 
concerns  about  both  falling  staff  numbers  and  the  increase  in  data  needs  expressed  by 
users  seeking  new  statistics  suggested  that  issues  of  production  cost  and  burden  may 
emerge in the future. 

c)  Progress  on  rendering  access  to  official  statistics  easier  and  more  user-friendly, 

including the provision of data on the Eurostat website 

71 

 
The  ESP  helped  make  progress  on  making  access  to  official  statistics  easier  and  more 
user-friendly, but there was still scope for further improvement. 

Eurostat’s  website  was  completely  redeveloped  in  2014  and  has  been  continuously 
improved since. Since the beginning of the ESP, Eurostat has developed and added many 
new electronic and interactive publications, visualisation tools, mobile apps and tools for 
data extraction. These tools present data from different statistical themes in an attractive 
and  easy-to-understand  way  for  everyone  to  explore  and  can  be  easily  accessed  on 
Eurostat’s website. 

Adding modern ways of communication, Eurostat has also been active on social media, 
with its three corporate social media accounts on Twitter, Facebook and Instagram. 

The provision of data on the Eurostat website was prioritised during the redevelopment of 
the website. However, different users gave differing views on how easy and user-friendly 
it  was  to  access  these  data.  Although  data  and  statistics  were  easily  accessible  to 
proficient users from the Commission, access was more challenging for less experienced 
users from the general public, due to the volume and complexity of the website’s content. 

d)  Progress  on  the  improvement  of  data  availability,  including  on  social-economy 

activities and on the Europe 2020 indicators 

increased 

From 2013 to 2020, the ESP included activities to develop new statistics and contributed 
to  improving  data  availability,  including  on  social-economy  activities  and  the  Europe 
2020 indicators. On social-economy activities, more data and statistics became available 
the 
on  population,  social  conditions,  and  migration.  Eurostat  also 
disaggregation  of  its  data  by  migratory  status.  Despite  these  improvements,  gaps 
remained  in  the  territorial  disaggregation  of  social  statistics.  On  the  Europe  2020 
indicators,  Eurostat  continuously  provided  comparable,  reliable,  and  timely  statistical 
information.  Eurostat  also  continued  its  work  to  develop  and  produce  these  indicators 
throughout the programme. The Europe 2020 indicators are only available at the EU and 
country levels because there is no territorial disaggregation of the targets that have been 
set.  Nevertheless,  some  of  the  data  used  for  these  indicators  are  available  at  regional 
level. The set of Europe 2020 indicators – used to monitor the Europe 2020 strategy for 
growth  and  jobs  for  2010-2020  –  was  updated  yearly  and  was  available  for  reference 
periods until 2021. The Europe 2020 strategy has now reached the end of its life cycle. In 
addition, the EU  aggregate (28 countries  including the United  Kingdom), for  which the 
Europe 2020 targets were set, no longer exists in official statistics. For these two reasons, 
the  Europe  2020  indicators  were  removed  from  the  Eurostat  reference  database  and 
archived. However, all five areas  covered by the Europe 2020 indicators remain part of 
the EU indicator set on the SDGs and will continue to be updated under the heading of 
‘sustainable development indicators’. 

72 

 
Annex 1: Procedural information 

1.  LEAD DG, DeCIDE PLANNING/CWP REFERENCES 

This evaluation staff working document was prepared by the evaluation team in Unit A2 
‘Legal affairs; Document management’ of Eurostat. 

The  purpose  of  the  evaluation  was  to  assess  the  implementation  of  the  ESP  (Decide 
planning reference: PLAN/2020/8378). 

2.  ORGANISATION AND TIMING 

The work on the evaluation started in the 3rd quarter of 2020. As the main objective of 
the ESP is the production of high-quality statistics on Europe, which serve the needs of 
many DGs, the work was supervised by a large ISG. It included representatives from the 
following DGs: 

  Eurostat 
  Joint Research Centre 
  Legal Service 
  Secretary-General 
  DG Agriculture and Rural Development 
  DG Budget 
  DG Climate Action 
  DG Communication 
  DG Communication Networks, Content and Technology 
  DG Competition 
  DG Economic and Financial Affairs 
  DG Education and Culture 
  DG Employment, Social Affairs and Inclusion 
  DG Energy 
  DG Environment 
  DG Financial Stability, Financial Services and Capital Markets Union 
  DG Health and Food Safety 
  DG Internal Market, Industry, Entrepreneurship and SMEs 
  DG International Partnerships 
  DG Justice and Consumers 
  DG Maritime Affairs and Fisheries 
  DG Migration and Home Affairs 
  DG Mobility and Transport 
  DG Neighbourhood and Enlargement Negotiations 
  DG Regional and Urban Policy 

73 

 
 
  DG Research and Innovation 
  DG Taxation and Customs Union 
  DG Trade 

Meetings of the ISG were held on: 

21/09/2020, 10/11/2020, 06/05/2021, 06/07/2021, 01/09/2021. 

3.  EXCEPTIONS TO THE BETTER REGULATION GUIDELINES 

There were no exceptions to the better regulation Guidelines. 

4.  EVIDENCE, SOURCES AND QUALITY 

The list of sources used to perform the evaluation can be found in the contractor’s final 
report  (Annex  6  of  the  contractor’s  final  report;  Annex  4  to  this  document)  for  the 
desktop  review  and  in  the  contractor’s  synopsis  report  (Annexes  3,  4  and  5  of  the 
contractor’s synopsis report) for the stakeholder consultation. 

5.  EXTERNAL EXPERTISE 

A  contractor,  Tetra  Tech  International  Development,  was  charged  with  writing  an 
independent evaluation report, to support the preparation of the Commission evaluation 
staff working document. 

The contract started on 10 September 2020 and the consultant delivered its final report on 
23 July 2021. 

The final report is annexed to this document. 

74 

 
Annex 2: Stakeholder consultation 

1. 

Introduction 

This  stakeholder  consultation  was  conducted  by  an  external  contractor,  Tetra  Tech 
International  Development,  in  the  context  of  the  final  evaluation  of  the  European 
Statistical Programme 2013-2020. The report has two main parts: one section, Section 2, 
describing  the  stakeholder  consultation  strategy,  and  one  section,  Section  3,  presenting 
the overarching analysis of the different consultation activities. Further information can 
be found in the synopsis report of the contractor (contract deliverable D4). 

2. 

Consultation strategy 

2.1.  Consultation objectives 

The  contractor  prepared  and  carried  out  several  consultation  activities  designed  to 
support  the  ongoing  evaluation  of  the  European  Statistical  Programme  (‘the  ESP’,  also 
‘the  programme’  thereafter).  The  objective  was  two-fold:  to  gather  stakeholders’  views 
with  respect  to  a  range  of  evaluation  criteria  and  then  to  compare  and  integrate  the 
evidence collected through other data collection activities and to gather primary evidence 
to address any data gaps identified following the desk review. 

2.2. 

Identification of stakeholders 

As  part  of  the  consultation  strategy,  the  contractor  conducted  a  stakeholder  mapping 
exercise to identify key ESP stakeholders, both producers and users of European statistics 
(as presented in the Inception Report and then revised in the D2 Stakeholder Consultation 
Report). In line with the Better Regulation guidelines, the mapping exercise involved two 
distinct activities: 

1.  Identification of stakeholder categories relevant for or interested in the ESP; 

2.  Prioritisation of stakeholder categories to engage with in line with their level of 

interest in or influence on the ESP. 

Based on the mapping exercise, the contractor designed a stakeholder consultation plan 
which then formed the backbone of all stakeholder consultation activities. The activities 
included: 

i. 

interviews, 

Stakeholder 
the  overall 
programme  evaluation,  as  well  as  interviews  in  the  framework  of  Member 
States and country case studies 

interviews  feeding 

including 

into 

ii.  A public consultation (PC) 

iii. 

Targeted surveys of producers and users 

75 

 
 
 
 
2.3.  Consultation methods and tools 

Scoping Interviews 

A  series  of  eight  scoping  interviews  were  conducted  early  on  in  the  evaluation  with 
representatives of Eurostat and other European Commission Directorates General (DGs). 
These  interviews  helped  the  contractor  to  develop  and  validate  the  draft  intervention 
logic  and  evaluation  questions;  establish 
the  evaluation  baseline;  develop 
a comprehensive  understanding  of  the  programme  context;  and  get  suggestions  for 
potential documents, interviewees and case studies that in turn informed the refinement 
of  the  evaluation  methodology.  The  interviews  took  approximately  one  hour  each  and 
were  semi-structured,  allowing  conversations  to  follow  up  on  the  interviewees’ 
background and expertise. 

Stakeholder interviews 

Stakeholder  interviews  allowed  the  contractor  to  collect  specific  information  on  the 
implementation of the ESP, as perceived by the participants in these activities (both users 
and producers of the statistics) but also to feed into the thematic and Member State case 
studies.  The  contractor  carried  out  a  general  interview  programme  of  50  in-depth 
interviews  with  key  stakeholders,  including  representatives  from  Eurostat  and  client 
DGs;  a  sample  of  other  EU  and  international  bodies,  candidate  and  EFTA  countries 
(European  Free  Trade  Area),  non-public  institutional  users,  as  well  as  producers  and 
users in five Member States, namely Estonia, Germany, Poland, Spain and Sweden. 

Due to the COVID-19 pandemic, the contractor conducted all interviews remotely (they 
took approximately one hour each), using a semi-structured format. 

Surveys of producers and institutional users 

The stakeholder consultation strategy included two separate, targeted, online surveys: 

  A  survey  of  producers:  this  online  survey  was  primarily  focused  on  NSIs 
(National  Statistical  Institutes)  and  ONAs  (Other  National  Authorities)  but  also 
included other individuals and organisations who were able to share a perspective 
on the effectiveness of ESP implementation and operations. The survey generated 
43 responses. 

  A  survey  of  users  of  European  statistics:  this  online  survey  was  primarily 
focused  on  institutional  users,  in  particular  EU  institutions  and  umbrella 
organisations,  but  not  exclusively.  The  contractor  received  45  responses  to  this 
survey. 

Implementing a two-pronged approach allowed the team to target questions specifically 
at  the  supply  side  (NSIs,  ONAs  and  other  ESS  (European  Statistical  System) 
stakeholders involved in production of European statistics) and demand side (institutional 
and other users) for the statistics and consult a broader range of stakeholders than what 
would have been possible through in-depth interviews. 

The  questionnaires  were  administered  through  EU  Survey  and  disseminated  to 
stakeholders  by  e-mail  with  an  accompanying  cover  text  explaining  their  purpose  and 

76 

 
how  the  results  were  to  be  used.  The  producer  survey  was  launched  on  17  December 
2020  and  remained  online  until  5  February  2021.  The  user  survey  was  launched  on  17 
December 2020 and remained online until 8 February 2021. 

The public consultation 

In  order  to  facilitate  a  broad  range  of  views,  the  questions  posed  in  the  public 
consultation were general questions about the statistics provided by Eurostat, in contrast 
to more specific questions about the ESP in the targeted surveys. This was done to make 
sure that the PC complemented, rather than substituted, the targeted consultations. 

The contractor prepared the questionnaire used in the public consultation in English, with 
the  translation  into  French  and  German  undertaken  by  European  Commission  services. 
The  three  versions  were  implemented  in  EU  Survey,  and  the  PC  was  published  on  16 
December  2020  and  remained  open  until  10  March  2021.  It  generated  424  responses. 
One  response  was  discarded  for  not  having  respected  the  rules  for  feedback  and 
suggestions. 

The  contributions  received  in  the  context  of  the  public  consultation  published  on  the 
‘Have  your say’ web portal cannot be regarded as the official position of the European 
Commission and its services, and therefore do not bind the European Commission. The 
contributions cannot be considered as a representative sample of the EU’s population. 

3. 

Results of stakeholder consultation 

3.1. 

Stakeholders engagement 

Table  1  below  presents  an  overview  of  all  stakeholders  consulted  in  the  framework  of 
this  evaluation.  Besides  demonstrating  that  both  statistics  users  and  producers  were 
included, the table also highlights the different stakeholder categories represented in the 
consultation  programme.  The  breakdown  evidences 
that  different  perspectives, 
institutional  and  non-institutional,  EU  and  non-EU,  are  taken  into  consideration.  The 
choice  of  the  most  relevant  consultation  tool  and  the  topics  of  the  consultation  reflects 
the level of knowledge, expertise and interest of each stakeholder category with the ESP. 

The  contractor  secured  the  engagement  of  all  stakeholder  groups  originally  included  in 
the consultation strategy. But some adjustments were made to further tailor the approach 
to  the  interview  programme  after  the  contractor  observed  a  slow  response  rate  to  the 
interview  requests  sent  out.  A  general  consultation  fatigue,  often  resulting  from  the 
COVID-19 crisis and the multiplication of online meetings, resulted in a slow response 
rate.  To  address  this  challenge,  the  contractor  worked  closely  with  Eurostat  to  identify 
relevant contacts (in the respect of the GDPR (General Data Protection Regulation) rules) 
to  send  out  tailored  interview  requests.  In  addition,  the  contractor  also  agreed  with 
Eurostat  that  it  would  be  more  efficient  to  carry  out  some  interviews  -  with  Eurostat 
thematic  units,  a  sample  of  European  Commission  DGs,  as  well  as  other  EU  and 
international bodies - in the framework of the thematic case studies instead of the general 

77 

 
 
interview programme.51 This more focused approach allowed the contractor to carry out 
more  interviews  and  avoiding  multiple  interview  requests,  while  allowing  to  collect 
specific data relevant to the evaluation. 

Table 1. Stakeholders engaged in consultation activities 

Consultation tool 

Stakeholder category 

Number of 
stakeholders 
engaged 

Scoping interviews 

General interview 
programme 

Targeted survey 

Public consultation 

Scoping interviews 

General interview 
programme 

Targeted survey 

Public consultation 

Producers 
Representatives of DG Eurostat  
Representatives of DG Eurostat and ESS 
Governance bodies 
NSIs and ONAs from a sample of 5 
Member States (MSs) 
Representatives of candidate and EFTA 
countries 
Sample of producers from targeted 
organisations 
Self-selected sample of producers 
Users 

Representatives of European Commission 
services  
Representatives of European Commission 
services and other EU bodies 
Users from a sample of 5 MS 
Sample of users from targeted 
organisations 
Self-selected sample of users  
Self-selected sample of users & 
producers52 

5 

11 

18 

4 

43 

30 

3 

6 

11 

45 

288 

84 

3.2. 

Summary of the methodology and tools used to process the data 

The  contractor  delivered  a  quantitative  and  qualitative  analysis  of  the  data  collected 
through 
included 
the  different  consultation  activities.  The  quantitative  analysis 
a descriptive  statistical  analysis  of  the  datasets,  notably  for  the  public  consultation  and 
the  targeted  surveys.  The  contractor  cross-tabulated  variables  to  check  whether  there 
were differences across different groups of respondents (e.g. between participants from a 
given country or stakeholder group). To analyse the data, the contractor used frequencies 
of  satisfaction,  revealing  trends  in  the  degree  to  which  the  European  Statistical 
Programme  has  achieved  its  intended  goals.  Averages  in  terms  of  responses  across 
respondents  were  also  used  as  part  of  the  descriptive  statistical  analysis.  In  order  to 
present  detailed  and  comprehensive  statistical  analysis,  data  were  analysed  using 
software such as SPSS and QResearch. All interview questions and open questions from 
the PC and targeted surveys were subjected to  qualitative analysis. Where  answers had 

51   The analysis of the case-study specific interviews is presented in the case study reports. 
52   A  share  of  PC  respondents  indicated  they  were  both  users  and  producers.  The  contractor  reports  on 
them  in  one  of  the  categories  to  avoid  double  counting.  The  PC  also  received  responses  from  22 
stakeholders  who  did  not  identify  themselves  as  users  or  producers.  These  are  not  included  in  this 
table. 

78 

 
 
                                                           
been provided in different languages, the contractor translated all answers into English in 
order to code them together in a systematic way. The coding of the answers per question 
along  several  categories  enabled the contractor to systematise the information provided 
and highlight the main trends emerging from open responses. 

The  analysis  was  first  done  at  the  level  of  individual  data  collection  tools,  before  the 
contractor  produced  the  overarching  analysis  to  identify  the  conclusions  across  all 
consultation  tools.  These  conclusions  in  turn  fed  into  the  preparation  of  the  main 
evaluation report. The contractor analysed and integrated the data from different sources 
to elaborate the answers to the evaluation questions. 

3.3.  Key findings from consultation activities 

Relevance of the ESP 

Under  the  relevance  criteria,  the  contractor  collected  stakeholder  feedback  about  the 
relationship between the objectives of the ESP and the needs of the European Statistical 
System  and  users  of  European  statistics  to  analyse  whether  the  ESP  was  adequately 
designed to address the needs and challenges it intended to address. 

The  feedback  from  stakeholders  generally  confirmed  that  the  European  Statistical 
Programme was fit for purpose. Most of the respondents of the targeted producer survey 
assessed  the  four  objectives  of  the  ESP  as  relevant  to  a  great  or  to  a  moderate  extent, 
including: 

  Objective  1:  To  provide  statistical  information  in  a  timely  and  cost-effective 
manner  to  support  the  adequate  development,  monitoring  and  evaluation  of  EU 
policies (86%, namely 37 responses) 

  Objective 2: To implement new methods of production to increase efficiency and 

improve quality (82%, namely 35 responses) 

  Objective  3:  To  strengthen  the  partnership  within  the  ESS  (71%,  namely  30 

responses) 

  Objective  4:  To  ensure  statistics  are  delivered  in  a  consistent  manner  and  can 

quickly adapt to new priorities (74% namely 31 responses) 

Four-fifths of the targeted producer survey respondents confirmed that the ESP activities 
were  appropriate  to  deliver  on  the  four  objectives  either  to  a  great  or  to  a  moderate 
extent. 

Users  contributing  to  the  targeted  user  survey  found  that  Eurostat  statistics  were 
appropriate  for  their  purpose  of  usage,  either  to  a  great  extent  (89%,  namely  39 
responses) or to a moderate extent (11% namely 5 responses). This group of respondents 
confirmed the importance of the first objective, with almost half of them indicating that 
they used Eurostat statistics in support of policy making at the EU level (42% namely 19 
responses). 

The  public  consultation  also  confirmed  the  overall  relevance  of  the  programme.  More 
than four-fifths of the 424 respondents indicated that European statistics were relevant to 

79 

 
 
the  respondents’  role  or  sector,  either  to  a  great  extent  (53%)  or  to  a  moderate  extent 
(31%). 

Out of all the interviews carried out, a majority of the stakeholders also highlighted that 
the  priorities  and  objectives  of  the  ESP  were  all  relevant  and  very  well  selected. 
Statistical strategies of the Member States were often based on ESP objectives, so there 
was  a  close  link  between  them.  Stakeholders  also  confirmed  that  the  ESP  individual 
objectives  were  aligned  with  the  needs  of  the  statistical  system.  Interviewees  from  the 
NSIs provided more detailed feedback on the four objectives: 

  Objective 1: in some fields such as social statistics, the programme was adapted 

at the time of the extension and they saw a clear improvement. 

  Objective  2:  a  lot  of  work  went  into  the  modernisation  of  production  methods 
that  allowed  the  ESS  to  improve  the  quality  of  statistics  and  deliver  efficiency 
gains.  The  sample  of  NSIs  interviewed  considered  the  close  links  to  the  ESS 
Vision  2020  strategy  to  be  very  helpful.  Interviewees  noted  however  that  more 
efforts were needed to modernise further production methods, which in turn was 
expected to enable the programme to meet increasing demands from users. 

  Objective 3: the interviews with the sample of NSIs revealed the existence of a 
consensus  on  the  importance  of  partnerships.  They  considered  that  these  were 
very  relevant  for  strengthening  the  relationships  within  the  system  and  with 
external  stakeholders  and  for  delivering  on  other  objectives  of  the  programme 
towards the provision of timely, consistent and high-quality European statistics. 

  Objective  4:  The  interviewees  considered  that  this  objective  responded  well  to 
NSIs’  needs  as  they  were  facing  numerous  requirements  and  demands.  They 
highlighted that the system for prioritisation had not worked that well through the 
programme, but that a new system would be piloted from 2021. 

Stakeholders  also  assessed  positively  the  introduction  of  multiannual  planning  and  the 
collaborative approach to the planning process. 

The representatives from EFTA and candidate countries interviewed confirmed that the 
ESP objectives in general were well chosen and very relevant, reflected the needs of the 
statistical organisations also outside the ESS and generally were relevant in 2013, as well 
as at the time of the extension, and had remained valid throughout the period evaluated. 
On  objective  3,  in  particular,  they  highlighted  the  close  and  intense  cooperation  with 
Eurostat.  One  interviewee  referred  to  Eurostat’s  open  and  generous  approach  to  their 
organisation since, as a non-ESS country, they were invited to and could participate in all 
meetings of the ESS. 

Interviewees from the European Commission DGs also confirmed that the ESP had been 
and remained a useful tool to meet their statistical needs. The continued commitment to 
better  and  evidence-based  policy  making  created  a  need  for  structured  statistics 
production  efforts  and  the  production  of  high  quality  and  comparable  data  across 
Member States. The ESP was set up to address this need and continues to do so. It also 
constituted an important instrument to support EU policy objectives. 

Most respondents to the targeted producer survey (55%, namely 23 responses) indicated 
that they did not see any gaps that the ESP objectives failed to cover in terms of existing 

80 

 
needs,  but  one  third  (31%,  namely  13  responses)  of  them  indicated  that  they  did.  46% 
also indicated they did not see such gaps in terms of future needs compared to 27% who 
did see them (19 responses against 11 responses). Some of the issues related to the design 
of  the  programme  that  have  been  identified  by  respondents  include:  time  constraints, 
challenges in keeping up with Eurostat’s requirements and the requests for new statistics, 
being  an  addition  to  an  already  demanding  ESP  (these  issues  are  discussed  more 
extensively  under  the  efficiency  criterion).  They  also  mentioned  they  felt  that 
partnerships  in  the  ESS  needed  to  be  strengthened,  that  there  could  be  a  closer 
coordination  from  Eurostat  of  the  work  carried  out  by  the  NSIs,  as  it  was  done  in  the 
COVID-19 crisis, and insufficient coverage of the access to privately held data. 

The respondents to the targeted producer survey  assessed that the ESP was effective in 
adapting to new challenges, such as increasing data security risks (55% assessed it was 
very  effective  or  effective,  namely  23  responses),  harmonisation  of  statistics  across 
countries (79% or 34 responses), Big Data (58% or 25 responses) and advances in data 
visualisation (71% or 30 responses). 

However, in the interviews, representatives of producers from the Member States pointed 
out some issues with the design of the programme such as: 

  a  lack  of  detail,  which  implied  that  the  ESP  was  formulated  at  a  very  generic 
level, so in fact all NSIs’ activities fell within its scope and that there was a lack 
of prioritisation. However by some this was perceived as an advantage allowing 
for flexibility; 

  a lack of flexibility in particular in reacting to emerging challenges, but also the 

ESP being a very ambitious, expensive and long programme for the NSIs; 

  a lack of strategy to access and integrate privately held data. 

The  respondents  from  the  European  Commission  also  emphasised  that  the  ESP  lacked 
flexibility:  any  change  to  the  scope  of  the  programme  required  a  change  in  the  EU 
legislation, which was a lengthy process. This implied that ad hoc needs often could not 
be met with  statistics which were reliable, timely  and comprehensive  at  the same time. 
While flexibility mechanisms existed (e.g. pilot projects), Member States participated in 
them on a voluntary basis. 

Users who responded to the targeted user survey generally indicated that they would like 
Eurostat to collect and publish more statistics within its existing statistical themes (64% 
or  29  responses  agreed  with  the  statement  while  20%  or  9  responses  were  unsure  and 
even fewer (16% or 7 responses) found the collection and publication of more statistics 
unnecessary).  Respondents  highlighted  the  need  for  a  more  detailed  and  granular 
breakdown of different  types of statistics at  regional  level,  as  well as the need for data 
subdivided  by  more  detailed  sociodemographic  attributes.  Statistical  areas  which  were 
perceived  as  requiring  more  development  by  respondents  were:  circular  economy, 
agricultural data relating to rural areas and urban and passenger mobility. 

However, producers were less eager to extend the datasets. In the targeted survey, 43% of 
producers  (18  responses)  indicated  that  they  would  not  like  Eurostat  to  collect  and 
publish  more  statistics  (26%  said  they  would).  This  was  confirmed  in  the  interviews. 

81 

 
Producers pointed out the need to keep a balance between demand for statistics and their 
(limited) capacity. 

Effectiveness of the ESP 

Under the effectiveness criteria, the contractor collected stakeholders’ feedback about the 
progress  towards  achieving  the  four  objectives  of  the  ESP  and  discussed  factors  that 
drove or hindered that progress. 

Overall,  the  feedback  from  stakeholders  confirmed  that  the  ESP  had  been  effective  in 
reaching  its  objectives,  in  particular  improving  access  to  statistics,  however  it  also 
identified several areas that could still be improved. 

Access to statistics 

In  the  public  consultation,  nine  out  of  ten  respondents  indicated  that  they  were  either 
satisfied (51% or 216 responses) or somewhat satisfied (39% or 165 responses) with the 
availability  of  Eurostat  statistics.  Interviews  with  users  confirmed  this  positive 
assessment. All the users noticed a general improvement over the last years, in elements 
such  as:  data  visualisation,  filters,  easiness  of  downloading  data  and  overall  user-
friendliness.  They  noticed  improvements  in  the  access  to  the  data  through  the  website 
and  its  user-friendliness.  They  also  indicated  that  statistics  were  better  explained  (the 
‘Statistics explained’ articles were referred to several times as well written and useful). 
Statistics  were  well  presented  for  different  types  of  users  and  were  accessible  not  only 
via  tables,  but  also  Application  Programming  Interface  (API)53.  Access  to  microdata, 
which  could  be  downloaded,  had  also  improved.  Different  elements,  such  as  detailed 
metadata with descriptions, sources, methods were useful and easily accessible via links. 
Producers  who  contributed  to  the  targeted  survey  agreed  that  the  ESP  increased  the 
availability  of  Eurostat  statistical  data,  either  to  a  great  (37%  or  16  responses)  or  to 
a moderate  extent  (44%  or  19  responses).  The  producers’  assessment  of  availability  of 
statistics in  the interviews was also  positive. The interviewees  assessed the situation as 
well as recent  developments  positively. They indicated that there was the right  balance 
between timeliness and availability of statistics. 

Despite  this  overall  positive  assessment,  interviewees  from  the  European  Commission 
DGs  still  identified  that some  statistical  needs  were  not  met  under  the  ESP.  There  was 
a need  for  more  granularity  of  the  statistics,  with  two  examples  mentioned  more 
particularly (providing statistics at the NUTS3 level instead of at the NUTS2 level, and 
gender breakdown) and new topics to be covered, including those related to the current 
European  Commission’s  priorities  (e.g.  mainstreaming  of  environment  in  all  statistical 
data to support the implementation of the EU’s Green Deal). 

Quality and harmonisation of statistics 

The stakeholder feedback confirmed the high quality of Eurostat statistics overall. It also 
confirmed that the comparability of statistics across the Member States was improving. 

53   API is a software intermediary allowing flow of information between two applications. 

82 

 
 
                                                           
In  the  targeted  user  survey,  80%  of  users  (36  responses)  considered  that  the  ESP  has 
improved the accuracy and reliability of statistics. Moreover, 77% of users (34 responses 
in  the  targeted  user  survey)  also  assessed  that  the  ESP  was  successful  in  ensuring  the 
comparability  of  statistics  among  the  Member  States.  The  interviewees  confirmed  that 
the quality of statistics was good and, since 2018, there had been constant improvement 
of  quality.  Interviewees  also  noted  that  the  statistics  produced  under  the  programme 
became  more  consistent  across  Member  States  through  the  use  of  new  tools,  guidance 
and methodologies implemented by Eurostat and that these were developed on the basis 
of  robust  methodologies,  according  to  the  principles  of  the  European  statistics  Code  of 
Practices.  Examples  provided  included  demographic  and  environmental  statistics.  The 
representatives  of  the  European  Commission  DGs  highlighted  the  importance  of  the 
ESP’s  contribution  to  the  definition  and  harmonisation  of  production  methods  and 
methodologies,  which  was  considered  a  prerequisite  for  the  production  of  reliable  and 
comparable data. Nevertheless, interviews with some users noted some inconsistencies in 
how data were collected for different countries or some gaps in statistics when data for 
certain  countries  were  not  available,  which  in  turn  might  affect  the  comparability  of 
statistics. But overall, consistency and comparability were considered as key strengths of 
the ESP and the ESS. Producers shared the view that the ESP improved the accuracy and 
reliability  of  statistics  (79%  or  34  responses  from  the  targeted  producer  survey).  There 
was  also  a  consensus  within  the  representatives  of  EFTA  and  candidate  countries 
interviewed that the ESP had delivered on its  objectives, with  a particular emphasis  on 
the  quality  improvements  achieved.  On  the  challenge  related  to  data  being  collected 
differently in different countries, producers had a rather positive perspective, highlighting 
it was introducing flexibility in the programme. 

Timeliness of statistics and new sources of data 

The  timeliness  of  statistics  was  assessed  to  be  good.  In  the  targeted  survey,  79%  of 
producers  (34  responses)  and  60%  of  users  (27  responses)  indicated  that  the  ESP  had 
been  successful  in  improving  the  availability  of  statistics  based  on  the  timeliness 
criterion. However, in the targeted user survey, the timeliness was rated lower than other 
criteria,  such  as  relevance  of  statistics  (80%  or  36  responses),  accuracy  (80%  or  36 
responses), coherence (77% or 34 responses) and accessibility (73% or 32 responses). In 
the  Public  Consultation,  respondents  indicated  that  they  were  not  using  European 
statistics  only  but  also  other  sources.  The  five  alternative  sources  of  statistics  most 
frequently mentioned by contributors were country specific National Statistical Institutes 
(32%),  the  Organisation  for  Economic  Cooperation  and  Development  (OECD,  14%), 
other national  institutes and organisations (11%), UN offices (9%) and the World bank 
(8%). Other sources such as National Banks, European organisations54, EU Offices, the 
World Health Organization, media and news, academic resources and local surveys were 
mentioned by 2% or less of those who gave an answer to this open-ended question. When 
respondents used other sources, they ranked European statistics compared to these other 
sources  as  higher  on  ‘coherence  and  compatibility’,  ‘relevance’  and  ‘accuracy  and 

54   For instance, the EU-IDB  network,  the ECDC, the European  Audiovisual Observatory and even EU 

agencies such as the European Environment Agency. 

83 

 
                                                           
reliability’ but lower on ‘timeliness and punctuality’ as well as ‘accessibility and clarity’. 
In the interviews, producers from the EU as well as EFTA/candidate countries confirmed 
that timeliness improved in the last years, especially with regard to European economic 
indicators  and  that  there  was  a  strict  calendar  for  the  publication  of  statistics,  which 
allowed  the  ESP  to  meet  the  needs  of  the  European  Commission’s  DGs  and  the  ECB. 
However, respondents pointed that not all countries were successful in this regard. 

Delays in publishing statistics were also mentioned in the interviews with users from the 
Member  States,  who  pointed  out  that  statistics  were  not  always  provided  in  a  timely 
manner, in particular for some countries. Some users also highlighted a time lag in their 
fields  of  interest  (e.g.  environmental  statistics).  Users  from  the  European  Commission 
DGs shared this assessment. While recognising the need for validation /  quality  review 
and that this could take time, interviewees also identified timeliness as an issue: certain 
policy needs were very pressing – including during a crisis situation such as COVID-19 – 
and timely statistics were not always available. 

The  interviews  with  producers  indicated  that  statistics  based  on  registers  were  timelier 
than those based on interviews, so this source should be used more when available – with 
the  caveat  added  by  Eurostat  and  users  that  this  source  should  produce  statistics  of 
sufficient  quality  to  be  used  more  systematically.  They  also  highlighted  that  more 
innovative  solutions  and  sources  should  be  used,  such  as  Big  Data  based  on  privately 
produced data, for instance on electricity usage, credit cards, mobile data, etc. They noted 
that such sources had already been used in experimental statistics. It was noted that the 
use  of  this  type  of  information  depended  at  the  time  on  the  legal  framework  of  each 
country and that there was a sense of a missed opportunity to develop this further under 
the ESP 2013-2020. They were of the opinion that more effort in this regard should be 
made in the future. At the same time, some interviewees expressed their concerns about 
the  quality  of  such  data.  They  pointed  to  the  need  for  an  in-depth  analysis  of  any  new 
sources before they could be used ‘with Eurostat’s quality stamp’. 

Dissemination channels 

The  vast  majority  of  respondents  to  the  targeted  user  survey  agreed  that  Eurostat 
dissemination channels (including the website) were easy to use and/or effective (40% to 
a  great  extent,  47%  to  a  moderate  extent  namely  39  responses  overall).  The  producer 
survey confirmed this positive assessment (42% of respondents agreed to a great extent 
that  Eurostat  dissemination  channels  were  easy  to  use  and/or  effective  and  42%  to 
a moderate extent namely 36 responses overall). 

However, one element that was often criticised by users in the interviews was the search 
engine  on  the  Eurostat  website  (this  issue  was  mentioned  by  all  interviewees). 
Respondents  indicated  they  often  used  Google  search  to  find  statistics  on  Eurostat 
website, which was  more effective than Eurostat search option. According to  them, the 
latter often provided misleading and unsatisfactory search results. 

84 

 
Metadata 

In  the  survey,  users  assessed  that  Eurostat  provided  sufficient  information  about  its 
statistics in terms of quality standards (83% or 35 respondents agreeing to a great or to 
a moderate extent), data collection methods (68% or 30 responses) and sources (86% or 
38  responses),  a  result  confirmed  by  the  PC  (83%  of  respondents  agreed  strongly  or 
somewhat  that  Eurostat  provided  sufficient  information  about  the  statistics  published). 
However, the interviews with users suggested that more advanced users noted substantial 
issues  with  metadata.  Several  respondents  indicated  that,  while  the  content  was  still 
relevant, metadata were not detailed enough. The lack of details sometimes made cross-
country comparability difficult. 

Efficiency of the ESP 

Under the efficiency  criterion, the contractor collected stakeholders’  feedback about  the 
costs and benefits of the ESP and the balance between them. 

The statistics producers provided the most meaningful feedback about cost-efficiency of 
the ESP. 

In the targeted producer survey, most of the respondents assessed the benefit to cost ratio 
of the ESP to be proportionate (55% namely 23 responses) or very proportionate (7% or 
3  respondents).  Fewer  respondents  (5%  or  2  respondents)  assessed  the  ratio  as 
disproportionate and one third of respondents were not able to assess it. 

In  terms  of  the  achievements  in  introducing  efficiency  gains  in  the  production  of 
statistics,  most  respondents  considered  that  the  ESP  was  successful  in  reducing  the 
duplication  of  effort  (70%  namely  30  responses,  including  19%  agreeing  with  the 
statement to a great extent and 51% to a moderate extent). Slightly more respondents also 
supported a statement that the ESP was successful by reducing the administrative burden 
(17%  or  7  respondents  agreed  with  the  statement  to  a  great  extent,  31%  to  a  moderate 
extent versus 31% to a small extent and 12% did not agree at all). 

For  most  of  the  producers  interviewed,  the  topic  of  efficiency  and  NSIs’  and  ONAs’ 
limited capacity to deal  with statistical requirements was the most important one in the 
study. They highlighted  that  the demand for timely statistics  was constantly increasing, 
including  the  demand  at  EU  level.  They  also  assessed  that  the  ESP  did  not  help  to 
address  these  problems.  They  pointed  out  that  the  producers  at  national  level  already 
worked  at  their  maximum  capacity  and  therefore  any  new  requests  for  statistics  were 
challenging, in particular for the smaller Member States. Some interviewees highlighted 
that the institutional users of statistics should be more aware of the production costs of 
statistics. 

Achievement of efficiency  gains  were mentioned in  the increased use of administrative 
and register data, estimation methods,  and the application of microdata exchanges. The 
management  and  governance  mechanisms  of  the  ESP  were  generally  considered 
efficient. 

85 

 
 
Several interviewees mentioned a ‘one in, one out’ approach or a need to select negative 
priorities,  i.e.  less  important  statistical  fields  which  could  be  deleted  from  the  ESP. 
However,  some  interviewees  also  acknowledged  the  difficulty  of  introducing  such 
approach  in  practice,  which  had  also  been  considered  at  country  level  but  never 
successfully implemented. 

The respondents provided similar feedback in the open comments to the targeted survey, 
indicating  that  the  elimination  of  unused  data  and  irrelevant  data  was  necessary.  They 
suggested the implementation of a more automated data validation process and reckoned 
that gains could stem from exchanging good practices between countries. 

Coherence of the ESP 

Under  the  coherence  criteria,  the  contractor  collected  stakeholder  feedback  about  the 
internal and external coherence of the ESP. Internal coherence is the alignment between 
the different objectives of the programme. External coherence is the degree to which the 
objectives of the ESP are aligned with those of other statistical authorities including EU 
bodies, as well as international partners. 

For all stakeholders engaged in the different consultation activities assessed that the ESP 
was both internally and externally coherent. 

In  the  targeted  producer  survey,  almost  all  respondents  indicated  that  the  ESP  was 
internally coherent. Most agreed with the statement that ‘the objectives of the ESP as set 
out in the Regulations and programme planning are internally coherent’ to a great extent 
(52% or 22 responses) and one third (33% or 14 responses) did so to a moderate extent. 
No  one  disagreed  (while  14%  selected  the  ‘don’t  know’  answer,  namely  6  responses). 
There was slightly less  agreement when the statement referred to  activities of the ESP: 
48% (20 respondents) agreed to a great extent and 38% (16 respondents) to a moderate 
extent.  As  with  the  statement  related  to  objectives,  no  one  disagreed  and  14%  selected 
the ‘don’t know’ answer, or 6 respondents). Moreover, 74% (31 respondents) found there 
to  be  effective  mechanisms  in  place  to  coordinate  activities  within  the  ESP  (50%  to 
a great extent, 24% to a moderate extent). 

These  findings  were  confirmed  by  the  interviews  with  producers.  The  objectives  and 
activities under the programme were considered to be coherent between each other and in 
relation  to  the ESS. The general  framework of the programme had  allowed for it to  be 
developed  through  the  multiannual  programming  and  the  annual  work  programmes. 
Although  the  annual  work  programmes  were  developed  by  the  European  Commission, 
considering the needs of the DGs, Member States were also consulted and could provide 
feedback. 

There  was  also  an  overall  agreement  about  the  external  coherence  of  the  programme. 
Respondents  agreed  to  a  great  extent  (30%  or  13  respondents)  or  to  a  moderate  extent 
(42% or 18 respondents) and outlined that there were  effective mechanisms in  place to 
coordinate with international partners. There was slightly less agreement in terms of the 
complementarity  of  Eurostat  activities  with  statistics  production  and  dissemination 

86 

 
 
activities of international  partners (39% or 16 respondents  agreed with  the statement to 
a moderate extent, and 30% or 12 respondents to a great extent). Most respondents also 
indicated there were mechanisms to coordinate with other EU bodies, 20% (9 responses) 
agreed with the statement to a great extent and 37% (16 responses) to a moderate extent. 

In the interviews, the representatives of European Commission DGs (including Eurostat), 
highlighted the existence of coordination mechanisms within the EU and the cooperation 
with  international  partners,  ensuring  the  complementarity  between  the  ESP  and  other 
statistics  produced  by  other  (political)  DGs  /  other  international  organisations.  The 
ongoing  coordination  between  Eurostat  and  thematic  DGs  was  effective  at  preventing 
overlap/duplication. 

In the interviews with producers of statistics, they assessed that the coordination between 
Eurostat,  Member  States  and  other  EU  bodies  was  very  effective,  but  they  also  noted 
some issues: 

 

Interviewees highlighted the increasing number of requests from different DGs to 
produce  statistics.  In  their  opinion,  the  DGs  did  not  always  recognise  the  costs 
and burden of producing statistics. 

  The  ESP  had  adapted  to  new  policy  needs  and  the  European  Union’s  strategies 
and objectives but was less able to address sudden changes or events or the pace 
of  certain  technological  developments  and  digitalisation.  It  was  noted  that  the 
European  Commission  could  have  better  considered  the  views  of  Eurostat  and 
Member States in the European Data Strategy, as statistics were only mentioned 
once or twice in the document. 

(FAO)  and 

Eurostat’s coordination with international partners was considered generally coherent 
and  effective  and  producers  of  statistics  and  the  representatives  of  the  European 
Commission  (EC)  highlighted  that  it  had  improved  over  the  last  years.  European 
Commission representatives agreed that there were complementarities between the ESP 
and  international  organisations,  such  as  the  OECD,  the  World  Bank,  the  Food  and 
Agriculture  Organisation 
(World  Trade  Organisation). 
Complementarities  also  existed  in  certain  policy  areas,  such  as  education,  for  instance 
when the European Commission  used the OECD’s  Pisa survey data to  complement the 
ESP for particular indicators in this policy field. Under the ESP, Eurostat also negotiated 
bilateral  agreements  with  international  organisations  to  address  data  needs,  joint  data 
collections and data exchanges overall. Some producers noted that, although there were 
still some overlaps between Eurostat and international organisations requirements, there 
were many initiatives with the objective of getting more coherent concepts that had been 
effective  at  preventing  the  duplication  of  work,  in  particular  between  Eurostat  and  the 
OECD. 

the  WTO 

Representatives  of  EFTA/candidate  countries  interviewed  agreed  that  the  ESP  was 
internally and externally coherent. 

87 

 
EU added value of the ESP 

Under the EU added value criteria, the contractor collected stakeholder feedback about 
the value resulting from the ESP that is additional to the value that would have resulted 
from similar activities only carried out at national level. 

The feedback collected clearly confirmed the EU added value of the ESP. 

In the targeted producer survey, nine out of ten respondents confirmed that the EU added 
value of the ESP. They agreed that the ESP responded to particular needs that would not 
otherwise have been addressed by NSIs. 56% of respondents (namely 24) agreed with the 
statement to a moderate extent and 23% (10 responses) to a great extent. 

When  looking  at  two  concrete  elements  of  the  EU  added  value,  namely  the 
harmonisation  of  statistics  across  Member  States  and  enhancing  the  pool  of  resources 
available, the results were similar. Nine-tenths of the producers confirmed that the ESP 
contributed  to  the  increased  comparability  of  national  statistics,  including  63%  (27 
responses)  indicating  it  contributed  to  a  great  extent  and  26%  (11  responses)  to 
a moderate extent. A similar share of respondents agreed that the ESP enhanced the pool 
of  resources  available  for  the  production  and  development  of  new  statistics  at  Member 
State level  (44% or 19 respondents  agreed with  the statement to  a moderate extent and 
28% (12 responses) to a great extent). 

In  the  interviews,  there  was  also  a  consensus  among  producers  that  the  ESP  provided 
clear  added  value  in  the  harmonisation  of  definitions  and  in  the  compilation  of 
comparable  high-quality  statistics  for  the  EU,  and  that  this  could  not  have  been  done 
without  Eurostat.  Some  interviewees  also  mentioned  that  the  ESP  provided  strategic 
foundations for producers of statistics, with a clear direction over seven years giving the 
system continuity. The programme worked by setting priorities and offered a legal basis 
for identifying upcoming fields of action and for financing projects. Member States could 
successfully  produce  official  statistics  at  national  level,  but  Eurostat  coordination  was 
necessary  for  the  production  of  European  statistics.  The  interviewees  also  highlighted 
that,  for  many  people,  Eurostat  was  more  trusted  as  an  institution  than  NSIs.  The 
respondents  also  highlighted  the  usefulness  of  Eurostat’s  grant  scheme,  bringing  up 
examples  of  grant  topics  such  as  visualisation,  education  activities,  or  fighting  against 
fake news (these were only a few examples identified by the respondents). 

The  European  Commission  and  other  EU  bodies’  officials  mentioned  in  the  interviews 
that the EU added value of the ESP through the support it provided to the delivery of the 
European  Commission’s  political  and  policy  priorities.  For  them,  the  ESP  constituted 
a central  mechanism  for  all  things  related  to  statistical  data  in  the  EU.  Thereby,  it 
decreased  the  transaction  costs  for  its  stakeholders  –  within  the  EU  as,  for  example, 
individual DGs do not have to go to individual Member States to request data and outside 
of  the  EU  as  the  ESP  was  developing  partnerships  with  international  organisations  to 
address  data  needs  –  while  making  available  the  necessary  data  and  tools.  As  with 
producers,  the  EU  officials  highlighted  that  the  ESP  supported  the  harmonisation  of 
statistics  across  the  EU.  They  also  highlighted  that  the  programme  delivered  an 
institutional  added  value.  It  gave  visibility  both  at  the  Member  State  level  and  at  the 

88 

 
international  level  to  statistical  data  and  their  importance  to  support  evidence-based 
decision making, as well as the strengthening of the international statistical community. 
In the future, the ESP could support a common approach to the challenges related to new 
data sources too. 

The EU added value was also perceived by the sample of representatives from EFTA and 
candidate countries. The interviewees considered that the same progress and benefits, in 
particular  with  the  modernisation  of  candidate  countries  statistical  systems,  would  not 
have  been  achieved  in  the  absence  of  the  programme.  However,  it  was  noted  that  the 
programme  included  some  heavy  and  lengthy  procedures  that  were  caused  by  the 
institutional  set  up,  the  different  legal  acts,  and  a  large  partnership  between  the  EU 
Member  States,  the  EC  and  the  EFTA  countries,  making  the  implementation  of  the 
programme quite complex. 

Individual  users  also  confirmed  the  EU  added  value  of  the  programme.  They  indicated 
that  there  was  clear  EU  added  value  in  Eurostat’s  work,  mainly  related  to  ensuring 
comparability  of  data  from  the  Member  States  and  providing  them  in  a  single  online 
location. Some more advanced users acknowledged the improvement in comparability of 
data  and  harmonisation  of  surveys  across  Member  States.  However,  the  users  also 
pointed out that there was still a lot to do in terms of ensuring comparability, mainly in 
terms of harmonising methods of collecting data across Member States. 

In terms of comparing the quality of Eurostat and other international sources of statistics, 
the  most  frequently  selected  answer  in  the  targeted  user  survey  was  that  quality  was 
similar  (between  47%-49%  of  responses  across  the  range  of  quality  criteria).  Between 
one third and half of respondents assessed the quality of European statistics was higher, 
depending on the indicator (51% or 23 responses for coherence and comparability, 49% 
or  22  responses  for  accuracy  and  reliability,  40%  or  18  responses  for  accessibility  and 
clarity,  as  well as  for relevance, 29% or 13 responses for timeliness).  In the interviews 
however, users highlighted that it would be challenging to compare the different sources 
(e.g. OECD, the World Bank) given the differences in the scope of the datasets. 

89 

 
Annex 3: Methods and analytical models 

3.1 Evaluation matrix 

An evaluation matrix was developed when designing the evaluation and used to draw the conclusions for each criteria. The questions proposed for the 
five evaluation criteria could cover also the points mentioned in Article 15 of the ESP amended regulation. 

Table 1: Evaluation matrix 

Evaluation questions  

Sub-questions  

Judgement criteria  

Indicators  

Data sources / methods  

ESP objectives 
reflect the needs of 
the ESS?  

1.1. ESP objectives and priorities corresponded 
to the identified needs and gaps of the 
European statistics sector and to their evolution 

Relevance: Are the activities supported under the different objectives of the ESP relevant to meeting the needs of users / producers and to supporting the policy priorities of the EU?  
EQ1: To what extent did 
  Survey of producers 
  Interviews with NSIs 
  Interviews with EU 
stakeholders 
  Interviews with 
representatives of selected 
Member States 
  Public Consultation 
  Document review including 
review of published statistical 
data and description of 
stakeholder consultation 
mechanisms 

in 2013 
in 2018 when the ESP was extended 
and revised 
currently 

1.2. Mechanisms for feedback and input by ESS 
stakeholders existed and were perceived as 
appropriate  

Stakeholder assessment of adequacy of 
consultation mechanisms to ensure ESP 
objectives remain relevant (1.2.1) 

design to identify and reflect ESS 
needs and their evolution (1.1.1) 

Stakeholder assessment of adequacy of ESP 

1. 
2. 

3. 

 

 To what extent did ESP objectives 
correspond to the needs of statistical 
organisations within the ESS when it was set 
up in 2013? 
 To what extent did the ESP objectives 
continue to meet the needs of statistical 
organisations within the ESS after its extension 
and redesign in 2018? 
 To what extent did reprioritisation exercises 
meet the identified needs at the time, especially 
regarding the 2018 extension? 
 What mechanisms were used to ensure the 
ESP objectives continue to meet statistical 
organisations’ needs? 
 To what extent were ESP activities 
appropriate to deliver the objectives set in 
2013? 
 To what extent did ESP activities continue to 
be appropriate to deliver the objectives revised 
in 2018 and to date? 
 What mechanisms were used to gather 

EQ2: To what extent were 
ESP activities 
appropriate to 
deliver the set 
objectives of the 
ESP?  

2.1. ESP activities continued to be expected to 
be the appropriate tools to deliver the set 
objectives (2013, 2018, today) 

1. 
2. 

3. 

in 2013 
in 2018 when the ESP was extended 
and revised 
currently 

90 

 

 

Stakeholder assessment of 
appropriateness of ESP activities to deliver 
the objectives of the ESP and their 
evolution (2.1.1) 

Stakeholder assessment of adequacy of 
consultation mechanisms to ensure ESP 
activities remain relevant (2.2.1) 

  Review of media and user 
satisfaction surveys 
  Document review 
  Survey of institutional users 
  Interviews with user 
stakeholders 

 
 
 
 
feedback from users? To what extent were 
these adequate and allow for adjustment? 
 To what extent did the ESP flexibly adapt to 
technological advances (e.g. big data)?  

2.2. Mechanisms for feedback and input by 
users existed and were perceived as 
appropriate 

 

Stakeholder assessment of the flexibility of 
ESP to adapt to technological advances 
(2.3.1) 

2.3. New technologies and statistical methods 
were considered in programme planning 
exercises  

Effectiveness: How effectively is the ESP delivering against its objectives?  
EQ 3: To what extent 

were the objectives 
of the ESP 2013-
2020 fulfilled?  

 To what extent was the ESP successful in 
providing timely statistical information (Obj. 1)? 
Was this delivery consistent (Obj. 4)? 
 To what extent were ESP data used in the 
development, monitoring and evaluation of EU 
policies? Was there evidence that the ESP 
contributed to improving policy making (at EU / 
MS level)? Was there evidence that the ESP 
contributed to other purposes? 
 To what extent did the ESP increase the 
availability of data, including social economy 
activities and on the Europe 2020 indicators? 
 To what extent was the ESP successful in 
introducing efficiency gains in the production of 
European statistics (Obj. 2) and avoiding 
duplication of effort (Obj. 1)? 
 To what extent was the ESP successful in 
introducing quality improvements in the 
production of European statistics (Obj. 2)? 
 How effective was the ESP in strengthening 
partnerships within and beyond the ESS 
(Obj.3)? 
 Which factors prevented or reduced the 
impact of ESP activities? How could these be 
overcome?  

3.1 Users expressed satisfaction as to the 
timeliness and completeness of data availability 

3.2 Eurostat data was downloaded and used by 
a range of stakeholders for a range of purposes 

3.3 Eurostat data were used in journalistic and 
research activity 
3.4 Data were used by EU policymakers to 
inform policy making and decisions 

3.5 Data were being used by MSs’ policymakers 
to inform policy making and decisions 

3.6 ESP activities resulted in a net increase of 
available data; users considered newly available 
data sources to be useful 

3.7 ESS stakeholders and/or users considered 
ESP activity to have increased the efficiency of 
European statistics production 

3.8 ESS stakeholders considered ESP activity 
to have avoided or prevented duplication of 
effort on the part of stakeholders 

3.9 ESS stakeholders and/or users considered 
ESP activity to have increased the quality of 
European statistics production 

91 

  Number of objectives met or in 
progress (3.1.1) 
  User assessment of timeliness, 
completeness, quality etc. of datasets as 
expressed in User Satisfaction Surveys (3.1.2) 
  Web analytics (3.2.1): 

  Change in number of website hits 

over time 

  Change in requests for microdata 

over time 

  Change in requests for data 

extraction of time 

  Change in frequency and nature of 

mention of Eurostat online 
  Stakeholder assessment of effectiveness of 
ESP in meeting user needs (3.2.2 and 3.9.1) 
  Net increase in the number of statistical data 
sets and long time-series (3.6.1) 
  Stakeholder assessment of the impact of 
ESP on quality and efficiency (3.7.1; 3.8.1 and 
3.9.1) 
  Increased or more in-depth activity taking 
place with partners within and beyond the 
ESS (3.8.2) 
  Stakeholder assessment of the impact of the 
ESP on strengthening partnerships (3.10.1) 

  Interviews with 
ESS stakeholders 
  Interviews with 
representatives of selected 
Member States 
  Public Consultation 
  Case studies 

  Review of administrative 
data and KPIs (including 
Eurostat Quality Reports) and 
programme output mapping 
  Document review 
  Review of media and user 
satisfaction surveys 
  Survey of institutional users 
  Survey of producers 
  Interviews with all 
stakeholder groups 
  Public Consultation 
  Case studies  

 
 
 
 
 
 
 
 
 
 
 
 
3.10 ESP activity strengthened existing 
partnerships or developed new ones 

  Stakeholder assessment of barriers to 
current or increased effectiveness of ESP 
activity (3.11.1) 

EQ 4: To what extent did 
the ESP make 
progress on 
rendering access to 
official statistics 
easier and more 
user-friendly? 

 Were effective feedback mechanisms in place 
to identify accessibility issues and 
improvements? 
 Did users find the Eurostat dissemination 
channels (including the website) easy to use? 
 Were users provided with sufficient 
information about key aspects of the data?  

3.11 Stakeholders identified factors that have 
prevented of reduced the impact of ESP activity 
4.1 Effective processes were in place to monitor 
and receive feedback about data accessibility 

4.2 Relevant information (e.g. new releases, 
data quality standards) was available in a clear 
and accessible manner 

4.3 Users expressed satisfaction with 
dissemination and communication methods 
used by Eurostat  

EQ 5: Did ESP activity 

result in wider 
economic, social or 
environmental 
impacts? 

 Was there evidence that the ESP contributed 
to wider social, economic or environmental 
impacts? 

5.1 Examples of wider social, economic or 
environmental impact were identified by 
stakeholders and attributed in part or in full to 
ESP activity  

  Description of the processes in place to 
monitor and receive feedback about data 
accessibility (4.1.1) 
  User assessment of user-friendliness of 
Eurostat communication channels as expressed 
in User Satisfaction Surveys or through other 
channels (4.2.1) 
  Change in the number of web impact and 
positive/negative mentions (4.2.1) 
  Fulfilment of objectives relating to improving 
communication and dissemination (4.3.1) 
  Examples of impact drawn from document 
review 
  Stakeholder reports of wider impact resulting 
from ESP activity (5.1.1) 

  Document (web) review 
  Review of administrative 
data and KPIs 
  Review of media, user 
satisfaction surveys and web 
impact 
  Survey of institutional users 
  Public Consultation 

  Document review 
  All stakeholder interviews 
  Case studies  

Efficiency: How efficient is the use of ESP resources?  
EQ 6: To what extent 
were ESP 
resources used 
efficiently to 
achieve the desired 
results? 

 What were the processes in place to collect 
information on costs and benefits across ESP 
activities and to what extent did they inform 
programme decisions and operations? 
 What were the systems in place to monitor 
and optimise the use of resources? 
 How effective were anti-fraud measures and 
processes in place to prevent misallocation of 
ESP funds? 
 To what extend was the ESP successful in 
introducing efficiency gains in statistics 
production? 

6.1 Sufficient information about costs and 
benefits across ESP activities was available and 
used to inform programme decisions and 
operations 
6.2 Measures and processes in place to monitor 
and optimise the efficient use of resources were 
effective 
6.3 Anti-fraud measures and processes in place 
were effective 
6.4 ESP operations spending was efficient 
6.5 The ESP was successful in introducing 
efficiency gains in statistics production 

  Stakeholder assessment of the availability of 
information on costs and how that feeds into 
programme decisions and operations (6.1.1) 
  Existence of measures to monitor and 
optimise efficient use of resources (6.2.1) 
  Reported cases of fraud (6.3.1) 
  Stakeholder assessment of ESP operations 
efficiency (6.4.1) 
  Relative output of Eurostat products to 
budget increase/decrease (6.4.2) 
  Relative output of Eurostat products to 
number of staff (6.4.3) 
  Stakeholder assessment of the impact of 

   Review of administrative 
data, including cost analysis 
and evaluation outputs 
  Document review 
  Interviews with Eurostat staff 
  Interviews with NSIs and 
ESS stakeholders 
  Case studies 
  Public Consultation 

92 

 
 
 
  
  
 
 
 
  
  
 
  
  
ESP on quality and efficiency (6.5.1) 

EQ7: To what extent were 
ESP activities 
successful in 
limiting the 
administrative 
burdens for ESS 
stakeholders, 
including Member 
States and data 
providers 
(respondents)?  

 What steps were taken to analyse the 
administrative burdens for NSIs and statistics 
producers? 
 What steps were taken to reduce the 
administrative burdens for NSIs and statistics 
producers? How effective were they? 
 To what extent did the ESP provide benefits 
for Member States and other data providers 
relative to the costs of delivering these results? 
 What national factors affected this balance? 

7.1 Measures and processes in place to analyse 
administrative burdens for NSIs and statistics 
producers were effective 
7.2 Measures undertaken to reduce 
administrative burdens for NSIs and statistics 
producers resulted in an improved cost-benefit 
balance 
7.3 NSIs and Member States considered the 
benefits of ESP to outweigh the administrative 
burdens they face  

  Existence of measures to monitor and 
analyse administrative burdens for NSIs and 
statistics producers (7.1.1) 
  Stakeholder assessment of efficiency of 
measures to monitor and reduce administrative 
burdens (7.2.1) 
  Stakeholder assessment of benefits of ESP 
activities relative to their administrative burdens 
(7.3.1)  

  Review of administrative 
data, including cost analysis 
and evaluation outputs 
  Interviews with Eurostat staff 
  Interviews with NSIs and 
ESS stakeholders 
  Interviews with user groups 
(e.g. business) 
  Survey of producers 
  Case studies  

EQ 8: Was the 

management / 
organisation of the 
ESP as a whole 
conductive to 
supporting efficient 
delivery? 

 How effective were systems in place to review 
the efficiency and performance of the ESP? 
 How effective were Eurostat governance 
mechanisms at monitoring the efficient use of 
resources?  

8.1 Effective systems were in place to review 
ESP performance and identify risks to efficient 
use of resources 
8.2 Stakeholders involved in the governance of 
the ESP and Eurostat had sufficient access to 
information to monitor the efficiency of resource 
use  

  Stakeholder assessment of the effectiveness 
of governance systems (8.1.1) 
  Official opinions provided by EU institutions 
and governance stakeholders on ESP/Eurostat 
operations (8.2.1) 

  Document review 
  Interviews with Eurostat staff 
  Interviews with ESS 
governance stakeholders  

Coherence: Is the ESP considered to be coherent?  
EQ9: To what extent did 
ESP activities and 
objectives 
contribute to the 
internal coherence 
of the ESS?  

 Were the activities and objectives of the ESP 
set out in the Regulations and programme 
planning internally coherent? 
 At the national and international level, were 
processes to ensure the coherence of statistical 
data identifiable and deemed fit-for purpose? 

9.1 Activities and objectives set out in the 
Regulations and internal planning documents 
were internally coherent 
9.2 Effective processes were in place to monitor 
and enable the coherence of internal ESS 
activities 

 EQ10: To what extent do 
ESP activities 
complement / 
contradict / overlap 

 Are ESP activities and data aligned with the 
needs of overarching EU strategies and 
objectives? Are there opportunities for further 
alignment? 

9.3 Stakeholders involved in the production of 
European statistics considered that ESP 
activities promoted internal coherence and/or 
did not negatively impact internal coherence 

10.1 ESP activities and data are 
actively aligned and/or not in contradiction 
with overarching EU strategies and objectives 

93 

  No coherence issues (overlap, duplication, 
contradiction) identified from document review 
during the evaluation period (9.1.1) 
  Identification of suitable coherence 
mechanisms (9.2.1) 
  Stakeholder assessment of the effectiveness 
of the coherence mechanisms (9.3.1) 
  Stakeholder assessment of the overall 
internal coherence of the ESS (9.3.2) 

  Document review 
  Interviews with Eurostat staff 
  Interviews with NSIs and 
ESS stakeholders 
  Survey of producers 
  Case studies 

  Stakeholder assessment of effectiveness 
of coordination with EU bodies (10.1.1) 
  No coherence issues identified from 
document review during the evaluation period 

  Document review 
  Interviews with Eurostat staff 
  Interviews with ESS 

 
 
 
  
  
 
 
 
with wider EU 
activity?  

 Are the statistics delivered throughout the 
ESP flexible to respond to new strategic 
priorities? 
 How effectively does Eurostat coordinate with 
other EU bodies? 

10.2 Effective mechanisms are in place to 
monitor and respond to wider EU strategies and 
priorities 

10.3 EU stakeholders consider that Eurostat 
coordinates effectively with other EU bodies  

(10.1.2) 
  Identification of suitable coherence 
mechanisms (10.2.1) 
  Number and rate of fulfilment of DG-driven 
data requests (10.2.2) 

stakeholders 
  Interviews with EU 
stakeholders 
  Case studies 

EQ11: To what extent are 
ESP activities 
coherent with the 
activities of 
international 
statistics 
organisations?  

 How effectively does Eurostat coordinate with 
international partners (e.g. OECD) on the 
development of international concepts, 
classifications, methods and other standards?  

11.1 Effective mechanisms are in place to 
coordinate with international partners 

11.2 International partners consider that 
Eurostat activities complement and/or are not in 
contradiction with their own statistics production 
and dissemination activities 

  Stakeholder assessment of alignment of 
ESP activities and data with EU 
priorities (indicator 10.3.1) 
  Identification of suitable coherence 
mechanisms (11.1.1) 
  No coherence issues identified from 
document review during the evaluation 
period (11.1.2) 
  Stakeholder assessment of effectiveness 
of coordination with international 
partners (11.2.1) 

EU Added Value: what is the EU added value of the ESP?  
EQ 12: What is the EU 

added value of the 
ESP?  

 Does the ESP contribute to increased 
comparability of national statistics? 
 Are ESP statistics preferred to other 
competing international statistical data sets? 
 Does the ESP contribute to reduced time lag 
between reference period and publication data 
of statistics? 
 Does the ESP enhance the pool of resources 
available for production and development of 
new statistics at MS level? 
 Can the production of European Statistics be 
successfully achieved at the MS level? 

12.1 Users of data consider ESP to be the 
preferred source of information when they 
compare indicators across MSs 
12.2 Individual and institutional users’ stated 
preference for Eurostat data over other 
international data sets 
12.3 There is a recognised advantage of having 
a “one stop shop” for harmonised and validated 
European statistics 
12.4 What would happen in the absence of 
ESP?  

94 

  Majority of surveyed and interviewed users 
prefer ESP data over other sources when 
comparing MS level statistical 
indicators (12.1.1) 
  Majority of surveyed and interviewed users 
prefer ESP data over other competing 
international data sets (12.2.1) 
  Majority of surveyed and interviewed users 
confirm the advantage of having “one stop 
shop” for harmonised and validated European 
statistics (12.3.1) 
  Stakeholders  qualitative  answers 
counterfactual on the absence of ESP (12.4.1) 

to 

the 

  Document review 
  Interviews with Eurostat staff 
  Interviews with international 
organisations 
  Case studies 
  Public Consultation 

  Survey of producers 
  Survey of institutional users 
  Case studies 
  Interviews with NSIs and 
ESS stakeholders 
  Public Consultation 

 
 
 
 
 
 
 
  
 
 
3.2 Overview of the methodological approach 

The evaluation team designed and implemented a phased approach to the evaluation. 

The designing phase (task 1) resulted in the definition of the evaluation framework. 
Building on a round of scoping interviews and an initial review of relatively high-level 
documents, the team furthered its understanding of the ESP, as well as the evaluation and 
its objectives. These preliminary data collection activities led to the development of the 
evaluation  baseline  (as  presented  in  the  inception  report,  deliverable  D1)  and  the 
preparation  of  the  Evaluation  Questions  Matrix  (EQM).  The  EQM  structured  all 
subsequent tasks under this evaluation, both in terms of the data collection and in terms 
of data  analysis. Key statistical  terms  applied in  the European Statistical  System  (ESS) 
are defined in  the Glossary of the European Statistics Code of Practice.55  In the report, 
the evaluation team often refers to the general terminology ‘data and statistics’, which is 
consistent with Eurostat’s mission statement.56 

The data collection phase (task 2) was dedicated to compiling all relevant data (both 
primary and secondary) to inform the evaluation. The evaluation team implemented a 
mixed-method  approach.  Data  collection  consisted  of  the  following.  First,  the  team 
carried  out  a  comprehensive  documentary  review  (see  the  reference  list  presented  in 
Annex 2). The systematic review of all sources identified allowed to identify and collect 
all data with regard to the performance of the ESP between 2013 and 2020, to feed into 
the  responses  to  the  evaluation  questions.  Second,  the  evaluators  designed  and 
implemented  a  stakeholder  consultation  strategy.  Consultation  activities  allowed  to 
corroborate  findings  of  the  documentary  review  as  well  as  to  address  any  data  gaps 
identified. The consultation strategy consisted of (1) targeted online surveys addressed to 
statistics  users  and  producers,  (2)  a  public  consultation  and  (3)  in-depth  stakeholder 
interviews  (all  carried  out  remotely).  The  synopsis  report  (deliverable  D4,  submitted 
separately)  provides  a  detailed  description  of  the  stakeholder  consultation  strategy,  its 
implementation  and  its  results.  Overall,  despite  the  challenges  resulting  from  the 
COVID-19  pandemic  and  the  need  to  consult  all  stakeholders  remotely,  the  level  of 
engagement  achieved  was  high.  The  following  table  provides  an  overview  of  this 
engagement: 

55   Available from: https://ec.europa.eu/eurostat/documents/4031688/9439112/Glossary/ 
56   ‘Our  mission 

high-quality 

statistics 

provide 

data 

and 

to 

is 

on  Europe.’  See: 

https://ec.europa.eu/eurostat/web/main/about/who-we-are 

95 

 
 
 
 
 
 
 
                                                           
Table 2. Stakeholders engaged in consultation activities 

Consultation tool 

Stakeholder category 

Number of 
stakeholders 
engaged 

Scoping interviews 

General interview 
programme 

Targeted survey 
Public consultation 

Scoping interviews 

General interview 
programme 

Targeted survey 

Public consultation 

Producers 

Representatives of DG Eurostat  
Representatives of DG Eurostat and ESS 
Governance bodies 
NSIs and ONAs from a sample of 5 Member 
States (MSs) 
Representatives of candidate and EFTA countries 
Sample of producers from targeted organisations 
Self-selected sample of producers 

Users 
Representatives of European Commission 
services  
Representatives of European Commission 
services and other EU bodies 
Users from a sample of 5 MS 
Sample of users from targeted organisations 
Self-selected sample of users  
Self-selected sample of users & producers57 
Non-users, non-producers 

Public consultation 
Source: Evaluation team’s consultation strategy 

Self-selected sample 

5 

11 

18 

4 
43 
30 

3 

6 

11 
45 
288 
84 

22 

The evaluation team also structured part of the data collection efforts into thematic and 
country case studies. Four thematic case studies explored the extent to which data made 
available under the ESP have added value in specific policy domains (see Annex 4): (1) 
European  business  statistics,  (2)  statistics  on  asylum  and  managed  migration,  (3) 
territorial  statistics  and  (4)  COVID-19.  The  case  studies  focused  on  the  downstream 
impact of ESP activities by conducting a ‘deep dive’ of detailed outputs for institutional 
and  non-institutional  users.  The  sample  of  topics  reflected  several  criteria,  namely  the 
balance  between  economic,  social  and  territorial  fields,  as  well  as  the  opportunity  to 
engage  with  a  range  of  users.  These  case  studies  built  on  a  documentary  review  of 
relevant sources and a programme of stakeholder interviews. In addition, the team carried 
out five country case studies. The sample covered Estonia, Germany, Poland, Spain and 
Sweden. These case studies built on interviews of officials responsible for the statistical 
work programme in the NSIs and in ONAs, and other stakeholders such as policy makers 
and regular professional users of the data at national level, and fed into the preparation of 
country  factsheets  (see  Annex  3)  providing  an  overview  of  the  statistical  systems 
implemented at Member State level and of the dynamics at play in statistics production. 

The final two phases of the evaluation, analysis (task 3) and judging (task 4), involved 
the systematic analysis of quantitative and qualitative data, mapping the data to the 
evaluation topics and questions, comparing the evidence collected through different 
tools  and  drawing  conclusions.  They  culminated  in  the  formulation  of  practicable 
recommendations from the evaluation. 

57   A share of PC respondents indicated they were both users and producers. The evaluation team reports 

on them in one of the categories to avoid double counting. 

96 

 
 
                                                           
Figure 1: Three components of analysis 

The conclusions and recommendations are strictly based on findings from the evaluation, 
providing  founded  judgements  based  on  a  solid  and  rigorous  analysis  using  both 
quantitative  and  qualitative  approaches  as  applicable.  The  answer  to  each  EQ  includes 
the following elements: 

 

 

interpretation and comprehension of the key terms of the EQs; 

indication of the judgement criteria allowing to answer the question as well as the 
quantitative analysis that is possible given the data available; 

  validity of the quantitative and qualitative information used; and 

  description of the evaluation methods used and an indication of their limitations. 

In addition, each answer provides a detailed description of the reasoning followed in the 
analysis, also indicating the underlying hypotheses and validity limits of that reasoning. 

Lastly, as the team delivered the evaluation, it faced a number of challenges to which it 
responded as follows: 

1.  The  COVID-19  crisis  prevented  the  team  from  undertaking  face-to-face  data 
collection activities. All activities took place remotely instead. This implied that 
the team  deployed stronger and more targeted efforts  to  ensure a good response 
rates for the interviews to be conducted for the overall programme assessment, as 
well as the country and thematic case studies. 

2.  The  scope  of  the  evaluation  means  there  have  been  numerous  information  and 
data  sources  processed.  Several  measures  were  implemented  to  avoid  overload. 
These included: (i) a strict relevance test applied to data gathering to discard not 
strictly  pertinent  data;  (ii)  the  accurate  use  of  data  already  available  in  order  to 
complement the primary data collection; and (iii) the use of a common template 
for  gathering  qualitative  and  quantitative  information,  following  the  evaluation 
questions and sub-questions, to allow easier processing. The preliminary mapping 
undertaken  as  part  of  the  inception  phase  was  instrumental  for  identifying 
relevant  documents  to  map  and  the  purpose  they  will  serve.  In  addition,  strong 
coordination with the evaluation manager at Eurostat was necessary to ensure that 
all relevant documentary sources, including those not publicly available yet, were 
made available to the evaluation team. 

97 

 
3.  One of the main sources of data for the evaluation have been the perceptions and 
opinions  of  relevant  stakeholders,  including  statistics  users  and  producers.  A 
consultation  programme  of  this  scale  presents  logistical  and  conceptual 
challenges. The list of proposed stakeholders and related tools has been designed 
to  achieve  the  best  possible  balance  between  interest  groups  and  topics.  The 
contractor  has  designed  different,  tailored  survey  tools  with  skip  logic  that 
directed stakeholders to relevant questions based on their profiles and familiarity 
with  key  elements  of  the  ESP.  Dissemination  was  a  particular  challenge,  in  the 
absence of a comprehensive list of users of statistics. The contractor relied on the 
support provided by members of the Inter-service Steering Group on that aspect. 
In  addition,  stakeholder  engagement  with  the  consultation  tools  creates  a  self-
selection bias: the sample of stakeholders who contributed to the evaluation is not 
statistically  representative  of  the  ESP  stakeholders  (for  instance,  the  sample  of 
users  contributing  to  this  evaluation  did  neither  cover  all  users,  nor  a 
representative  sample  of  users).  Findings  are  only  indicative  of  stakeholders’ 
opinions  and  need  to  be  considered  with  caution.  Lastly,  the  findings  are  also 
caveated  by  the  fact  that  stakeholders’  feedback  might  have  been  subject  to  a 
bias,  the  interviewer  effect  (interviewees  might  give  the  responses  they  thought 
were  desired  by  an  interviewer)  and  conditioned  by  the  self-selection  of 
respondents contributing to online consultation tools, because the respondents and 
interviewees were aware that the study was carried out on behalf of Eurostat. The 
contractor  mitigated  this  methodological  challenge  through  the  use  and  analysis 
of  different  sources  of  evidence,  which  fed  into  the  answers  to  the  evaluation 
questions, and caveated carefully the analysis. 

98 

 
 
Annex 4: Final Report of the Contractor 

99 

Contractor's final report