Commission staff working document - Report from the Commission to the European Parliament and the Council on the ex-post evaluation of the community statistical programme 2003-2007 {COM(2009) 1 final}
Date: 2009-01-13
Institutions: European Commission
Subjects: EU statistics, Eurostat, data collection, project evaluation, report, statistics

EN 

EN 

   EN 

 
COMMISSION OF THE EUROPEAN COMMUNITIES 

Brussels, 13.1.2009 
SEC(2009) 14 final 

COMMISSION STAFF WORKING DOCUMENT 

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND 
THE COUNCIL ON THE EX-POST EVALUATION OF THE COMMUNITY 
STATISTICAL PROGRAMME 2003-2007 

{COM(2009) 1 final} 

EN 

   EN 

 
 
  
 
 
 
COMMISSION STAFF WORKING DOCUMENT 

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND 
THE COUNCIL ON THE EX-POST EVALUATION OF THE COMMUNITY 
STATISTICAL PROGRAMME 2003-2007 

This Commission staff working document contains the executive summary of the evaluation 
report on the Community statistical programme (CSP) 2003-2007. 

1. 

INTRODUCTION 

The requirement to carry out an evaluation and present a report on the Community statistical 
programme (CSP) 2003-2007 is enshrined in Article 4 of Decision No 2367/2002/EC of the 
European Parliament and of the Council of 16 December 2002 on the Community statistical 
programme 2003-2007. Article 4 of that Decision states: 

"At  the  end  of  the  period  covered  by  the  programme,  the  Commission,  after  consulting  the 
Statistical  Programme  Committee,  shall  present  an  appropriate  evaluation  report  on  the 
implementation of the programme, taking into account the views of independent experts. The 
report  is  to  be  completed  by  the  end  of  2008  and  subsequently  submitted  to  the  European 
Parliament and the Council." 

The task of carrying out this ex-post evaluation was awarded to IDEA Consult - a company 
specialized  in  research-based  consulting.  Starting  in  October  2007,  the  evaluation  was 
conducted  over  a  period  of  13  months;  it  included  desk  research,  review  and  analysis  of 
quantitative data, a number of interviews with various stakeholders, and a survey addressed to 
the  National  Statistical  Institutes  (NSIs).  In  March  2008,  an  interim  evaluation  report  was 
submitted  which  presented  facts  and  factual  trends,  but  no  evaluative  conclusions.  In  the 
ensuing period, more qualitative information was gathered and processed. This whole process 
resulted  in  an  evaluation  report,  of  which  this  document  is  the  executive  summary.  The 
summary provides an overview of the main findings and conclusions from the report and is 
accompanied by the evaluator's recommendations. 

2. 

MAIN FINDINGS FROM THE EVALUATION BUILDING BLOCKS 

In  order  to  maintain  a  structured  approach  in  the  evaluation,  we  identified  a  number  of 
aspects, which together characterise the Community Statistical Programme (CSP) 2003-2007 
and  the  European  Statistical  System  (ESS).  These  aspects  have  served  as  building  blocks 
throughout  the  whole  evaluation  process.  In  the  paragraphs  below  we  present  the  most 
important  facts  and  findings  for  each  of  the  building  blocks  of  the  evaluation  framework. 
These are: 

•  Relevance of the programme 
•  Progress of activities 
•  User satisfaction and user needs 
•  Development of the ESS and subsidiarity 
•  Staff and internal productivity 
•  Dialogue and Interaction between users and partners 
•  Quality of products and services 
•  Monitoring of implementation 

EN 

2 

   EN 

 
2.1. 

Relevance of the CSP 2003-2007 

THE LINK BETWEEN THE 5-YEAR PROGRAMME AND THE ANNUAL PROGRAMMES 

The  5-year  programme  serves  as  the  framework  document  for  the  five  individual  annual 
programmes  over  the  period  in  question.  However,  no  explicit  reference  to  the  5-year 
programme  is  to  be  found  in  either  the  structure  or  the  content  of  the  annual  programmes. 
Every  title  in  the  CSP  2003-2007  contains  a  list  of  its  relevant  themes  in  the  annual 
programmes, but there is no trace in the annual programmes of either the objectives described 
in the titles or of the six strategic principal objectives formulated separately from the titles of 
the 5-year programme. The annual programmes seem to lead the work without any reference 
to the 5-year programme. 

The evaluator has conducted an exercise ex post to try to link the key operational priorities in 
the horizontal themes of the annual programmes with the six principal objectives of the 5-year 
programme.  This  exercise  has  provided  a  very  useful  insight  into  the  extent  to  which  the 
annual  programmes  do  actually  cover  the  content  of  the  5-year  programme.  It  has 
demonstrated that there is a match between the 5-year programme and the annual programmes 
as  far  as  content  is  concerned.  By  working  on  the  priorities  of  the  annual  programme  the 
Eurostat units are - whether consciously or not - making progress on the strategic objectives 
of the 5-year programme. The results of the survey show that this perception is shared by the 
National Statistical Institutes (NSIs), with 92% of them believing that the priorities included 
in the annual programmes were derived directly from the 5-year CSP. 

Despite  the  apparent  match  between  the  5-year  programme  and  the  annual  programmes  in 
terms  of  content,  the  conducting  of  the  abovementioned  exercise  ex  ante  for  the  whole 
programme by explicitly linking the key operational priorities from the annual programmes to 
each  of  the  principal  strategic  objectives  in  the  5-year  programme  would  provide  the 
necessary  reference  framework  for  the  different  units  to  draw  up  the  annual  programmes. 
Furthermore, a clear link between operational priorities and strategic objectives would allow 
progress towards the objectives of the 5-year programme to be monitored more efficiently and 
transparently. 

The indicators show that, each year, a number of new priorities are adopted and/or old ones 
are discarded. This confirms that the practice of working with annual statistical programmes 
in a multi-annual framework gives Eurostat the flexibility to achieve its objectives as set out 
in the 5-year programme. Each year, new emphases and adjustments to changing needs can be 
introduced  into  the  annual  programmes.  As  a  result,  this  combination  has  within  itself  the 
potential  to  provide  a  flexible  and  permanently  relevant  approach  to  meeting  the  statistical 
challenges as they change over time. 

The  broad  and  flexible  formulation  of  the  objectives  of  the  5-year  programme  also  has 
another aspect to it. In the interviews and the survey, concerns were raised that the objectives 
are too general, and that clear-cut priorities or strategic direction are lacking. This is a concern 
for the NSIs because their own statistical programmes are based on the Eurostat programmes. 
Furthermore, by far the majority of the NSIs' activities derive from EU regulations, and are 
therefore  obligatory.  Moreover,  the  fact  that  the  formulated  objectives  are  not  linked  to 
quantifiable targets makes it difficult to evaluate them. 

INVOLVEMENT OF USERS AND PRODUCERS IN THE FORMULATION PROCESS 

In  general,  both  the  Directorates-General  (DGs)  and  the  NSIs  feel  that  they  have  been 
involved sufficiently in the formulation of the CSP 2003-2007 and that their input has been 
taken  into  account.  On  the  other  hand,  they  find  that  the  process  of  influencing  the 
programmes is difficult and often time-consuming. Both groups seem to place a great deal of 

EN 

3 

   EN 

emphasis  on  influencing  the  annual  statistical  programmes,  as  these  are  more  specific  and 
aimed at the short term. 

The two groups obviously have certain opposing interests. The DGs want to see new requests 
implemented,  whereas  the  NSIs  focus  mostly  on  keeping  the  number  of  requests  in  the 
programmes  under  control  and  on  introducing  more  prioritisation.  The  growing  demand, 
coupled  with  the  absence  of  priority  setting,  seems  to  be  a  serious  problem  for  the  NSIs, 
because  their  resources  often  do  not  match  the  large  and  growing  number  of  requests.  The 
NSIs are constantly trying to strike a balance between the demands they have to meet and the 
resources  at  their  disposal,  and  they  are  finding  it  increasingly  difficult  to  achieve  that 
balance.  This  has  led  to  the  introduction  of  so-called  "negative  priorities"  -  i.e.  looking  for 
ways  to  reduce  the  statistical  burden  by  abandoning  outmoded  statistics  or  by  simplifying 
existing obligations. 

Furthermore,  the  NSIs  feel  that  new  demands  are  not  always  dictated  by  necessity. 
Incidentally,  according  to  the  survey,  the  aspects  ‘keeping  the  demand  under  control’  and 
‘burden reduction by prioritising statistics’ are the only ones where Eurostat has achieved less 
than the NSIs expected. 

2.2. 

Progress of activities 

All  Eurostat  units  have  reported  on  their  activity  and  progress  towards  the  objectives  by 
individual  title  in  the  CSP  2003-2007  by  completing  'fact  sheets'  prepared  by  the  evaluator. 
Reporting  by  production  units  and  by  horizontal  units  has  been  processed  differently. 
Reporting  by  the  horizontal  units  was  added  to  the  hierarchy  of  objectives  which  had 
previously been used to make explicit the link between the 5-year programme and the annual 
programmes.  For  the  production  units  the  matrix  shown  below  was  devised  (Table  1).  The 
matrix consists of three core areas of action (rows) – data, regulation/legislation, methodology 
&  tools  -  and  four  major  relevant  Eurostat  issues  (columns)  –  Quality,  Development  ESS, 
Simplification/modernisation  and  Enlargement  –  plus  a  residual  category  entitled  ‘Regular 
activity’. The activities reported by the units have been added up and classified in the various 
categories of this matrix. This provides a valuable overview of the number of activities carried 
out  in  each  area  of  action  and  on  each  major  issue  for  Eurostat.  The  specific  content  and 
individual  actions  included  in  this  table  are  discussed  in  the  relevant  sections  of  the 
evaluation. 

Table 1: Categorisation and sums of reporting on vertical fact sheets 

EN 

4 

   EN 

Regular 
activity

Enlargement

Simplification/ 
modernisation Quality

Development 
ESS

Total

Main area of 
action

Data

Regulation/ 
legislation

Methodology 
& tools

Subareas
Coordinating 
surveys
Production of 
existing statistics
Production of new 
statistics
Total
Preparing new 
legislation
Adopting new 
legislation
Revising old 
legislation
Total
Preparing new 
methodology or tool
Implementing new 
methodology or tool
Revising 
methodology
Monitoring 
methodology
Total

8

40

56
104

9

14

3
26

24

13

7

2
46

1

15

2
18

0

0

0
0

0

1

0

1
2

Overall total

176

20

Source: Idea Consult, 2008 

1

4

2
7

6

4

9
19

3

4

10

3
20

46

2

24

9
35

5

5

6
16

6

16

18

15
55

106

3

4

5
12

3

0

1
4

9

17

5

5
36

52

15

87

74
176

23

23

19
65

42

51

40

26
159

400  

Significant action has been taken on all of the principal objectives, areas of activity and major 
issues  identified.  Unsurprisingly,  most  actions  reported  by  the  production  units  have  been 
related  to  the  production  of  statistics.  The  number  of  actions  involving  production  of  new 
statistics  is  quite  high  compared  to  the  other  activities.  A  large  proportion  of  the  actions  of 
both groups of units also falls under the heading of quality. Furthermore, many activities were 
dedicated to the development of the ESS. 

Although a fair share of the activities related to simplification or modernisation, only a few of 
these  actions  focused  on  data  production.  Simplifications  and  modernisations  are  more  a 
feature of the legislation and the methodology than of the actual production of statistics. 

There  is  little  activity  to  report  in  relation  to  the  enlargement  project,  which  all  sources 
describe  as  ‘running  smoothly’.  The  interviewed  NSIs  from  new  Member  States  stated  that 
their  accession  had  been  well  prepared  on  both  the  input  and  output  sides,  and  that  their 
integration  into  the  ESS  had  gone  very  well.  Also,  as  the  enlargement  to  include  10  new 
countries had taken place in 2004, most of the preparations for enlargement had already been 
completed before the CSP 2003-2007 began. 

The NSIs generally agree that considerable progress has been made towards the objectives of 
the  5-year  programme.  Furthermore,  the  results  of  the  survey  largely  support  the  findings 
from  Table  1,  as  the  NSIs  indicated  that  the  aspects  ‘production  of  new  statistics’, 
‘development of methodology’ and ‘data accessibility’ were among those receiving the most 
attention  during  the  CSP  2003-2007.  However,  the  NSIs  felt  that  the  CSP  had  paid  less 
attention to all aspects related to “reduction of burden” and aspects related to the monitoring 
and  management  of  processes  and  performances.  Encouragingly,  the  NSIs  point  out  that 
greater attention is being paid to these aspects in the CSP 2008-2012. 

2.3. 

User satisfaction and user needs 

All indicators relating to the ‘use’ of Eurostat statistics and publications point to a significant 
upward trend. Examples of this include the monthly number of users of online dissemination 
and  the  number  of  downloads  from  the  Eurostat  website,  both  of  which  have  risen 
significantly  (Annual  Activity  Report  2007,  Annual  Management  Plan  2007,  website 

EN 

5 

   EN 

monitoring reports 2005, 2006, 2007).  The trend can be seen most clearly in the number of 
hits  on  the  website  per  month  (performance  scoreboard,  2006/2007),  which  increased  from 
34.7 million in 2004 to 73.1 million in 2007 indicating every item displayed on the website. 
One of the contributory factors was the remarkable rise in the number of requests when the 
free dissemination policy was introduced. The accession of the 10 new Member States to the 
European Union also probably contributed to this growth. 

The distribution of downloads across the various statistical areas has remained stable, with a 
strong  interest  being  shown  in  general  socio-economic  statistics.  For  the  period  2004-2007, 
the  areas  of  Population  &  Social  Conditions  and  Economy  &  Finance  accounted  for  more 
than 20% of the total number of downloads. Areas such as Environment & Energy, Science & 
Technology and Transport barely reached 5%. 

User monitoring reports reveal no significant increase in the number of requests for support 
by  users  over  the  period  2004-2007  (up  from  23993  to  24928).  This  can  be  regarded  as  a 
positive trend, since it suggests that the increase in the number of users was not followed by a 
big  increase  in  the  number  of  clients  who  felt  that  the  presentation  or  the  content  of  the 
website was insufficient to meet their needs. 

The  main  source  for  rating  user  satisfaction  was  the  user  satisfaction  survey,  which  was 
carried out in 2007 on two groups: internet respondents and key users. Although it should be 
borne  in  mind  that  surveys  of  this  kind  add  a  negative  bias  (as  they  usually  attract  more 
complaining,  negative  respondents  than  satisfied  users),  they  do  offer  interesting  insights. 
From  an  aggregate  perspective,  approximately  half  of  all  users  rate  the  overall  quality  of 
statistics  as  'good'  to  'very  good'.  The  timeliness  aspect  of  the  statistics  is  the  quality 
dimension  which,  in  general,  had  a  relatively  higher  proportion  of  user  satisfaction. 
Completeness and accessibility received somewhat lower ratings. 

This  aggregate  approach  conceals  major  differences  between  statistical  areas.  Price  and 
employment  statistics  are  generally  rated  highly,  whereas  the  regional  and  environmental 
statistics receive lower ratings. Also, key users tend to be slightly more positive than internet 
respondents. 

The aggregate perspective gained from the user satisfaction survey is rather negatively biased 
compared to quality surveys on separate statistical areas. Other surveys conducted for separate 
statistical areas show higher user satisfaction ratings. A survey of international trade statistics 
shows that 85% of users are 'very satisfied' or 'fairly satisfied' with these statistics. However, 
this survey also reveals less satisfaction with certain aspects, such as comparability of the data 
(39% satisfied) and coherence between different trade data sets (35% satisfied). 

Most  of  the  DGs  interviewed  feel  that  Eurostat  performs  quite  well  on  its  core  task,  i.e. 
supplying other Commission Services with high quality statistics. All DGs regard Eurostat as 
the  main  authority  when  it  comes  to  statistics  and  the  first  port  of  call  when  they  need 
statistics of any kind. It is only in cases where Eurostat does not have the information that the 
DGs try other options – usually the OECD, the World Bank, the IMF or the ILO. Some DGs 
note that other sources are sometimes more policy-oriented than Eurostat – and thus easier for 
them to use – but then those sources can cause other problems. 

2.4. 

Development of the ESS and subsidiarity 

In order to gain an initial insight into the development of the ESS and the application of the 
subsidiarity principle, we examined the new legal acts in each statistical domain. We recorded 
a total of 33 new legal acts over the period 2003-2006. The subsidiarity principle is referred to 
in 11 of these acts. 

EN 

6 

   EN 

The statistical requirements compendia for the years 2003 to 2007, which include overviews 
of legislative acts and data requirements, have also been studied. Both seem to have peaked in 
2006. Eleven extra legal acts have been added to the compendium, i.e. the number of new acts 
minus the acts that have been abolished. The increase in data requirements is very noticeable 
(up by 138). This indicates that the ESS is  expanding  and becoming increasingly regulated. 
During the 2003-2007 programme period, a start was made on the ex ante evaluations of new 
statistical  acts  adopted  during  the  programme  period,  with  12  such  evaluations  being 
conducted in 2007. 

By far the majority of NSIs claim that they have to contend with an imbalance between users' 
demands and the resources at their disposal. The interviews and the survey show clearly that 
NSIs  are  suffering  from  the  constant  increase  in  users  needs  for  more  and  greater 
sophistication of data, together with the fact that they barely have the resources to cope with 
this  increased  workload.  They  feel  that  (more)  effort  should  be  put  into  reducing  the 
workload, either by limiting the demands or by increasing the resources. 

The  DGs  acknowledge  this  problem,  but  argue  that  lowering  the  existing  statistical 
requirements  is  complicated,  as  it  would  negate  much  of  the  previous  investment  in 
developing the statistics and their  particular  features.  Another consideration is that  reducing 
the  burden  for  survey  respondents  has  been  identified  as  a  major  concern  in  the  ESS  too. 
Some argue that this burden is inversely proportional to the burden for the NSIs. Less work 
for  respondents  often  means  a  heavier  burden  for  the  data  producers  (NSIs)  in  terms  of 
processing the surveys. 

All of this goes to show how delicate this issue is, and how difficult the position of Eurostat is 
in this respect. The ESS can be likened to a constant delicate balancing act between the supply 
side (producers) and the demand side (users). Those interviewed on both sides acknowledge 
that Eurostat is in a difficult position in the middle of the system, and has the complex task of 
managing not only the system but also the expectations of both users and producers. 

Both the DGs and NSIs fully recognise the working groups, task forces and committees as the 
decision-taking and executive bodies of the ESS, are closely involved in them and perfectly 
satisfied  with  the  way  they  function.  Nevertheless,  the  NSIs  feel  that  the  ESS  provides  too 
few opportunities for discussion at a more strategic level. The sector groups are named as the 
bodies  that  should  be  given  broader  mandates  and  greater  prominence,  since  the  role  of 
working groups is primarily a technical one. 

Furthermore,  both  NSIs  and  DGs  are  aware  of  the  importance  of  the  SPC,  and  find  its 
functioning to be increasingly efficient and effective. In order to improve the efficiency of the 
SPC,  efforts  have  been  made  to  reduce  the  number  of  agenda  items  discussed  in  this  body. 
Some NSIs have argued that this aspect should not be exaggerated, and that the SPC should 
maintain its strong position within the system. 

The  DGs  underline  the  importance  of  a  continuous  dialogue  between  producers,  users  and 
Eurostat.  Direct  communication  between  DGs  and  Eurostat  normally  takes  place  through 
personal, informal contacts, and thus is not highly institutionalised. This means that a lot of 
information does not trickle down to the whole DG. This has to do with the lack of statistical 
coordination  within  the  DGs,  although  the  DGs  point  out  that  Eurostat,  too,  could  be  more 
proactive when it comes to the dissemination/communication aspects. Many of the main client 
DGs  have  separate  direct  agreements  and  contracts  with  Eurostat.  Certainly,  in  the 
development phase of new statistics this can provide clarity as to what each one can expect 
from the other, and avoids miscommunication. 

EN 

7 

   EN 

The derogations that were granted during the period 2003-2007 shed light on another aspect 
of  the  development  of  the  ESS.  Derogations  are  basically  exceptions  granted  to  Member 
States from certain (parts of the) requirements, allowing them to not to deliver that part of the 
demand.  The  highest  total  number  of  derogations  for  one  country  over  the  full  programme 
period  is  12;  two  countries  have  been  granted  10  derogations.  These  figures  can  be 
supplemented  by  so-called  "compliance  monitoring",  which  shows  the  extent  to  which  the 
Member  States  are  complying  with  a  (wide)  range  of  ESS  statistical  regulations.  Overall 
compliance can be regarded as very high. 

The number of derogations for new Member States is strikingly low. At the same time, non-
compliance by the new Member States is 1.8%, as against 4.5% for the old Member States. 
One possible explanation for this is that the new Member States are not yet very familiar with 
the  system  of  derogations.  The  NSIs  from  new  Member  States  that  were  interviewed  also 
acknowledge  that  they  have  a  slight  advantage,  because  in  the  case  of  NSIs  from  existing 
Member States the implementation of the ESS often involves replacing or modifying existing 
national  regulations  on  which  all  actors  had  reached  a  consensus  and  to  which  they  had 
become  accustomed.  For  the  new  NSIs,  it  simply  means  implementing  a  new  Regulation, 
which may possibly require a shorter adaptation period. 

Eurostat's units have devoted much of their activity to developing the ESS. One of the most 
relevant  facets  of  this  effort  is  the  elaboration  and  implementation  of  ESSnet  -  a  system  in 
which  certain  Member  States  become  experts  in  specific  statistical  fields,  perform  specific 
tasks and share their knowledge throughout the ESS. This initiative is strongly supported and 
highly appreciated by the NSIs; it appears to work well, and is an example of a measure that 
has a significant and evident impact on the burden. Another potentially important body within 
the ESS - the European Advisory Committee on Statistical Information in the Economic and 
Social Spheres (CEIES) - has been reformed to make it a more strategic body, whose aim is to 
facilitate reflection on the whole strategic programming and interaction within the ESS. 

Another interesting stage in the development of the ESS is the agreement reached at the end 
of 2006 on the Krakow action plan. This resulted from the meeting of the Directors General of 
the ESS in Krakow in 2006 at which three task forces (TF) were set up: (1) a TF for a basic 
legislative framework for Community Statistics (2) a TF for statistical challenges and (3) a TF 
for ESS resources management and programming. These TFs have not yet reached the end of 
their  mandate,  but  have  each  formulated  a  number  of  recommendations  on  their  respective 
themes. 

2.5. 

Staff and internal productivity 

The smallest number of key priorities in the annual programmes related to principal objective 
4 ‘Boosting staff motivation’. However, the reporting in the fact sheets demonstrates that the 
horizontal units have largely compensated for this through extensive action on these priorities. 

Over  the  period  there  has  been  a  clear  increase  in  the  number  of  staff,  reaching  a  peak  in 
2007. This has meant fewer vacancies. The extra staff were distributed relatively equally over 
the units. The percentage of staff involved in statistical production has increased from around 
80% to 83%. 

The  gender  division  of  the  staff  has  remained  fairly  stable  across  all  levels  (Equal 
Opportunities in Eurostat report 2007). In both higher and middle management, around 20% 
of staff is female; at non-managerial level, the figure is closer to 30%. The fact sheets report 
that a mentoring programme for female administrators is in place. 

During  the  period  under  review,  various  actions  were  taken  to  increase  the  internal 
productivity  of  the  work  at  Eurostat.  An  important  factor  in  this  context  has  been  the 

EN 

8 

   EN 

implementation  of  the  Single  Entry  Point  (SEP). Other  examples  taken  from  the  fact  sheets 
include: 

•  the development and improvement of an internal communication strategy and tools; 
•  the development of a new intranet portal; 
•  the maintenance and improvement of CIRCA. 

According  to  various  sources  (Annual  Management  Plan  2007,  Performance  scoreboard 
2006/2007),  about  one  third  of  the  data  transmission  from  the  Member  States  takes  place 
through the SEP. This means that around 65% of all transmissions are still taking place via 
other, mainly electronic, channels such as mail, CD and FTP, which entails risks as regards 
confidentiality and complicates administration within Eurostat. The introduction of the SEP is 
highly  appreciated  by  the  NSIs,  as  was  clear  from  the  interviews  and  the  survey.  The  NSIs 
view the SEP as a useful  simplification for both the NSIs and Eurostat.  They also maintain 
that this tool is not used sufficiently and that it should be expanded further. 

2.6. 

Dialogue and interaction between users and partners 

The  indicator  sheets  show  that  since  2005  the  number  of  working  group  meetings  has 
gradually increased from 117 to 130. Satisfaction with these meetings has remained very high 
(around 87% in 2006). 

The number of grant agreements rose from 319 (€35.148.909) in 2003 to 536 (€60.660.156) 
in 2005. In 2006, the number fell back to 428 (€53.280.474). 

An important issue in the report is the reduction of the burden on the producers of statistics 
(NSIs). The assessment of what has been achieved in this field is to a large extent based on 
interviews and on the results of the survey sent to the Member States. The working group on 
‘Coordination  and  Programming’  and  the  task  force  on  ‘Priority  Setting’  address  the 
possibilities of reducing the burden and recommend practical measures. These documents and 
initiatives make it clear that this is a live issue among stakeholders within the ESS and one 
which is certainly receiving a great deal of attention in the current CSP 2008-2012. 

For the time being, however, the NSI are only moderately satisfied with the achievements in 
this area. A number of NSIs have had to make serious cutbacks in their organisation, whereas 
demand  just  keeps  increasing.  Their  view  is  that  many  more  initiatives  can  be  taken  to 
simplify, modernise and just reduce the production of statistics. 

A structural  concern arising from the interviews is  that  there is little direct contact between 
the NSIs and DGs. The NSIs do not complain about Eurostat performing the difficult task of 
intermediary between the two, but they still feel that there is a lack insight into each others' 
wishes and problems, and of mutual fine-tuning. The NSIs would like to have more contact 
with the DGs at a strategic level. They would also welcome greater involvement of the DGs in 
the  development  of  statistics,  so  that  DGs  would  gain  a  better  idea  of  the  costs  involved  in 
developing  and  producing  statistics.  On  the  other  hand,  the  DGs  are  reluctant  to  do  this,  as 
they rely on the coordinating work done by Eurostat. 

The  mutual  lack  of  understanding  of  their  respective  positions  and  activities  takes  concrete 
shape  in  the  opposing  views  expressed  by  NSIs  and  DGs  when  interviewed  on  the  issue  of 
accepting  and  meeting  new  demands.  In  the  survey,  56%  of  the  NSIs  state  that  they  are 
insufficiently informed about the discussions that are taking place between Eurostat and the 
Commission services on new statistical demands. This creates a feeling within the NSIs that 
their  point  of  view  is  not  being  taken  into  account  in  this  process.  The  NSIs  have  the 
impression  that  Eurostat  accepts  new  demands  too  readily,  without  considering  all  the 
implications for  the  statistical institutes. On the other hand, the DGs believe Eurostat is too 

EN 

9 

   EN 

"soft"  on  the  NSIs  in  terms  of  enforcing  the  requirements.  The  NSIs  therefore  advocate  a 
more  transparent  and  clearer  process  for  accepting  new  requests  and  turning  them  into  a 
regulation, and for paying more attention to the implications of this for their work. 

2.7. 

Quality of products and services 

Many of the actions listed on the fact sheets for both the production units and the horizontal 
units  within  Eurostat  fall  into  the  "quality"  category.  The  demand  for  a  more  in-depth  and 
detailed examination of the statistics and other quality improvements is constant, and unlikely 
to decrease.  Throughout  the  evaluation it can be seen that Eurostat and the ESS as  a  whole 
have put a lot of emphasis and investment into this, in order to accommodate these requests 
and raise the overall quality level. 

Between  2003  and  2007,  some  43  quality  reports  were  produced,  and  five  statistical  areas 
were fully reviewed in terms of quality. Also, one third (12 out of 33) of the new legal acts 
explicitly  include  a  quality  monitoring  requirement.  Furthermore,  the  quality  aspect  has 
increasingly become a topic of discussion in working group meetings and on the agendas of 
the Statistical Programme Committee (SPC). 

Various  initiatives  have  led  to  the  so-called  “Quality  assurance  framework”,  which  brings 
together a range of tools and methods to assess, monitor and improve statistical quality. The 
Code of Practice encompasses institutional and quality principles in the ESS, while the peer 
reviews  performed  within  the  ESS  on  the  implementation  of  the  Code  reveal  that  overall 
quality has improved significantly. Furthermore, the Eurostat peer review report expresses the 
widespread  appreciation  on  the  part  of  the  review  panel  and  all  stakeholders  for  the 
commitment and leadership shown by Eurostat on the quality issue. 

The  Eurostat-wide  effort  to  work  on  and  improve  the  quality  of  statistics  is  undeniable. 
Therefore, it would be very interesting to assess and disclose the effects of this effort on the 
various  aspects  of  statistical  quality.  Uniform,  transparent  and  correct  monitoring  of  the 
quality  of  statistics  is  obviously  essential.  However,  it  is  this  uniformity  –  in  particular  – 
which appears to be creating a major obstacle to the conducting of such an assessment. 

The two main general sources on statistical quality - the performance scoreboard (2006/2007) 
and the quality barometer (2007) - are summaries of quality reports for the different statistical 
areas. These reports are compiled by different units, who each perform their own monitoring 
and  use  differing  methods  and  definitions.  These  data  are  thus  highly  dispersed  across 
different databases, aspects and years, and cannot be compared with any degree of accuracy. 

In  order  to  gain  an  insight  into  the  results  of  all  these  efforts  to  improve  quality,  an 
appropriate next step would be to develop a clear and coherent quality monitoring framework, 
alongside  the  quality  assurance  framework.  The  approach  and  methods  to  measure  quality 
should be the same for all parts of Eurostat, and should be used throughout the whole Eurostat 
organisation. 

The evaluator has nevertheless tried to summarize some of the available data on quality. For 
the  Euro  indicators,  a  number  of  figures  on  completeness  of  the  data  are  available.  Around 
22%  of  the  data  series  in  the  Euro  indicators  do  not  contain  values,  and  therefore  are  not 
available. Of the 78% of the data series that are available, only 1.5% includes missing data. 
Furthermore, 96% of the Euro indicator data covers a period greater than 5 years. The non-
response  rate  for  the  different  surveys  (Labour  Force  Survey,  Structural  Business  Survey, 
Structure of Earning Survey) is approximately 20%. 

Concerning timeliness, we have found figures on the freshness of the Euro indicators which 
show that 55% are between 60 and 90 days old. The LFS data are published with an average 
delay of 116 days after the reference period. 

EN 

10 

   EN 

Thanks to the continued attention to quality, none of the DGs question Eurostat’s position as 
the main authority and source when it comes to statistics. However, the DGs do have a lot to 
say  about  the  quality  of  the  statistics,  and  have  expressed  a  desire  to  see  improvements. 
Ongoing  work  is  therefore  required  in  order  to  cope  with  a  demand  which  is  becoming 
increasingly sophisticated and complex. 

The interviews highlight two structural concerns: both the frequency and the degree of detail 
of the data are often not high enough. More and more, the Commission services need monthly 
or weekly data rather than annual figures, in order to be able to react more quickly to all kinds 
of developments. Furthermore, they need data at lower, more detailed levels, and increasingly 
even at micro level. Eurostat has difficulties delivering these data because of confidentiality 
issues  -  as  it  is  often  not  permitted  to  publish  individual  data  -  and  because  of 
representativeness issues, because statistical samples (and thus the number of respondents) are 
not  constructed  to  provide  robust  information  at  detailed  levels.  Moreover,  the  balance 
between timeliness (frequency) and accuracy/level of detail is a very delicate one. The more 
quickly statistics have to be produced, the less accurate they are likely to be. While the DGs 
obviously understand this, it nevertheless remains a problem for them. 

Other concerns that are voiced include the user friendliness of the website and NewCronos. It 
appears that the website and the NewCronos database are difficult for new users to understand 
and  use.  Second,  the  consistency  within  and  between  databases  could  be  improved.  For 
instance,  each  Member  State  produces  price  statistics,  but  these  are  not  available  in  a 
consistent form at Eurostat level. Price data are used to produce both the HICP (harmonised 
index  of  consumer  prices)  and  purchasing  power  parities.  This  work  is  done  in  different 
Eurostat  units,  with  price  statistics  being  spread  over  two  different  directorates.  Therefore, 
compiling a comparable overview is a very complicated exercise. 

2.8.  Monitoring of implementation 

There is no framework for monitoring and reporting Eurostat’s progress towards attaining the 
objectives of the 5-year programme. There is some reporting on content in the annual activity 
reports (AAR), but the link between these annual activity reports and the 5-year programme is 
weak. This is why, at each evaluation round (mid-term, ex post), information on the progress 
of this programme has to be collected from the various units. The transparency of Eurostat’s 
activities  would  be  greatly  increased  if  a  strategic  exercise  were  to  be  carried  out  which 
involved: 

linking  the  (operational)  objectives  of  the  annual  programmes  to  the  (strategic) 

(i) 
objectives of the 5-year programme, and then 

(ii) 

defining key actions and key monitoring indicators. 

3. 

RECOMMENDATIONS 

3.1.  Main recommendations 

LINK 

The 5-year programme is designed as a framework programme that sets out a strategic vision 
for the next five years. It ensures, in conjunction with the annual (working) programmes, that 
the  ESS  will  continue  to  adopt  a  flexible  and  relevant  approach  to  meet  the  statistical 
challenges that are changing over time. 

However, in order to improve the transparency of Eurostat's activities, and to ensure that the 
5-year programme does indeed provide the necessary reference framework within which the 

EN 

11 

   EN 

various  Eurostat  units  have  to  draw  up  the  annual  programmes,  these  annual  programmes 
should refer explicitly to the 5-year programme. 

This explicit reference can be achieved by devising a hierarchy of objectives, i.e. by linking 
the  key  operational  priorities  (in  the  themes)  from  the  annual  programmes  to  the  strategic 
objectives and titles in the 5-year programme. The evaluator has provided an example of such 
a hierarchy for the horizontal objectives. 

It  is  also  clear  that,  during  a  programme  period,  new  developments  in  society  might  occur 
which the 5-year programmes cannot anticipate, and that this may give rise to unexpected new 
statistical demands. In that case, the annual programmes should be used as a way to adapt to 
these  developments  and  to  cope  with  the  new  demands.  Therefore,  the  annual  programme 
may indeed contain issues and/or priorities that cannot be linked to the 5-year programme. We 
believe it would appropriate to clarify the reasons for deviating from the 5-year programme in 
explicit terms, and to provide a clear explanation as to why additional issues and/or priorities 
should be included in the annual programme. 

An explicit reference of this kind will ensure that the combination of the 5-year programme 
and the annual programmes is used to its full potential as the main strategic driver behind the 
work.  Moreover,  a  clear  link  between  the  operational  priorities  and  the  strategic  objectives 
would allow the efficient and transparent monitoring of progress towards the objectives of the 
5-year programme. 

MONITORING 

Within Eurostat there are a number of monitoring activities designed to closely follow up the 
various  plans,  including  monitoring  of  data  quality,  user  satisfaction  surveys,  staff  opinion 
surveys,  compliance  monitoring,  and  (website)  user  monitoring.  These  initiatives  are 
definitely a sign of good governance. 

Two points merit greater attention. First of all, there is no overall monitoring of the 5- year 
programme  or  of  the  achievements  towards  its  objectives.  Eurostat's  progress  towards 
achieving the objectives of the 5-year programme is only reported on during the (mandatory) 
evaluation  stage.  Secondly,  the  various  monitoring  activities  and  initiatives  are  conducted 
relatively  independently  of  each  other:  for  example,  the  quality  aspects  surveyed  differ 
considerably from statistic to statistic; the various categories of statistics defined in the user 
satisfaction survey differ from those defined on the website and also from those defined in the 
performance scoreboard. 

Overall monitoring of the 5-year programme can be achieved, as already mentioned above, by 
establishing  a  closer  link  between  the  5-year  programme  and  the  annual  programmes,  and 
subsequently  by  choosing/defining  (from  among  the  many  potential  candidates)  a  few  key 
indicators that provide information leading to the achievement of both the operational and - 
thanks to the link to be established - the strategic objectives. A better streamlining of lower-
level monitoring initiatives can be achieved by defining clearer links and greater uniformity 
between the different operational parts of the organisation. 

A more streamlined monitoring of the various independent initiatives, together with an overall 
“tableau  de  bord”  to  follow  up  Eurostat’s  activities,  will  not  only  simplify  reporting 
requirements, but at the same time improve the complementarity, transparency, comparability 
and - hence - the visibility of several of the initiatives. 

ESS: BALANCING DEMAND AND RESOURCES 

The ESS can be characterised as a constant delicate balancing act between its producer and its 
user sides. Eurostat is at the heart of the ESS and has to fulfil the difficult task of being an 

EN 

12 

   EN 

intermediary  between  the  “main  requesters”  (DGs)  and  the  “main  providers”  (NSIs)  of  the 
statistics.  Many  instruments  (such  as  ESSnet,  derogations,  single  entry  point)  have  been 
developed and new structures (e.g. the European Statistical Advisory Committee (ESAC)) put 
in place to support and improve the efficiency and effectiveness of (the providing side of) the 
ESS. The parties involved are generally satisfied with the functioning of the various bodies, 
such  as  the  SPC,  working  groups  and  task  forces.  However,  there  is  still  a  lot  of  room  for 
improvement. 

On the demand side, the main trend - which is confirmed by the various DGs - is an increase 
in the demand for statistics (in terms of quantity, level of detail or quality). Increasingly, the 
Commission's  services  need  frequent  and  detailed  data  in  order  to  be  able  to  react  to 
developments of all kinds. Despite all the efforts made to ensure efficiency and effectiveness 
on  the  providers'  side,  there  is  a  limit  to  the  possibilities  on  the  supply  side,  where  the 
situation  is  one  of  growing  or  increasingly  sophisticated  and  complicated  demand.  This  is 
confirmed by the NSIs, who are concerned about the balance between the resources at their 
disposal and the statistical demands and requirements. Moreover, they are not very satisfied 
with how they are kept informed about the discussions between Eurostat and the Commission 
Services on the subject of new demands for statistics 

This situation calls for either an increase in resources for the producers or a strategic rethink 
of demand - and probably aspects of both. There are a number of channels to guide a strategic 
rethinking  of  demand,  and  these  must  be  used  to  provide  a  context  not  only  for  existing 
demand,  but  also  for  new  demand:  they  include  priority  setting,  simplification  and  burden 
reduction for existing demands and ex ante impact assessments for new demands. To sound a 
cautionary note, there are certain forms of burden reduction for the providers of statistics (e.g. 
enterprises) that may lead to even greater burden creation for the producers (the NSIs), unless 
the various legislations are adapted accordingly. 

These  subjects  are  delicate  and  difficult,  and  they  will  need  time;  but  they  are  regarded  as 
being of the utmost importance. The ground was prepared during the period 2003-2007 and 
this  aspect  is  receiving  a  great  deal  of  attention  in  the  current  CSP  2008-2012.  Given  that 
tangible results from the working groups and task forces dedicated to these issues are essential 
to  the  sustainability  of  the  ESS,  we  would  recommend  that  sufficient  resources  should  be 
devoted to them. The concrete proposals for action from these WGs and TFs should be given 
consideration at the highest levels. 

TRUST 

Trust between the various stakeholders is fundamental to the proper functioning of the ESS. 
Eurostat is in the delicate position of being an intermediary between the DGs and the NSIs. 
Fundamentally the system is made up of interactions between the players involved, who – to 
some  extent  –  may  have  opposing  interests.  The  ESS  consists  of  several  bodies  whose 
purpose  is  to  stimulate  dialogue,  and  thereby  create  trust.  In  recent  years,  the  further 
development of the ESS and the increase of this trust seem to have gone hand in hand. The 
survey results provide an excellent illustration of this, as the growth of trust within ESS is as 
one of the aspects most highly rated by the NSIs. 

The  DGs'  confidence  in  Eurostat's  ability  to  defend  their  needs  vis-à-vis  the  NSIs,  and  - 
equally - the trust that the NSIs place in Eurostat to defend their aspirations vis-à-vis the DGs, 
must be intensively nurtured in the light of their common interest in quality statistics. 

3.2. 

Other recommendations 

ENLARGEMENT 

EN 

13 

   EN 

The  main  preparations  for  the  enlargement  process  took  place  before  2004  and,  thus, 
enlargement  during  the  period  2003-2007  did  not  pose  any  problems.  The  Eurostat  units 
describe the enlargement process as ‘smooth’, and the new NSIs have likewise expressed their 
satisfaction with this process. 

However,  it  appears  that  the  process  has  not  yet  reached  its  culminating  point,  and  that  the 
newest Member States - Romania and Bulgaria - deserve the same investment in order to be 
integrated  into  the  system.  Moreover,  our  own  observations  reveal  that  data  for  these  new 
Member States are not complete on Eurostat’s website. 

This  successful  integration  into  the  ESS  should  also  be  reflected,  during  the  current 
programme period, in the availability of data from the new Member States. Investment in the 
integration of the new NSIs should be continued so as to ensure that the statistics from new 
Member States are complete and available to users in the (website) databases. 

The further integration of new Member States and extension of the availability of data from 
these countries are essential for two reasons. First of all, a more complete database can only 
improve  the  service  to  users,  i.e.  both  Commission  Services  and  non-institutional  users. 
Secondly, this will be beneficial for the development of the ESS, as increasingly all members 
are becoming equal partners both in theory and in practice. This enhances the credibility and 
legitimacy of Eurostat and the ESS as a whole. 

QUALITY 

The ESS-wide effort to work on and improve the quality of statistics is undeniable. Eurostat 
has  placed  a  great  deal  of  emphasis  on,  and  efforts  and  investment  into,  improving  quality; 
this  includes  the  quality  assurance  framework,  the  Code  of  Practice  and  surveys  to  monitor 
the quality of different statistics. Nevertheless, continued efforts are still required in order to 
cope with a demand which is becoming increasingly sophisticated and complicated. Thanks to 
this  continuing  attention  to  quality,  all  DGs  regard  Eurostat  as  the  main  authority  when  it 
comes to statistics. Eurostat is the first source they look to when they need data. 

Monitoring  the  quality  aspects  of  the  different  statistics  is  essential  to  the  "visibility"  of 
quality. However, the various monitoring activities are carried out by different statistical areas 
using differing methods and definitions. For example, there are several sources to which one 
can refer in order to get an idea of data completeness: (1) completeness of data expressed as a 
percentage of data frequency for NewCronos as a whole, (2) number and percentage of data 
series available; number and percentage of data series with missing values and length of series 
in years for Euro-indicators; and (3) average percentage availability of data for dissemination 
for  the  Structural  Business  Statistics  in  EU25.  The  approach  and  methods  used  to  measure 
quality  should  be  more  similar  for  all  parts  of  Eurostat,  and  should  be  used  throughout  the 
entire Eurostat organisation. 

The  development  of  a  clear  and  coherent  quality  monitoring  framework  would  be  an 
appropriate  next  step.  Such  streamlined  monitoring  can  be  defined  both  alongside,  and 
building  on,  the  quality  assurance  framework.  Uniform  (though  flexible)  quality  indicators 
could considerably advance such harmonisation of quality reporting. 

Harmonisation of quality reporting would improve Eurostat’s quality image even further, and 
at the same time contribute to greater credibility and transparency in its relations with users.  

EN 

14 

   EN