EN    EN EN 
EN    EN  COMMISSION OF THE E UROPEAN COMMUNITIES 
Brussels, 13.1.2009 
SEC(2009) 14 final 
  
COMMISSION STAFF WORKING DOCUMENT 
REPORT FROM THE COMMISSION TO  THE EUROPEAN PARLIAMENT AND 
THE COUNCIL ON THE EX-POST EVALUATION OF THE COMMUNITY 
STATISTICAL PROGRAMME 2003-2007 
 
{COM(2009) 1 final} 
  
EN 2   EN COMMISSION STAFF WORKING DOCUMENT 
REPORT FROM THE COMMISSION TO  THE EUROPEAN PARLIAMENT AND 
THE COUNCIL ON THE EX-POST EVALUATION OF THE COMMUNITY 
STATISTICAL PROGRAMME 2003-2007 
 
This Commission staff working document contains  the executive summary of the evaluation 
report on the Community statis tical programme (CSP) 2003-2007. 
1. I NTRODUCTION  
The requirement to carry out an evaluation a nd present a report on the Community statistical 
programme (CSP) 2003-2007 is enshrined in Ar ticle 4 of Decision No 2367/2002/EC of the 
European Parliament and of th e Council of 16 December 2002 on the Community statistical 
programme 2003-2007 . Article 4 of that Decision states: 
"At the end of the period covered by the prog ramme, the Commission, after consulting the 
Statistical Programme Committee, shall present an appropriate evaluation report on the 
implementation of the programme , taking into account the vi ews of independent experts . The 
report is to be completed by the end of 2008  and subsequently submitted to the European 
Parliament and the Council." 
The task of carrying out this ex-post evalua tion was awarded to IDEA Consult - a company 
specialized in research-based consulting. Starting in October 2007, the evaluation was 
conducted over a period of 13 months; it incl uded desk research, review and analysis of 
quantitative data, a number of interviews with va rious stakeholders, and a survey addressed to 
the National Statistical Institutes (NSIs). In  March 2008, an interim evaluation report was 
submitted which presented facts and factual trends, but no evaluative conclusions. In the ensuing period, more qualitative information was gathered and processed. This whole process 
resulted in an evaluation report, of which this document is the executive summary. The summary provides an overview of the main findi ngs and conclusions from the report and is 
accompanied by the evaluator's recommendations. 
2. M
AIN FINDINGS FROM THE EVALUATION BUILDING BLOCKS  
In order to maintain a structured approach in the evaluation, we identified a number of 
aspects, which together char acterise the Community Sta tistical Programme (CSP) 2003-2007 
and the European Statistical System (ESS). Th ese aspects have served as building blocks 
throughout the whole evaluation pr ocess. In the paragraphs below we present the most 
important facts and findings for each of the building blocks of the evaluation framework. 
These are: 
• Relevance of the programme 
• Progress of activities 
• User satisfaction and user needs 
• Development of the ESS and subsidiarity 
• Staff and internal productivity 
• Dialogue and Interaction be tween users and partners 
• Quality of products and services 
• Monitoring of implementation 
EN 3   EN 2.1. Relevance of the CSP 2003-2007 
THE LINK BETWEEN THE 5-YEAR PROGRAMME AND THE ANNUAL PROGRAMMES  
The 5-year programme serves as the framew ork document for the five individual annual 
programmes over the period in question. However, no explicit reference to the 5-year 
programme is to be found in either the stru cture or the content of the annual programmes. 
Every title in the CSP 2003-2007 contains a lis t of its relevant themes in the annual 
programmes, but there is no trace in the annual programmes of either the objectives described 
in the titles or of the six strategic principal obje ctives formulated separately from the titles of 
the 5-year programme. The annual programmes seem  to lead the work without any reference 
to the 5-year programme. 
The evaluator has conducted an exercise ex post to  try to link the key operational priorities in 
the horizontal themes of the annual programmes with  the six principal obje ctives of the 5-year 
programme. This exercise has provided a very useful insight into the extent to which the 
annual programmes do actually cover the c ontent of the 5-year programme. It has 
demonstrated that there is a match between th e 5-year programme and the annual programmes 
as far as content is concerned. By working on the priorities of the annual programme the Eurostat units are - whether consciously or no t - making progress on the strategic objectives 
of the 5-year programme. The results of the surv ey show that this perception is shared by the 
National Statistical Institutes (NSIs), with 92% of them believing that the priorities included in the annual programmes were derive d directly from the 5-year CSP. 
Despite the apparent match between the 5-y ear programme and the annual programmes in 
terms of content, the conducting of the above mentioned exercise ex ante for the whole 
programme by explicitly linking the key operation al priorities from the annual programmes to 
each of the principal strategic objectives in  the 5-year programme would provide the 
necessary reference framework for the differe nt units to draw up the annual programmes. 
Furthermore, a clear link between operational pr iorities and strategic objectives would allow 
progress towards the objectives of  the 5-year programme to be monitored more efficiently and 
transparently. 
The indicators show that, each year, a number of  new priorities are adopted and/or old ones 
are discarded. This confirms th at the practice of working with  annual statistical programmes 
in a multi-annual framework gives Eurostat the fl exibility to achieve its objectives as set out 
in the 5-year programme. Each year, new empha ses and adjustments to changing needs can be 
introduced into the annual programmes. As a re sult, this combination has within itself the 
potential to provide a fl exible and permanently relevant ap proach to meeting the statistical 
challenges as they change over time. 
The broad and flexible formulation of the ob jectives of the 5-year  programme also has 
another aspect to it. In the interviews and the su rvey, concerns were raised that the objectives 
are too general, and that clear-cut  priorities or strategic directi on are lacking. This is a concern 
for the NSIs because their own statistical pr ogrammes are based on the Eurostat programmes. 
Furthermore, by far the majority of the NSIs' activities derive from EU regulations, and are 
therefore obligatory. Moreover, the fact that the formulated objectives are not linked to 
quantifiable targets makes it difficult to evaluate them. 
I
NVOLVEMENT OF USERS AND PRODUCER S IN THE FORMULATION PROCESS  
In general, both the Directorates-General (DGs ) and the NSIs feel that they have been 
involved sufficiently in the formulation of the CSP 2003-2007 and that their input has been 
taken into account. On the ot her hand, they find that the process of influencing the 
programmes is difficult and often time-consuming. Both groups seem to place a great deal of 
EN 4   EN emphasis on influencing the annua l statistical programmes, as these are more specific and 
aimed at the short term. 
The two groups obviously have certain opposing in terests. The DGs want to see new requests 
implemented, whereas the NSIs focus mostly on keeping the number of requests in the 
programmes under control and on introducing mo re prioritisation. The growing demand, 
coupled with the absence of priority setting, seems to be a serious problem for the NSIs, 
because their resources often do not match the large and grow ing number of requests. The 
NSIs are constantly trying to st rike a balance between the demands  they have to meet and the 
resources at their disposal, and they are fi nding it increasingly diffi cult to achieve that 
balance. This has led to the introduction of so-called "negative priorities" - i.e. looking for ways to reduce the statistical burden by aban doning outmoded statisti cs or by simplifying 
existing obligations. 
Furthermore, the NSIs feel that new dema nds are not always dictated by necessity. 
Incidentally, according to the survey, the as pects ‘keeping the demand under control’ and 
‘burden reduction by prioritising statistics’ are the only ones where Eurostat has achieved less 
than the NSIs expected. 
2.2. Progress of activities 
All Eurostat units have reported on their ac tivity and progress towards the objectives by 
individual title in the CSP 2003-2007 by completi ng 'fact sheets' prepared by the evaluator. 
Reporting by production units and by horizontal units has been processed differently. Reporting by the horizontal
 units was added to the hierarchy of objectives which had 
previously been used to make explicit the li nk between the 5-year programme and the annual 
programmes. For the production  units the matrix shown belo w was devised (Table 1). The 
matrix consists of three core areas of action (rows) – data, regulati on/legislation, methodology 
& tools - and four major relevant Eurostat is sues (columns) – Quality, Development ESS, 
Simplification/modernisation and Enlargement – plus a residual categor y entitled ‘Regular 
activity’. The activities reported by the units have  been added up and classified in the various 
categories of this matrix. This provides a valuab le overview of the number  of activities carried 
out in each area of action and on each major issue for Eurostat. The specific content and individual actions included in this table ar e discussed in the relevant sections of the 
evaluation. 
Table 1: Categorisation and sums of  reporting on vertical fact sheets 
EN 5   EN Main area o f 
action SubareasRegular 
activity EnlargementSimpli fication / 
modernisation QualityDevelopment 
ESS Total
DataCoordinating 
surveys 8 1 1 2 3 15
Production of 
existing statistics 40 15 4 24 4 87
Production of new 
statistics 56 2 2 9 5 74
Total 104 18 7 35 12 176
Regulation/ 
legislationPreparing new 
legislation 9 0 6 5 3 23
Adopting new 
legislation 14 0 4 5 0 23
Revising old 
legislation 3 0 9 6 1 19
Total 26 0 19 16 4 65
Methodology 
& toolsPreparing new 
methodology or tool 24 0 3 6 9 42
Implementing new 
methodology or tool 13 1 4 16 17 51
Revising 
methodology 7 0 10 18 5 40
Monitoring 
methodology 2 1 3 15 5 26
Total 46 2 20 55 36 159
Overall total 176 20 46 106 52 400  
Source: Idea Consult, 2008 
Significant action has been taken on all of the pr incipal objectives, areas of activity and major 
issues identified. Unsurprisingly, most acti ons reported by the production units have been 
related to the production of statistics. The number of actions involving production of new 
statistics is quite high compared to the other activities. A large proport ion of the actions of 
both groups of units also falls under the headi ng of quality. Furthermore, many activities were 
dedicated to the development of the ESS. 
Although a fair share of the activ ities related to simplification or modernisation, only a few of 
these actions focused on data production. Simp lifications and modernisations are more a 
feature of the legisl ation and the methodology than of th e actual production of statistics. 
There is little activity to report in relation to the enlargement project, which all sources 
describe as ‘running smoothly’. Th e interviewed NSIs from new Member States stated that 
their accession had been well prepared on both the input and output si des, and that their 
integration into the ESS had gone very well. Also, as the enlargement to include 10 new countries had taken place in 2004, most of the pr eparations for enlargement had already been 
completed before the CSP 2003-2007 began. 
The NSIs generally agree that considerable pr ogress has been made to wards the objectives of 
the 5-year programme. Furthermore, the results  of the survey larg ely support the findings 
from Table 1, as the NSIs indicated that the aspects ‘production of new statistics’, 
‘development of methodology’ and ‘data accessibi lity’ were among those receiving the most 
attention during the CSP 2003-2007. However, th e NSIs felt that the CSP had paid less 
attention to all aspects related to “reduction of  burden” and aspects related to the monitoring 
and management of processes and performan ces. Encouragingly, the NSIs point out that 
greater attention is being paid to these aspects in the CSP 2008-2012. 
2.3. User satisfaction and user needs 
All indicators relating to the ‘use’ of Eurostat statistics and publications point to a significant 
upward trend. Examples of this  include the monthly number of  users of online dissemination 
and the number of downloads from the Eu rostat website, both of  which have risen 
significantly (Annual Activit y Report 2007, Annual Management Plan 2007, website 
EN 6   EN monitoring reports 2005, 2006, 2007). The trend can be seen most clearly in the number of 
hits on the website per month (performance scoreboard, 2006/2007), which increased from 
34.7 million in 2004 to 73.1 million in 2007 indicat ing every item displayed on the website. 
One of the contributory factors was the remarkable rise in th e number of requests when the 
free dissemination policy was introduced. The acce ssion of the 10 new Member States to the 
European Union also probably contributed to this growth. 
The distribution of downloads across the various statistical areas has remained stable, with a 
strong interest being shown in general soci o-economic statistics. For the period 2004-2007, 
the areas of Population & Social Conditions  and Economy & Finance accounted for more 
than 20% of the total number of downloads. Areas such as Environment & Energy, Science & 
Technology and Transport barely reached 5%. 
User monitoring reports reveal no significant in crease in the number of requests for support 
by users over the period 2004-2007 (up from 23993 to 24928). This can be regarded as a 
positive trend, since it suggests that the increase in the number of users was not followed by a big increase in the number of clients who felt that the presentation or the content of the 
website was insufficient to meet their needs. 
The main source for rating user satisfaction was the user satisfaction survey, which was 
carried out in 2007 on two groups: internet re spondents and key users. Although it should be 
borne in mind that surveys of this kind add a negative bias (as they usually attract more 
complaining, negative respondents than satisfied users), they do offer interesting insights. 
From an aggregate perspective, approximately ha lf of all users rate the overall quality of 
statistics as 'good' to 'very good'. The timelin ess aspect of the statistics is the quality 
dimension which, in general, had a relative ly higher proportion of user satisfaction. 
Completeness and accessibility r eceived somewhat lower ratings. 
This aggregate approach conceals major differe nces between statistic al areas. Price and 
employment statistics are generally rated hi ghly, whereas the regional and environmental 
statistics receive lower ratings. Also, key users tend to be slightly more positive than internet 
respondents. 
The aggregate perspective gained from the user sa tisfaction survey is rather negatively biased 
compared to quality surveys on separate statisti cal areas. Other surveys conducted for separate 
statistical areas show higher user  satisfaction ratings. A survey of international trade statistics 
shows that 85% of users are 'very satisfied' or 'f airly satisfied' with these statistics. However, 
this survey also reveals less satis faction with certain aspects, such  as comparability of the data 
(39% satisfied) and coherence between diffe rent trade data sets  (35% satisfied). 
Most of the DGs interviewed feel that Eurostat performs quite well on its core task, i.e. 
supplying other Commission Services with high quali ty statistics. All DGs regard Eurostat as 
the main authority when it comes to statistics and the first port of call when they need statistics of any kind. It is onl y in cases where Eurostat does not  have the information that the 
DGs try other options – usually  the OECD, the World Bank, the IMF or the ILO. Some DGs 
note that other sources are sometimes more policy- oriented than Eurostat  – and thus easier for 
them to use – but then those s ources can cause other problems. 
2.4. Development of the ESS and subsidiarity 
In order to gain an initial insight into the de velopment of the ESS and the application of the 
subsidiarity principle, we examined the new lega l acts in each statistical domain. We recorded 
a total of 33 new legal acts over the period 2003- 2006. The subsidiarity prin ciple is referred to 
in 11 of these acts. 
EN 7   EN The statistical requirements compendia for the years 2003 to 2007, which include overviews 
of legislative acts and data requirements, have al so been studied. Both s eem to have peaked in 
2006. Eleven extra legal acts have been added to the compendium, i.e. the number of new acts 
minus the acts that have been abolished. The increase in data requirements is very noticeable 
(up by 138). This indicates that the ESS is expanding and becoming in creasingly regulated. 
During the 2003-2007 programme period, a start was ma de on the ex ante evaluations of new 
statistical acts adopted during the progra mme period, with 12 such evaluations being 
conducted in 2007. 
By far the majority of NSIs claim that they ha ve to contend with an imbalance between users' 
demands and the resources at their disposal. The interviews and the survey show clearly that 
NSIs are suffering from the constant increa se in users needs for more and greater 
sophistication of data, together w ith the fact that they barely ha ve the resources to cope with 
this increased workload. They feel that (more) effort s hould be put into reducing the 
workload, either by limiting the dema nds or by increasing the resources. 
The DGs acknowledge this problem, but argue  that lowering the existing statistical 
requirements is complicated, as it would nega te much of the previous investment in 
developing the statistics and thei r particular features. Another consideration is that reducing 
the burden for survey respondents has been id entified as a major co ncern in the ESS too. 
Some argue that this burden is inversely pr oportional to the burden fo r the NSIs. Less work 
for respondents often means a heavier burden for the data producers (NSIs) in terms of 
processing the surveys. 
All of this goes to show how de licate this issue is, and how diffi cult the position of Eurostat is 
in this respect. The ESS can be likened to a c onstant delicate balanci ng act between the supply 
side (producers) and the demand side (users).  Those interviewed on both sides acknowledge 
that Eurostat is in a difficult position in the mi ddle of the system, and has the complex task of 
managing not only the system but also the expectations of both users and producers. 
Both the DGs and NSIs fully recognise the working groups, task forces and committees as the 
decision-taking and execu tive bodies of the ESS, are closel y involved in them and perfectly 
satisfied with the way they function. Neverthele ss, the NSIs feel that the ESS provides too 
few opportunities for discussion at a more strategi c level. The sector groups are named as the 
bodies that should be given broader mandates and greater prominence, since the role of 
working groups is primarily a technical one. 
Furthermore, both NSIs and DGs are aware of the importance of the SPC, and find its 
functioning to be increasingly effi cient and effective. In order to  improve the efficiency of the 
SPC, efforts have been made to reduce the num ber of agenda items di scussed in this body. 
Some NSIs have argued that this aspect shoul d not be exaggerated, and that the SPC should 
maintain its strong posi tion within the system. 
The DGs underline the importance of a contin uous dialogue between producers, users and 
Eurostat. Direct communication between DGs  and Eurostat normally takes place through 
personal, informal contacts, and thus is not hi ghly institutionalised. This means that a lot of 
information does not trickle down to the whole DG. This has to do with the lack of statistical 
coordination within the DGs, although the DGs poi nt out that Eurostat, too, could be more 
proactive when it comes to the dissemination/co mmunication aspects. Many of the main client 
DGs have separate direct agreements and c ontracts with Eurostat. Certainly, in the 
development phase of new statistics this can pr ovide clarity as to what each one can expect 
from the other, and avoids miscommunication. 
EN 8   EN The derogations that were gr anted during the period 2003-2007 sh ed light on another aspect 
of the development of the ESS. Derogations are basically exceptions granted to Member 
States from certain (parts of the) requirements, allowing them to not to deliver that part of the 
demand. The highest total number of derogatio ns for one country over the full programme 
period is 12; two countries have been granted 10 derogations. These figures can be supplemented by so-called "compliance monitori ng", which shows the extent to which the 
Member States are complying with a (wide) ra nge of ESS statistical regulations. Overall 
compliance can be regarded as very high. 
The number of derogations for new Member Stat es is strikingly low. At the same time, non-
compliance by the new Member States is 1.8%, as against 4.5% for the old Member States. 
One possible explanation for this is that the new Member States are not yet very familiar with 
the system of derogations. The NSIs from new Member States that were interviewed also 
acknowledge that they have a slight advantage,  because in the case of NSIs from existing 
Member States the implementation of the ESS of ten involves replacing or modifying existing 
national regulations on which all actors had reached a consensus and to which they had 
become accustomed. For the new NSIs, it simply means implementing a new Regulation, which may possibly require a shorter adaptation period. 
Eurostat's units have devoted much of their ac tivity to developing the ESS. One of the most 
relevant facets of this effort is the elabora tion and implementation of ESSnet - a system in 
which certain Member States become experts in specific statistical fields, perform specific 
tasks and share their knowledge  throughout the ESS. This initi ative is strongly supported and 
highly appreciated by the NSIs; it appears to work  well, and is an exam ple of a measure that 
has a significant and evident impact on the bur den. Another potentially important body within 
the ESS - the European Advisory Committee on Statistical Information in the Economic and 
Social Spheres (CEIES) - has been reformed to make it a more strategic body, whose aim is to 
facilitate reflection on the whole strategic programming and interaction within the ESS. 
Another interesting stage in the development of  the ESS is the agreement reached at the end 
of 2006 on the Krakow action plan. This resulted fr om the meeting of the Directors General of 
the ESS in Krakow in 2006 at whic h three task forces (TF) were  set up: (1) a TF for a basic 
legislative framework for Community Statistics (2 ) a TF for statistical challenges and (3) a TF 
for ESS resources management and programming. These TFs have  not yet reached the end of 
their mandate, but have each formulated a number of recommendations on their respective 
themes. 
2.5. Staff and internal productivity 
The smallest number of key priorities in the annual programmes related to principal objective 
4 ‘Boosting staff motivation’. However, the repor ting in the fact sheets demonstrates that the 
horizontal units have largely compensated for th is through extensive action on these priorities. 
Over the period there has been a clear increase in the number of  staff, reaching a peak in 
2007. This has meant fewer vacancies. The extra st aff were distributed relatively equally over 
the units. The percentage of staff involved in statistical production ha s increased from around 
80% to 83%. 
The gender division of the staff has remained  fairly stable across all levels (Equal 
Opportunities in Eurostat report 2007). In  both higher and middle management, around 20% 
of staff is female; at non-managerial level, the figure is closer to 30%. The fact sheets report 
that a mentoring programme for fema le administrators is in place. 
During the period under review, various actions were taken to increase the internal 
productivity of the work at Eurostat. An impo rtant factor in this context has been the 
EN 9   EN implementation of the Single Entry Point (SEP). Other examples taken from the fact sheets 
include: 
• the development and improvement of an in ternal communication strategy and tools; 
• the development of a new intranet portal; 
• the maintenance and improvement of CIRCA. 
According to various sources  (Annual Management Plan 2007, Performance scoreboard 
2006/2007), about one third of the data transmission from the Member States takes place 
through the SEP. This means that around 65% of  all transmissions ar e still taking place via 
other, mainly electronic, channels such as ma il, CD and FTP, which entails risks as regards 
confidentiality and complicates administration wi thin Eurostat. The introduction of the SEP is 
highly appreciated by the NSIs, as was clear from the intervie ws and the survey. The NSIs 
view the SEP as a useful simplification for bot h the NSIs and Eurostat. They also maintain 
that this tool is not used sufficiently  and that it should be expanded further. 
2.6. Dialogue and interaction between users and partners 
The indicator sheets show that since 2005 the number of working group meetings has 
gradually increased from 117 to 130. Satisfaction with these meetings has remained very high 
(around 87% in 2006). 
The number of grant agreements rose  from 319 (€35.148.909) in 2003 to 536 (€60.660.156) 
in 2005. In 2006, the number fell back to 428 (€53.280.474). 
An important issue in the report is the reductio n of the burden on the producers of statistics 
(NSIs). The assessment of what has been achieved in this field is to a large extent based on 
interviews and on the results of  the survey sent to the Memb er States. The working group on 
‘Coordination and Programming’ and the task force on ‘Priority Setting’ address the possibilities of reducing the burden and recomm end practical measures. These documents and 
initiatives make it clear that this is a live issue among stakeholders within the ESS and one 
which is certainly receivi ng a great deal of attenti on in the current CSP 2008-2012. 
For the time being, however, the NSI are only mode rately satisfied with the achievements in 
this area. A number of NSIs have had to make serious cutbacks in their organisation, whereas 
demand just keeps increasing. Their view is th at many more initiatives can be taken to 
simplify, modernise and just reduc e the production of statistics. 
A structural concern arising from the interviews is that there is little direct contact between 
the NSIs and DGs. The NSIs do not complain a bout Eurostat performing the difficult task of 
intermediary between the two, but they still feel that there is a lack insight into each others' 
wishes and problems, and of mutual fine-tunin g. The NSIs would like to have more contact 
with the DGs at a strategic level. They would also welcome greater involvement of the DGs in 
the development of statistics, so that DGs w ould gain a better idea of  the costs involved in 
developing and producing statistics . On the other hand, the DGs are reluctant to do this, as 
they rely on the coordinati ng work done by Eurostat. 
The mutual lack of understanding of their resp ective positions and activ ities takes concrete 
shape in the opposing views expressed by NSIs and DGs when interviewed on the issue of 
accepting and meeting new demands. In the surv ey, 56% of the NSIs state that they are 
insufficiently informed about the discussions th at are taking place between Eurostat and the 
Commission services on new statistical demands. This creates a feeling within the NSIs that 
their point of view is not being taken into account in this process. The NSIs have the 
impression that Eurostat accepts new demand s too readily, without considering all the 
implications for the statistical institutes. On the other hand, the DGs believe Eurostat is too 
EN 10   EN "soft" on the NSIs in terms of enforcing the requirements. The NSIs therefore advocate a 
more transparent and clearer process for accepting new requests and turning them into a regulation, and for paying more attention to the implications of this for their work. 
2.7. Quality of products and services 
Many of the actions listed on the fact sheets for both the produc tion units and the horizontal 
units within Eurostat fall into the "quality " category. The demand for a more in-depth and 
detailed examination of the statistics and othe r quality improvements is constant, and unlikely 
to decrease. Throughout the evaluation it can be seen that Eurostat and the ESS as a whole have put a lot of emphasis and investment into  this, in order to accommodate these requests 
and raise the overall quality level. 
Between 2003 and 2007, some 43 quality reports we re produced, and five  statistical areas 
were fully reviewed in terms of quality. Also, one third (12 out of 33) of the new legal acts 
explicitly include a quality monitoring requirement. Furthermore, the quality aspect has increasingly become a topic of  discussion in working group m eetings and on the agendas of 
the Statistical Programme Committee (SPC). 
Various initiatives have led to the so-called “Quality assurance framework”, which brings 
together a range of tools and methods to asse ss, monitor and improve statistical quality. The 
Code of Practice encompasses inst itutional and quality principles in the ESS, while the peer 
reviews performed within the ESS on the implem entation of the Code reveal that overall 
quality has improved significantly. Furthermore, th e Eurostat peer review report expresses the 
widespread appreciation on the part of the review panel and all stakeholders for the 
commitment and leadership shown by Eurostat on the quality issue. 
The Eurostat-wide effort to work on and improve the quality of statistics is undeniable. 
Therefore, it would be very interesting to assess and disclose the effects of this effort on the 
various aspects of statistical quality. Uniform, transparent a nd correct monitoring of the 
quality of statistics is obviously essential. Ho wever, it is this uniformity – in particular – 
which appears to be creating a major obstacl e to the conducting of such an assessment. 
The two main general sources on statistical qua lity - the performance scoreboard (2006/2007) 
and the quality barometer (2007) - are summaries of quality reports  for the different statistical 
areas. These reports are compiled by different units, who each perform their own monitoring 
and use differing methods and definitions. Thes e data are thus highly dispersed across 
different databases, aspects and years, and canno t be compared with any degree of accuracy. 
In order to gain an insight into the result s of all these efforts to improve quality, an 
appropriate next step would be to develop a cl ear and coherent quality monitoring framework, 
alongside the quality assurance framework. The approach and methods to measure quality 
should be the same for all parts of Eurostat, a nd should be used throughout the whole Eurostat 
organisation. 
The evaluator has nevertheless tried to summari ze some of the availabl e data on quality. For 
the Euro indicators, a number  of figures on completeness
 of the data are available. Around 
22% of the data series in the Euro indicators  do not contain values, and therefore are not 
available. Of the 78% of the data series that  are available, only 1.5% includes missing data. 
Furthermore, 96% of the Euro indicator data covers a period greate r than 5 years. The non-
response rate for the different surveys (Labou r Force Survey, Structural Business Survey, 
Structure of Earning Survey) is approximately 20%. 
Concerning timeliness , we have found figures on the fres hness of the Euro indicators which 
show that 55% are between 60 and 90 days old. The LFS data are published with an average delay of 116 days after the reference period. 
EN 11   EN Thanks to the continued attention to quality, none of the DGs question Eurostat’s position as 
the main authority and source when it comes to statistics. However, the DGs do have a lot to say about the quality of the st atistics, and have expressed a desire to see improvements. 
Ongoing work is therefore requi red in order to cope with a demand which is becoming 
increasingly sophisticated and complex. 
The interviews highlight two stru ctural concerns: both the frequency 
and the degree of detail  
of the data are often not high enough. More and more, the Commission services need monthly 
or weekly data rather than annual figures, in orde r to be able to react mo re quickly to all kinds 
of developments. Furthermore, they need data at  lower, more detailed levels, and increasingly 
even at micro level. Eurostat has difficulties de livering these data because of confidentiality 
issues - as it is often not permitted to publish individual data - and because of 
representativeness issues, because  statistical samples (and thus the number of respondents) are 
not constructed to provide robust informati on at detailed levels. Moreover, the balance 
between timeliness (frequency) and accuracy/level  of detail is a very delicate one. The more 
quickly statistics have to be produced, the less a ccurate they are likely to be. While the DGs 
obviously understand this, it neverthe less remains a problem for them. 
Other concerns that are voiced include the user friendliness of the website and NewCronos. It 
appears that the website and the NewCronos data base are difficult for new users to understand 
and use. Second, the consistency within and between databases could be improved. For 
instance, each Member State produces price statistics, but these are not available in a 
consistent form at Eurostat level. Price data  are used to produce both the HICP (harmonised 
index of consumer prices) and purchasing powe r parities. This work is done in different 
Eurostat units, with price statistics being spr ead over two different di rectorates. Therefore, 
compiling a comparable overview is a very complicated exercise. 
2.8. Monitoring of implementation 
There is no framework for monitoring and reporti ng Eurostat’s progress towards attaining the 
objectives of the 5-year program me. There is some reporting on c ontent in the annual activity 
reports (AAR), but the link betw een these annual activity reports and the 5-year programme is 
weak. This is why, at each evaluation round (mid -term, ex post), information on the progress 
of this programme has to be collected from the various units. The transp arency of Eurostat’s 
activities would be greatly increased if a strategic exercise were to be carried out which involved: 
(i) linking the (operational) objectives of the annual progr ammes to the (strategic) 
objectives of the 5-year programme, and then 
(ii) defining key actions a nd key monitoring indicators. 
3. R
ECOMMENDATIONS  
3.1. Main recommendations L
INK 
The 5-year programme is designe d as a framework programme that  sets out a strategic vision 
for the next five years. It ensures, in conjunc tion with the annual (working) programmes, that 
the ESS will continue to adopt a flexible and relevant approach to meet the statistical 
challenges that are changing over time. 
However, in order to improve the transparency of Eurostat's activ ities, and to ensure that the 
5-year programme does indeed provide the necessary reference framework within which the 
EN 12   EN various Eurostat units have to draw up th e annual programmes, these annual programmes 
should refer explicitly to  the 5-year programme. 
This explicit reference can be achieved by devisi ng a hierarchy of objectiv es, i.e. by linking 
the key operational priorities (in the themes) from the annual programmes to the strategic 
objectives and titles in the 5-y ear programme. The evaluator has provided an example of such 
a hierarchy for the horizontal objectives. 
It is also clear that, during a programme period, new developm ents in society might occur 
which the 5-year programmes cannot  anticipate, and that  this may give rise  to unexpected new 
statistical demands. In that case , the annual programmes should be  used as a way to adapt to 
these developments and to cope with the new demands. Therefore, the annual programme 
may indeed contain issues and/or  priorities that cannot be linke d to the 5-year programme. We 
believe it would appropriate to clarify the reasons for deviating from the 5-year programme in 
explicit terms, and to provide a clear explanati on as to why additional i ssues and/or priorities 
should be included in the annual programme. 
An explicit reference of this kind will ensure that the combination of the 5-year programme 
and the annual programmes is used to its full pote ntial as the main stra tegic driver behind the 
work. Moreover, a clear link between the operational priorities and the strategic objectives would allow the efficient and transparent monito ring of progress towards the objectives of the 
5-year programme. 
M
ONITORING  
Within Eurostat there are a nu mber of monitoring activities designed to closely follow up the 
various plans, including monitoring of data qu ality, user satisfaction surveys, staff opinion 
surveys, compliance monitoring, and (website ) user monitoring. These initiatives are 
definitely a sign of good governance. 
Two points merit greater a ttention. First of all, there is no overall monitoring of the 5- year 
programme or of the achievements towards its  objectives. Eurostat's progress towards 
achieving the objectives of the 5-year programme is  only reported on during the (mandatory) 
evaluation stage. Secondly, the various mon itoring activities and in itiatives are conducted 
relatively independently of each other: for ex ample, the quality aspects surveyed differ 
considerably from statistic to statistic; the various categories of statistics defined in the user satisfaction survey differ from those defined on th e website and also from those defined in the 
performance scoreboard. 
Overall monitoring of the 5-year programme can  be achieved, as already mentioned above, by 
establishing a closer link between the 5-ye ar programme and the annual programmes, and 
subsequently by choosing/defining (from amon g the many potential candidates) a few key 
indicators that provide inform ation leading to the achievement  of both the operational and - 
thanks to the link to be established - the st rategic objectives. A better streamlining of lower-
level monitoring initiatives can be achieved by  defining clearer links and greater uniformity 
between the different operati onal parts of the organisation. 
A more streamlined monitoring of the various independent initiative s, together with an overall 
“tableau de bord” to follow up Eurostat’s  activities, will not only simplify reporting 
requirements, but at the same time improve th e complementarity, transparency, comparability 
and - hence - the visibility of  several of the initiatives. 
ESS: B
ALANCING DEMAND AND RESOURCES  
The ESS can be characterised as a constant de licate balancing act betw een its producer and its 
user sides. Eurostat is at the heart of the ESS and has to fulfil the difficult task of being an 
EN 13   EN intermediary between the “main requesters” ( DGs) and the “main provid e r s ”  ( N S I s )  o f  t h e  
statistics. Many instruments (such as ESSnet , derogations, single entry point) have been 
developed and new structures (e.g. the European  Statistical Advisory Committee (ESAC)) put 
in place to support and improve the efficiency a nd effectiveness of (the providing side of) the 
ESS. The parties involved are generally satisfied with the f unctioning of the various bodies, 
such as the SPC, working groups and task forces . However, there is st ill a lot of room for 
improvement. 
On the demand side, the main trend - which is c onfirmed by the various DGs - is an increase 
in the demand for statistics (in te rms of quantity, level of detail  or quality). Increasingly, the 
Commission's services need fre quent and detailed data in orde r to be able to react to 
developments of all kinds. Despite all the effort s made to ensure efficiency and effectiveness 
on the providers' side, there is a limit to the possibilities on the supply side, where the 
situation is one of growing or  increasingly sophisticated and complicated demand. This is 
confirmed by the NSIs, who are concerned abou t the balance between the resources at their 
disposal and the statistical de mands and requirements. Moreover,  they are not very satisfied 
with how they are kept informed about the di scussions between Eurostat and the Commission 
Services on the subject of new demands for statistics 
This situation calls for either an increase in resour ces for the producers or  a strategic rethink 
of demand - and probably aspects of both. There ar e a number of channels to guide a strategic 
rethinking of demand, and these must be used  to provide a context not only for existing 
demand, but also for new demand: they include  priority setting, si mplification and burden 
reduction for existing demands and ex ante im pact assessments for new demands. To sound a 
cautionary note, there are certain forms of burde n reduction for the providers of statistics (e.g. 
enterprises) that may lead to even greater bur den creation for the produ cers (the NSIs), unless 
the various legislations are adapted accordingly. 
These subjects are delicate and difficult, and they will need time; but they are regarded as 
being of the utmost importance. The ground was prepared during the period 2003-2007 and 
this aspect is receiving a gr eat deal of attentio n in the current CSP 2008-2012. Given that 
tangible results from the working groups and task forces dedicated to thes e issues are essential 
to the sustainability of the ESS, we woul d recommend that sufficien t resources should be 
devoted to them. The concrete proposals for action from these WGs and TFs should be given 
consideration at the highest levels. 
TRUST  
Trust between the various stakeholders is funda mental to the proper functioning of the ESS. 
Eurostat is in the delicate pos ition of being an intermediary between the DGs and the NSIs. 
Fundamentally the system is made up of inte ractions between the pl ayers involved, who – to 
some extent – may have opposing interests. The ESS consists of several bodies whose 
purpose is to stimulate dialogue, and thereby create trust. In recent years, the further 
development of the ESS and the increase of th is trust seem to have gone hand in hand. The 
survey results provide an excellent illustration of this, as the growth of trust within ESS is as 
one of the aspects most highly rated by the NSIs. 
The DGs' confidence in Eurostat's ability to defend their needs vis-à-vis the NSIs, and - 
equally - the trust that the NSIs place in Eurost at to defend their aspira tions vis-à-vis the DGs, 
must be intensively nurtured in the light of  their common interest in quality statistics. 
3.2. Other recommendations 
ENLARGEMENT  
EN 14   EN The main preparations for the enlargemen t process took place before 2004 and, thus, 
enlargement during the period  2003-2007 did not pose any probl ems. The Eurostat units 
describe the enlargement process as ‘smooth’, a nd the new NSIs have likewise expressed their 
satisfaction with this process. 
However, it appears that the process has not yet reached its culminating point, and that the 
newest Member States - Romania and Bulgaria - deserve the same investment in order to be 
integrated into the system. Moreover, our own observations reveal that data for these new 
Member States are not comple te on Eurostat’s website. 
This successful integration into the ESS sh ould also be reflected, during the current 
programme period, in the availability of data fr om the new Member States. Investment in the 
integration of the new NSIs should be continued so  as to ensure that the statistics from new 
Member States are complete and availabl e to users in the (website) databases. 
The further integration of new Member States a nd extension of the availability of data from 
these countries are essential for two reasons. Firs t of all, a more complete database can only 
improve the service to users, i.e. both Commission Services and non-institutional users. 
Secondly, this will be beneficial for the develo pment of the ESS, as increasingly all members 
are becoming equal partners both in theory and in  practice. This enhances the credibility and 
legitimacy of Eurostat and the ESS as a whole. 
QUALITY  
The ESS-wide effort to work on and improve th e quality of statistics is undeniable. Eurostat 
has placed a great deal of emphasis on, and ef forts and investment into, improving quality; 
this includes the quality assurance framework, the Code of Practice an d surveys to monitor 
the quality of different statistics. Nevertheless, c ontinued efforts are still required in order to 
cope with a demand which is becoming increasing ly sophisticated and co mplicated. Thanks to 
this continuing attention to quality, all DGs re gard Eurostat as the main authority when it 
comes to statistics. Eurostat is the first source they look to when they need data. 
Monitoring the quality aspects of the different st atistics is essential to the "visibility" of 
quality. However, the various monitoring activities  are carried out by different statistical areas 
using differing methods and defini tions. For example, there are several sources to which one 
can refer in order to get an idea of data comple teness: (1) completeness of  data expressed as a 
percentage of data frequency for NewCronos as  a whole, (2) number and percentage of data 
series available; number and percen tage of data series with missi ng values and leng th of series 
in years for Euro-indicators; and (3) average pe rcentage availability of data for dissemination 
for the Structural Business Statistics in EU 25. The approach and methods used to measure 
quality should be more simila r for all parts of Eurostat, an d should be used throughout the 
entire Eurostat organisation. 
The development of a clear and coherent quality monitoring framework would be an 
appropriate next step. Such streamlined monitoring can be  defined both alongside, and 
building on, the quality assurance framewor k. Uniform (though flexible) quality indicators 
could considerably advance such  harmonisation of quality reporting. 
Harmonisation of quality reporting would improve Eurostat’s quality image even further, and 
at the same time contribute to greater credibility and transparency in its relations with users.