REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL evaluating the main findings of the pilot studies established under Regulation (EU) 2016/792 on harmonised indices of consumer prices and the house price index
Date: 2020-12-14
Institutions: European Commission, Eurostat
Subjects: EU financing, EU statistics, consumer price, data collection, harmonisation of prices, national statistics, price index, price indexing, publishing of prices, statistical method

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 14.12.2020  COM(2020) 810 final  REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL evaluating the main findings of the pilot studies established under Regulation (EU) 2016/792 on harmonised indices of consumer prices and the house price index  
 
 
1. 

BACKGROUND 

Regulation  (EU)  2016/792  of  the  European  Parliament  and  of  the  Council  on 
harmonised  indices  of  consumer  prices  and  the  house  price  index  was  adopted  in 
May 20161. This Regulation lays down a common framework for the development, 
production and dissemination of the Harmonised Index of Consumer Prices (HICP), 
the  Harmonised  Index  of  Consumer  Prices  at  Constant  Tax  rates  (HICP-CT),  the 
Owner-Occupied  Housing  Price  Index  (OOHPI)  and  the  House  Price  Index  (HPI). 
These four indices together are referred to as the ‘harmonised indices’. 

Article  8  of  Regulation  (EU)  2016/792  makes  it  possible  for  the  Commission  to 
launch  pilot  studies,  to  be  carried  out  on  a  voluntary  basis  by  Member  States, 
whenever  improved  basic  information  is  required  for  the  compilation  of  the 
harmonised indices, or when the need for improved comparability of the harmonised 
indices is identified. 

The  Union’s  general  budget  shall  contribute  to  the  financing  of  such  pilot  studies, 
where appropriate. 

These studies shall assess the feasibility of obtaining improved basic information or 
adopting new methodological approaches. 

The results of the pilot  studies  shall be evaluated by the  Commission  (Eurostat) in 
close cooperation with Member States and the main users of the harmonised indices, 
taking  into  account  the  benefits  of  having  improved  basic  information  or  new 
methodological  approaches  relative  to  the  additional  costs  of  production  of 
harmonised indices. 

The Commission shall submit a report to the European Parliament and the Council 
evaluating, if applicable, the main findings of the pilot studies by 31 December 2020 
and every 5 years thereafter. This report fulfils this obligation. 

2. 

PILOT STUDIES CONDUCTED UNDER REGULATION (EU) 2016/792 

Since  the  adoption  of  Regulation  (EU)  2016/792,  two  rounds  of  pilot  studies  have 
been completed, supported by grants from the Union’s general budget for 2016 and 
2017. Further rounds have been launched in subsequent years, but the studies funded 
under these rounds are ongoing and cannot yet be evaluated. 

2.1.  Pilot studies based on the 2016 budget 
A call for proposals2 was launched in 2016, covering the following objectives: 

Objective A: 

Automatic  linking  of  GTIN3  codes/shop-specific  codes  to 
ECOICOP4 

Objective B:   

Implementing scanner data from supermarkets or other retailers 

Objective C: 

Development  of  new  methodologies  for  processing  scanner 
data or other transaction data 

1  
2  

3  
4  

OJ L 135, 24.5.2016, p. 11. 
Restricted to national statistical authorities of the Member States and European Free Trade 
Association (EFTA) countries. 
Global Trade Item Number. 
European Classification of Individual Consumption according to Purpose. 

1 

 
 
                                                           
Objective D:  

Web scraping as a source for the HICP 

Objective E: 

Improvement of the compilation of expenditure weights for the 
HICP, including weights for internet purchases 

The first three objectives aimed to study various technical aspects related to the use 
of improved basic information derived from scanner data from supermarkets or other 
retailers.  Another  new  source  for  improved  basic  information  is  data  collected 
automatically  from  the  internet  (‘web  scraping’),  which  is  covered  by  the  fourth 
objective.  These  objectives  also  support  methodological  studies  that  improve  the 
comparability of the harmonised indices regarding the processing of these new data 
sources.  The  fifth  objective  aims  for  better  comparability  between  countries  and 
access  to  improved  basic  information  on  the  compilation  of  expenditure  weights, 
specifically  given  the  rising  share  of  purchases  on  the  internet  in  many  Member 
States. 

Grants  were  awarded  to  15  pilot  studies  in  10  Member  States  and  one  EFTA 
country. Of the 15 studies, eight related to scanner data (objectives A, B and C), six 
to  web  scraping  (objective  D)  and  one  to  expenditure  weights  (objective  E).  The 
total budget committed for these projects was EUR 1 180 835.04. 

The 2016 call for proposals also included objectives related to real estate statistics. 
However, these studies  covered subjects  that were outside the scope of  Regulation 
(EU)  2016/792  (quarterly  house  sales  indicators,  regional  house  price  indices  and 
commercial  real  estate  indicators).  The  projects  funded  under  these  objectives  are 
therefore  not  regarded  as  pilot  studies  in  the  sense  of  article  8  of  Regulation  (EU) 
2016/792. 

2.2.  Pilot studies based on the 2017 budget 
A new call for proposals5 was launched in 2017, covering the following objectives: 

Objective 1: 

Implementing transactions data in HICP compilation processes 

Objective 2:  

Adapting  the  sampling  and  compilation  methodologies  of  the 
HICP to the challenges of the digital economy 

Objective 3: 

Improvement of quality adjustment methods for the HICP 

The  first  objective  invited  further  pilot  studies  on  the  use  of  scanner  data  from 
supermarkets or other outlets, to improve the basic information and comparability of 
the  methods  underlying  the  HICP6.  Objectives  2  and  3  aimed  to  improve 
comparability  of  various  aspects  of  the  HICP  methodology,  particularly  sampling 
and quality adjustment. The increasing digitalisation of society and of the economy 
brings  challenges  for  price  measurement,  which  could  be  studied  under  these 
objectives. 

Grants  were  awarded  to  15  pilot  studies  in  nine  Member  States  and  one  EFTA 
country.  Of  the  15  studies,  five  related  to  scanner  data  (objective  1),  five  to  the 
digital economy (objective 2) and five to quality adjustment (objective 3). The total 
budget committed for these projects was EUR 1 081 642.46. 

5  
6  

Also restricted to national statistical authorities of the Member States and EFTA countries. 
As  the  HICP  at  constant  tax  rates  (HICP-CT)  is  derived  from  the  HICP,  any  improvement  in  the 
HICP will apply equally to the HICP-CT. In the remainder of this report, ‘HICP’ also encompasses 
HICP-CT. 

2 

 
 
                                                           
Like the 2016 call for proposals, the 2017 call also included objectives related to real 
estate  statistics  covering  subjects  outside  the  scope  of  Regulation  (EU)  2016/792 
(quarterly  house  sales  indicators,  regional  house  price  indices  and  commercial  real 
estate indicators). The projects funded under these objectives are therefore also not 
regarded as pilot studies in the sense of article 8 of Regulation (EU) 2016/792. 

3. 

EVALUATION OF THE PILOT STUDIES 

While the Commission  is solely responsible for the  evaluation  and approval  of the 
final  reports  of  the  pilot  studies,  the  results  are  shared  with  the  national  statistical 
institutes  of  other  Member  States  and  EFTA  countries  and  the  European  Central 
Bank,  which  is  the  main  user  of  the  HICP.  The  pilot  studies  are  presented  at 
workshops, both while in progress and when completed. The final reports are shared 
among 
through  an  on-line  application  (CIRCABC, 
Communication  and  Information  Resource  Centre  for  Administrations,  Businesses 
and Citizens). 

the  above 

institutions 

This  sharing  of  information  allows  all  national  statistical  institutes  to  take  note  of 
and learn from the experiences of others. The findings of the pilot  studies are also 
discussed  among  peers,  which  helps  them  to  identify  best  practices.  This  fosters 
common  understanding  and  common  approaches,  which  is  vital  to  improving  the 
comparability of the harmonised indices on a continuous basis.  

The pilot studies launched in 2016 and 2017 have led to a larger number of countries 
using scanner data and/or employing web scraping as data sources for the HICP. The 
number of product groups that are covered by these new data sources is also steadily 
increasing.  

Currently, 10 Member States and all three EFTA countries use scanner data to some 
extent as a data source for their HICP. In addition, a survey carried out in early 2020 
among national statistical institutes revealed that 15 Member States and two EFTA 
countries currently apply web scraping, to varying degrees, in the production of their 
HICP. 

The  major  advantages  of  the  new  data  sources  are  the  increases  in  coverage  and 
representativeness of samples that can be achieved, compared to traditional ‘manual’ 
price collection in shops, leading to more reliable measurement of price changes. At 
the  same  time,  the  new  data  sources  are  generally  less  costly  than  traditional 
methods of price collection. 

The increase in the use of new data sources is not only due to the EU-funded pilot 
studies,  as  many  countries  have  undertaken  the  necessary  testing  and  development 
work  from  their  own  budgets.  Nevertheless,  the  pilot  studies  made  a  significant 
contribution to progress in the use of new data sources for the HICP. 

It  is  less  straightforward  to  assess  the  contribution  of  the  projects  on  improved 
comparability  (objective  E  in  2016  and  objectives  2  and  3  in  2017).  However,  the 
results of most projects (e.g. improved sampling and quality adjustment procedures) 
have been, or will be, implemented in the HICP and will thereby improve the quality 
of  the  HICP  in  the  respective  countries.  The  projects  have  also  contributed  to  the 
common  pool  of  knowledge  and  experience  on  sampling  and  quality  adjustment 
procedures, so they generally promote harmonisation. For example, the findings will 
be  used  in  developing  improved  guidance  and  recommendations  on  these 

3 

 
 
procedures, including updates to the HICP Methodological Manual7 and, ultimately, 
could  lead  to  implementing  acts  in  line  with  Article  4(4)  of  Regulation  (EU) 
2016/792. 

4. 

CONCLUSIONS 

The evaluation of the findings of the pilot studies funded by the Union’s budget in 
2016  and  2017,  as  presented  in  this  report,  shows  that  they  have  contributed 
significantly  to  the  development  and  use  of  new  data  sources  in  the  HICP.  In 
addition,  several  Member  States  and  EFTA  countries  have  been  able  to  improve 
their  methodological  approaches,  thereby  contributing  to  the  store  of  common 
knowledge on these issues. In particular, the use of new data sources will improve 
the quality of the HICP while reducing their production costs. 

The  Commission  considers  the  pilot  studies  to  be  an  important  instrument  in  the 
continuous improvement of the harmonised indices and therefore intends to continue 
their funding. 

7  

See https://ec.europa.eu/eurostat/documents/3859598/9479325/KS-GQ-17-015-EN-N.pdf/d5e63427-
c588-479f-9b19-f4b4d698f2a2. 

4