COMMISSION STAFF WORKING DOCUMENT EVALUATION of the Joint Harmonised EU Programme of Business and Consumer Surveys
Date: 2024-01-25
Institutions: Directorate-General for Economic and Financial Affairs, European Commission
Subjects: EU Member State, EU candidate countries, EU programme, consumer survey, economic development, economic forecasting, economic indicator, economic survey

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 25.1.2024  SWD(2024) 27 final  COMMISSION STAFF WORKING DOCUMENT  EVALUATION  of  the        Joint Harmonised EU Programme of Business and Consumer Surveys {SWD(2024) 28 final}  
 
 
Table of contents 

1. 

INTRODUCTION ....................................................................................................... 1 

1.1 

Purpose and scope of the evaluation .................................................................... 1 

2.  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? ............. 3 

2.1  Description of the intervention and its objectives ............................................... 3 

2.2 

Point(s) of comparison ......................................................................................... 6 

3.  HOW  HAS  THE  SITUATION  EVOLVED  OVER  THE  EVALUATION 

PERIOD? ..................................................................................................................... 7 

3.1  Current state of play ............................................................................................. 7 

3.2 

Evolution over the evaluation period ................................................................... 9 

4.  EVALUATION FINDINGS (ANALYTICAL PART) ............................................. 10 

4.1.  TO  WHAT  EXTENT  WAS  THE  INTERVENTION  SUCCESSFUL  AND 

WHY? ........................................................................................................................ 10 

4.2.  HOW DID THE EU INTERVENTION MAKE A DIFFERENCE AND TO 

WHOM? .................................................................................................................... 20 

4.3.  IS THE INTERVENTION STILL RELEVANT? .................................................... 21 

5.  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? ....................... 25 

5.1.  CONCLUSIONS ....................................................................................................... 25 

5.2.  LESSONS LEARNED .............................................................................................. 26 

ANNEX I.   PROCEDURAL INFORMATION ............................................................... 28 

ANNEX II.   METHODOLOGY AND ANALYTICAL MODELS USED ..................... 30 

ANNEX  III. 

  EVALUATION  MATRIX  AND,  WHERE  RELEVANT, 
DETAILS  ON  ANSWERS  TO    THE  EVALUATION  QUESTIONS  (BY 
CRITERION) ............................................................................................................ 50 

ANNEX  IV.      OVERVIEW  OF  BENEFITS  AND  COSTS  AND,  WHERE 
RELEVANT,  TABLE  ON  SIMPLIFICATION  AND  BURDEN 
REDUCTION ............................................................................................................ 63 

ANNEX V.   STAKEHOLDERS CONSULTATION - SYNOPSIS REPORT ............... 66 

ANNEX VI.   CHANGES IN THE BCS PROGRAMME SINCE 2012 .......................... 80 

 
 
 
 
 
 
 
 
Glossary 

Term, abbreviation or 
acronym 

Meaning or definition 

API 

AT 

BCI 

BCS 

BE 

BG 

BVAR 

BLS 

CAPI 

CATI 

CAWI 

CES 

CPI 

CY 

CZ 

DE 

DG ECFIN 

DG FISMA 

DIW ECON 

DK 

DSGE 

EA 

Application Programming Interface 

Austria 

Business Climate Indicator 

Business and consumer surveys 

Belgium 

Bulgaria 

Bayesian Vector Autoregression 

Bank Lending Survey 

Computer-Assisted Personal Interviewing 

Computer-Assisted Telephone Interviewing 

Computer-Assisted Web Interviewing 

Consumer Expectations Survey 

Consumer Price Index 

Cyprus 

Czechia 

Germany 

Directorate-General for Economic and Financial Affairs 

Directorate-General for Financial Stability, Financial 
Services and Capital Markets Union 

Consulting Company of the German Institute for 
Economic Research 

Denmark 

Dynamic Stochastic General Equilibrium 

euro area 

 
 
EC 

ECB 

EE 

EEI 

EL 

ES 

ESI 

EU 

EUI 

FI 

FR 

FSSS 

GDP 

HICP 

HR 

HU 

IE 

IT 

LT 

LU 

LV 

European Commission 

European Central Bank 

Estonia 

Employment Expectations Indicator 

Greece 

Spain 

Economic Sentiment Indicator 

European Union 

Economic Uncertainty Indicator 

Finland 

France 

Financial Services Sector Survey 

Gross Domestic Product 

Harmonised Index of Consumer Prices 

Croatia 

Hungary 

Ireland 

Italy 

Lithuania 

Luxembourg 

Latvia 

MIDAS 

Mixed-data sampling 

MK 

MNE 

MT 

NL 

Republic of North Macedonia 

Multinational Enterprise 

Malta 

Netherlands 

 
 
OECD 

OLS 

PL 

PMI 

PT 

RMSE 

RO 

RS 

SE 

SI 

SME 

SK 

TR 

UK 

UN 

VAR 

VEC 

VDMA 

Organisation for Economic Co-operation and 
Development 

Ordinary Least Squares 

Poland 

Purchasing Managers’ Index 

Portugal 

Root Mean Squared Error 

Romania 

Serbia 

Sweden 

Slovenia 

Small and Medium-sized Enterprise 

Slovakia 

Türkiye 

United Kingdom 

United Nations 

Vector Auto Regression 

Vector Error Correction 

German Mechanical and Plant Engineering Association 

 
 
1. 

INTRODUCTION 

1.1 

Purpose and scope of the evaluation 

Under  the  Joint  Harmonised  EU  Programme  of  Business  and  Consumer  Surveys 
(BCS  Programme)  monthly  business  and  consumer  surveys  are  carried  out  in  all 
EU  Member  States  and  five  candidate  countries  (Bosnia  and  Herzegovina,  Ukraine 
and  Moldova,  which  were  granted  candidate  status  in  2022,  are  not  covered).  The  data 
gathered provide quasi  real-time information  on  various aspects  of  economic sentiment 
among  business  managers  and  consumers  and  are  widely  used  both  inside  the 
Commission  and  by  external  parties,  such  as  the  European  Central  Bank  (ECB), 
economic research institutes, the media, market participants, etc. The surveys are carried 
out nationally by partner institutes of the Commission, which receive grants covering up 
to 50% of the costs incurred for the implementation of the surveys.  

The  Commission  is  committed  to  evaluating  proportionately  all  EU  spending  and 
non-spending  activities  that  are  intended  to  have  an  impact  on  society  or  the 
economy  with  a  view  to  supporting  organisational  learning,  as  well  as  transparency, 
accountability  and  efficiency  in  the  allocation  of  resources.  The  last  comprehensive 
evaluation  of  the  BCS  Programme  was  completed  in  2012  and,  since  then,  the 
programme  has  been  subject  to  a  number  of  important  changes.  This  evaluation  of  the 
BCS Programme therefore covers the period from 2012 to 2021.  

The aim of the evaluation is to assess the degree to which the BCS Programme has 
achieved  its  objectives  by  using  the  Better  Regulation  criteria  of  effectiveness, 
efficiency,  coherence,  EU  added  value  and  relevance.  Several  specific  evaluation 
questions were associated with each evaluation criterion. The answers to these questions 
guided  the  evaluation  study  and  fed  the  conclusions  and  lessons  learned  from  each 
evaluation criterion. 

This Commission evaluation is supported by an external evaluation study. The study 
was  conducted  from  June  2022  to  March  2023,  with  data  collection  activities 
(stakeholder interviews, online questionnaire and literature review) completed in October 
2022. The study’s methodological approach relies on four pillars:  

1.  a  thorough  desk  research,  focussing  on  previous  evaluation  reports,  economic 
analyses  and  research  using  the  survey  data  generated  by  the  BCS  Programme 
(both  from  the  Commission’s  Directorate-General  for  Economic  and  Financial 
Affairs (DG ECFIN), as well as academia) and other outlets for the data, such as 
newspaper articles;  

2.  a  stakeholder  consultation  targeting  the  various  types  of  users  of  the  data 
(institutional users, academia, economic journalists, etc.) as well as the producers 
of  the  data  (i.e.  the  Commission’s  national  partner  institutes  conducting  the 
surveys).  In  total,  91  stakeholders  (out  of  269  contacted)  participated  in  semi-
structured interviews and 51 (out of 111 contacted) completed an online survey1;  

1 More information on the stakeholder consultation can be found in Annex V. 

1 

 
 
 
 
3.  quantitative  analysis  examining  how  accurately  and  reliably  the  survey  data 
collected by the BCS Programme capture economic developments in the Member 
States and candidate countries;  

4.  triangulation  of  the  evidence  gathered  via  the  three  preceding  methods  to  distil 

answers to DG ECFIN’s evaluation questions.  

Some  limitations  were  experienced  during  the  preparation  of  the  external  study 
and, consequently, the  completion of the Commission’s evaluation. Those included 
the  lack  of  external  evaluations  of  the  BCS  Programme  not  mandated  by  the 
Commission  and,  for  some  specific  stakeholder  groups,  a  rather  low  participation 
rate  in  the  consultation.  While  such  limitations  had  a  bearing  on  the  quality  of  some 
analyses conducted in the evaluation, the main conclusions presented in this report can be 
regarded  as  sufficiently  robust  and  reliable  as  a  basis  for  reflections  on  the  BCS 
Programme. Limitations and mitigating factors are elaborated on in Annex II. 

2 

 
 
 
 
 
2.  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? 

2.1   Description of the intervention and its objectives 

According  to  the  Treaty  on  the  Functioning  of  the  European  Union,  the 
Commission  shall  contribute  to  informing  the  EU  authorities,  Member  States  and 
the  various  economic  agents  on  the  economic  situation  at  both  national  and 
Community level. The concrete objective derived from this obligation is the assessment 
of the business cycle, for the EU as a whole and across Member States, in a timely and 
sufficiently  frequent  way.  To  allow  meaningful  cross-country  comparisons  and 
aggregation for the EU and the euro area (EA), the information should be compiled based 
on harmonised concepts and a harmonised methodology in all countries.  

In response to these requirements, the Commission set up a programme in 1961 for 
the  implementation  of  business  surveys  in  the  Member  States  according  to  a 
common methodology. While business tendency surveys deliver results  which are less 
exact  than  official  statistics  (e.g.  on  Gross-Domestic  Product  or  Industrial  Production), 
they  can  be  produced  much  faster  than  official  data  and  hence  allow  users  to  have  an 
early idea of where Member State economies and the EU/euro area economy as a whole 
are heading.  

The sectoral coverage of the BCS Programme has continuously expanded and today 
includes  manufacturing,  services,  retail  trade,  construction,  as  well  as  consumers. 
From 2006 to  2023, the programme also  comprised a Financial  Services  Sector Survey 
(FSSS)2.  The  geographic  scope  of  the  BCS  Programme  has  been  regularly  widened  to 
include  new  Member  States  as  well  as  new  candidate  countries.  The  programme 
currently  covers  all  27  EU  Member  States  and  five  EU  candidate  countries  (Albania, 
Montenegro, Republic of North Macedonia, Serbia and Türkiye)3.  

The implementation of the BCS Programme relies on inputs from the Commission 
as well as national partner institutes. The Commission defines a key set of harmonised 
methodological  features4  of  the  surveys,  namely  a  catalogue  of  harmonised  survey 
questions  to  be  asked  to  business  managers/consumers,  a  common  timetable  for  the 
surveys  and  a  list  of  best  practice  for  the  conduct  of  business  and  consumer  surveys, 

2  Based  on  the  results  of  the  stakeholder  consultation  conducted  by  the  external  contractor,  DG  ECFIN 
decided to discontinue the financial services sector survey. The survey was for the last time conducted 
in March 2023. 

3 The integration of candidate countries into the BCS Programme at an early stage is necessary to provide 
reliable  and  comparable  data  to  follow  their  economic  situation  and  to  guarantee  the  production  of 
accurate EU aggregates once these countries become members of the EU. 

4  The  harmonisation  of  the  surveys  is  only  partial  as,  apart  from  the  harmonised  questionnaire  and 
harmonised timetable, partner institutes have discretion in the specific design of the data collection and 
sampling. This includes, for example, the possibility of adding questions other than those harmonised 
by  the  EC  in  order  to  capture  some  countries’  specificities.  Partner  institutes  also  decide  on  other 
methodological aspects, such as the sample design and the sample size, whereby they are encouraged 
to follow the Commission’s ‘list of best practice for the conduct of business and consumer surveys’. 

3 

 
 
 
 
which surveying institutes are encouraged to follow. Every 4 to 6 years, it launches a call 
for proposals to select national partner institutes to conduct the surveys on its behalf. The 
work of the partner institutes is supported by grants covering up to 50% of the institutes’ 
survey-related  costs.  The  total  amount  of  grants  paid  by  the  Commission  for  the  BCS 
Programme  amounts  to  about  5  million  EUR  per  year.  At  the  end  of  each  month,  the 
Commission  summarises  the  survey  results  mainly  through  composite  (national  and 
EU/EA)  indicators,  which  are  disseminated  via  dedicated  press  releases  and  a  website 
where they can be downloaded free of charge.  

The  task  of  the  partner  institutes  is  to  conduct  the  monthly  fieldwork  and  submit 
the collected data to the Commission for further processing and dissemination. In a 
number  of  cases,  the  work  of  the  partner  institutes  is  supported  by  complementary 
national  co-financing  from  public  contributions,  membership  fees,  sponsorships  and/or 
data sales. 

4 

 
 
 
Figure 1: Intervention logic of the BCS Programme  

Source: Deloitte and DIW Econ 

5 

 
 
 
 
2.2 Point(s) of comparison  

The benchmark / point(s) of comparison to evaluate the BCS Programme cannot be the 
situation as it was before the programme was first implemented, in 1961. Over the past 
60  years,  the  economic  and  political  environment,  the  knowledge  needs  and  the 
information  basis  have  changed  radically.  This  evaluation  therefore  uses  different 
benchmarks,  namely  relevant  academic  literature,  comparable  survey  programmes  and 
the identified information gaps.  

6 

 
 
 
 
 
3.  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? 

Since  the  completion  of  the  last  evaluation  in  2012,  the  main  features  of  the  BCS 
Programme  have  remained  broadly  unchanged,  with  the  most  important  changes 
related  to  the  addition  of  new  composite  indicators  and  harmonised  survey 
questions.  This  section  starts  with  a  detailed  description  of  the  current  set-up  of  the 
programme,  which  is,  in  the  following,  complemented  by  an  overview  of  the  most 
significant changes over the period 2012-21.   

3.1 Current state of play 

The  BCS  Programme  currently  covers  all  27  EU  Member  States  and  five  EU 
candidate  countries  (Albania,  Montenegro,  Republic  of  North  Macedonia,  Serbia 
and  Türkiye)5  and  encompasses 
the 
(manufacturing)  industry,  services,  retail  trade,  construction  and  consumers.  The 
surveys are carried out at national level by public and private partner institutes, such as 
central banks, research institutes or private market research firms.  

five  sectoral  surveys 

focussing  on 

In total,  around 135,000 businesses and 32,000  consumers are surveyed across  the 
EU every month. The sample size varies across countries and sectors.  

The  business  surveys  provide  information  on  a  wide  range  of  variables  that  are 
useful to monitor cyclical developments, such as expectations with respect to output, 
selling  prices  and  employment.  There  is  a  number  of  questions  which  are  identical 
across  surveys,  as  well  as  some  sector-specific  questions,  for  instance,  on  the  current 
level  of  order  books  in  industry,  or  the  expected  orders  placed  with  suppliers  in  retail 
trade. Nearly all survey questions are of a qualitative nature, with respondents asked to 
categorise  past  or  expected  developments  in  a  given  variable  (e.g.  production)  as 
“increase”,  “no  change”,  “decrease”  and  current  levels  as  “too  large”,  “adequate”,  “too 
small”.  However,  there  are  also  some  quantitative  questions,  e.g.  the  percentage  of 
capacity  utilisation  in  industry  and  services.  Most  of  the  questions  are  asked  on  a 
monthly  basis, but  a few additional questions are added every quarter to  the surveys in 
industry, services and construction. Furthermore, there are bi-annual questions on firms’ 
investment activities in industry and services.  

The  consumer  survey  covers  monthly  and  quarterly  questions  on  households’ 
financial  situation, 
including  prices  and 
unemployment,  intentions  about  saving  and  major  purchases  as  well  as  perceived 
economic uncertainty.  

the  general  economic  situation, 

For the vast majority of survey questions, the responses of the firms/consumers are 
summarised  in  so-called  balances,  i.e.  the  difference  between  the  percentages  of 
respondents  giving  positive  and  negative  replies  to  a  given  survey  question6. 

5 The BCS Programme does not (yet) include Bosnia and Herzegovina, Ukraine and Moldova, which were 

granted candidate status in 2022. 

6  In  order  to  allow  analysts  and  researchers  to  quantify  the  survey  results  using  methods  other  than  the 
balance  statistics,  detailed  results  by  answer  categories  are  provided  online  on  DG  ECFIN’s  BCS 
website. 

7 

 
 
 
 
However,  there  are  also  a  few  questions  where  the  responses  are  condensed  into 
percentages  (e.g.  the  share  of  firms  reporting  their  production  to  be  constrained  by 
specific factors or the average capacity utilisation).    

The Commission also produces a number of composite indicators, which summarise 
the  information  emanating  from  a  selection  of  particularly  pertinent  survey 
questions, namely:      

▪  Sectoral  confidence  indicators:  These  indicators  are  arithmetic  means  of  the 
respondents’  assessments  on  selected  economic  developments  in  the  different 
sectors, as well as among consumers.  

▪  Economic Sentiment Indicator (ESI): The ESI draws on the results from all the 
business surveys and from the consumer survey. Roughly speaking, the ESI can 
be viewed as a summary of the five sector-specific confidence indicators. 

▪  Business  Climate  Indicator  (BCI):  The  BCI  is  based  on  balances  calculated 
from  the  questions  on  production  trends  in  recent  months,  order  books,  export 
order  books,  stocks  and  production  expectations  in  the  industry  survey.  The 
indicator  serves  as  a  timely  composite  indicator  for  the  manufacturing  sector  in 
the  EA  and  can  therefore  be  seen  as  a  complement  to  the  industrial  confidence 
indicator.  

▪  Employment  Expectations  Indicator  (EEI):  Since  2020, 

the  EEI  has 
summarised  managers’  employment  plans  in  the  four  business  sectors  surveyed 
(industry,  services,  retail  trade  and  construction)  and  thus  provides  a  timely 
indication of expected changes in dependent employment.  

▪  Economic Uncertainty Indicator (EUI):  Since 2021, the EUI has helped track 
the  development  of  economic  uncertainty  within  the  EU.  The  indicator  is  a 
weighted average of the answers to the questions in the four business surveys and 
the  consumer  survey  on  the  difficulties  of  predicting  the  future  economic 
situation. 

To  get  a  more  granular  view  of  developments  in  the  different  sectors  of  the 
economy, the results for the industry, services, retail trade and construction survey 
are not only available at aggregate level, but also broken down by sub-sectors7. For 
the  consumer  survey,  the  results  are  categorised  according  to  income,  occupation, 
employment regime (part versus full-time), education level, age and gender.  

The  monthly  BCS  results  are  disseminated  via  a  press  release  on  the  penultimate 
working  day  of  each  month.  In  the  particular  case  of  the  Consumer  Confidence 
Indicator, which traditionally attracts a lot of media attention, the results at the aggregate 
EU and EA levels are published already one week earlier in a flash release (around the 
20th of each month). In addition to the press releases, DG ECFIN offers all survey data 
(i.e. the historic time-series including the latest data-point) for free-of-charge download 
on its website. A sub-set of the same data is also made available in Eurostat’s database. 
Finally,  every  three  months,  DG  ECFIN  publishes  the  ‘European  Business  Cycle 
Indicators’ (EBCI), which is a report focussing on quarterly developments in the survey 

7  The  sub-sectors  correspond  to  the  “branches”  defined  by  the  Statistical  classification  of  economic 

activities in the European Community, NACE Rev. 2. 

8 

 
 
 
 
data8 and zooming in into specific topics with descriptive analysis using the survey data 
(so-called ‘special topics’).    

3.2 Evolution over the evaluation period  

Whilst  the  main  features  of  the  BCS  Programme  have  remained  unchanged  during  the 
period  covered  by  the  evaluation,  changes  have  occurred.  A  comprehensive  list  of  all 
changes to the BCS Programme since 2012 is included in Annex VI.  The most important 
changes introduced by the Commission include:   

▪  adaptation  of  survey  questions:  Introduction  of  new  questions  on  capacity 
utilisation in services  (2012), uncertainty (2021) and a change from  quantitative 
to qualitative questions on the investment plans of industry, as well as extension 
of the investment survey to the services sector (2021);  

▪ 

▪ 

▪  methodological  adjustments:  Revision  of  the  Consumer  Confidence  Indicator 
(2019),  country  weights  used  to  calculate  the  EU  and  EA  aggregates  have  been 
revised  in  cases  where  countries  joined  or  left  the  programme,  changes  in  the 
seasonal adjustment procedure (2022);  
introduction of new indicators: the Employment Expectations Indicator (2020) 
and the Economic Uncertainty Indicator (2021);  
inclusion  of  new  countries  in  EU/EA  aggregates:  Croatia  was  included  in  the 
EU  aggregates  (2013)  and  Latvia  (2014),  Lithuania  (2015)  and  Croatia  (2023) 
were  included  in  the  EA  aggregates.  Furthermore,  partially  missing  data  from 
Ireland  were  back-casted  for  the  period  2008  to  2016.  The  historical  values  as 
well  as  the  country  weights  were  revised  accordingly  (2019).  The  UK  data  was 
removed from the EU aggregates (Construction in 2019 and all other aggregates 
in 2020) following the UK's exit from the EU. 

▪  Financial Services Sector Survey (FSSS): Based on, inter alia, the results of the 
stakeholder consultation conducted by the external contractor (see section 4.3. for 
more  details),  the  Commission  decided  to  discontinue  the  FSSS  in  2022.  The 
survey was for the last time conducted in March 2023. 

8 Focussing on quarterly developments in the survey data allows to abstain from month-to-month volatility 

and often helps to better identify meaningful business cycle signals. 

9 

 
 
 
 
 
 
 
4.  EVALUATION FINDINGS (ANALYTICAL PART) 

4.1.  To what extent was the intervention successful and why?  

Main conclusions of this section 

The data generated by the BCS Programme are 

timely, frequently updated and harmonised  

▪ 
▪  used by a wide array of users  
▪  highly correlated with and thus useful to forecast GDP 
▪  not  replaceable  by  cheaper  indicators  (e.g.  based  on  big  data)  without 

lowering quality   

▪  based on a methodology which is coherent across countries/surveyed sectors 
▪  complementary to other EU and private survey programmes   

The BCS Programme could be improved by 

▪  enhancing the documentation of the underlying methodology 
▪  offering users alternative ways to download the data 
▪  simplifying  administrative  procedures  for  participating  institutes  and  more 
efficient communication with the Commission on administrative matters  

In  answering  the  question,  the  performance  of  the  BCS  Programme  has  been 
assessed  on  the  basis  of  the  evaluation  criteria  of  effectiveness,  coherence  and 
efficiency. Applied to the BCS Programme,  

▪  “effectiveness”  describes  the  programme’s  track  record  in  meeting  its  stated 
objective, namely to provide a (timely, frequently updated and harmonised) tool 
for  economic  surveillance  in  the  EU,  enabling  comparison  of  business  cycles 
between the Member States and giving  an overall view of the business  cycle in 
the Union; 

▪  “efficiency”  looks  at  the  costs  of  the  programme  and  compares  them  to  its 
benefits.  Furthermore,  it  inquires  whether  there  are  potential  alternative 
approaches to monitoring the economy at lower cost; 

▪  “coherence” depicts the extent to which the programme was: 

o 

internally  consistent,  i.e.  between  the  different  countries  and  sectors 
covered by it; 

o  externally  consistent,  i.e.  complementing  information  available  at  EU  or 

national level.   

Effectiveness in providing a tool for economic surveillance: 

The  BCS  Programme  was  and  remains  effective  in  providing  timely,  frequently 
updated and harmonised data for economic surveillance in the EU Member States 
and  candidate  countries,  as  well  as  the  EU  and  EA  as  a  whole.  Interviews  with 
stakeholders  indicate  that  academic  researchers  and  economic  and  market  analysts 
consider the survey data generated by the programme as an important tool for monitoring 
the current state of the economy. Furthermore, the forward-looking survey questions (i.e. 

10 

 
 
 
 
 
 
 
inquiring respondents’ expectations) were highlighted as helpful for predicting the future 
course  of  the  economy.  In  the  online  survey  conducted  by  the  external  contractor,  all 
surveyed users reported that the EU BCS data are an essential input for monitoring and 
now-/forecasting of economic developments in their country. 

BCS  data  are  often  quoted  in  the  media,  publicly  available  economic  analyses,  as 
well as academic articles. Google lists 1,920 results for the year 2022 when searching 
for  "Economic  Sentiment  Indicator",  while  Google  Scholar,  a  search-engine  tapping 
academic outlets, shows 150 publications citing the sentiment indicator in 2022.   

Quantitative  analysis  confirms  the  accuracy  and  reliability  of  the  BCS  data  as  a 
gauge of economic developments in EU Member States and candidate countries, as 
well as in the EU/EA as a whole.  As reported  in Table 1, the correlation between the 
programme’s  headline  index,  the  Economic  Sentiment  Indicator  (ESI),  and  real  GDP 
growth  (year-on-year)  is  consistently  above  0.5  in  all  countries  except  Bulgaria.  In  13 
countries (42%), the contemporaneous correlation is at least 0.7, implying a particularly 
strong correlation between the ESI and real GDP growth. Furthermore, for 28 countries 
(85%),  the  correlation  is  highest  when  relating  the  indicator  for  a  given  month  to  real 
GDP growth one month later (=lag 1) or two months later (=lag 2). This implies that the 
ESI leads real GDP growth and is therefore particularly useful for forecasting economic 
developments.  

Table 1: Cross-correlations between the ESI and real GDP growth 

AT 
BE 
BG 
CY 
CZ 
DE 
DK 
EA19 
EE 
EL 
ES 
EU27 
FI 
FR 
HR 
HU 
IE 
IT 
LT 
LU 
LV 

lag 3 
0.58 
0.56 
0.45 
0.73 
0.63 
0.67 
0.53 
0.64 
0.69 
0.76 
0.58 
0.65 
0.77 
0.45 
0.64 
0.58 
0.64 
0.54 
0.59 
0.45 
0.59 

lag 2 
0.71 
0.73 
0.50 
0.84 
0.72 
0.67 
0.66 
0.79 
0.74 
0.79 
0.73 
0,80 
0,80 
0.61 
0.76 
0.70 
0.68 
0.66 
0.60 
0.50 
0.65 

11 

lag 1 
0.69 
0.73 
0.43 
0.86 
0.73 
0.66 
0.61 
0.79 
0.74 
0.84 
0.72 
0.80 
0.81 
0.63 
0.76 
0.72 
0.72 
0.78 
0.58 
0.52 
0.65 

lag 0 
0.62 
0.69 
0.35 
0.86 
0.73 
0.66 
0.57 
0.75 
0.73 
0.84 
0.65 
0.76 
0.78 
0.59 
0.75 
0.68 
0.73 
0.74 
0.58 
0.54 
0.64 

 
 
 
  
MK 
MT 
NL 

PL 
PT 
RO 
RS 
SE 
SI 
SK 
TR 
UK 

0.24 
0.68 
0.74 
0.54 
0.61 
0.65 
0.25 
0.60 
0.62 
0.55 
0.65 
0.57 

0.62 
0.83 
0.85 
0.63 
0.75 
0.73 
0.59 
0.67 
0.74 
0.69 
0.70 
0.67 

0.63 
0.81 
0.83 
0.64 
0.81 
0.73 
0.68 
0.67 
0.71 
0.62 
0.67 
0.68 

0.62 
0.71 
0.79 
0.68 
0.76 
0.70 
0.58 
0.64 
0.66 
0.62 
0.71 
0.66 

Note : Lag “x” means that the indicator’s value in a given month is correlated with the value of GDP “x” 
month/s later. 
Sample sizes: AT, FI, SE (n= 327, Oct 1995-Sep 2022), BE, DE, DK, EL, FR, IE, NL (n= 456, Jan 1985-
Sep 2022), BG, CY, LT, LV, PL (n= 260, May 2001-Sep 2022), CZ (n= 336, Jan 1995-Sep 2022), EA19 
(n= 384, Jan 1991-Sep 2022), EE (n= 312, Jan 1997-Sep 2022),  ES, PT (n= 439, Jun 1986-Sep 2022), 
EU27 (n= 265, Dec 2000-Sep 2022),  HR (n= 212, May 2005-Sep 2022), HU (n= 359, Feb 1993-Sep 
2022), IT (n= 515, Jan 1980-Sep 2022), LU (n= 252, Jan 2002-Sep 2022), MK (n= 128, May 2012-Sep 
2022), MT (n= 242, Nov 2002-Sep 2022), RO (n= 228, May 2001-Sep 2022), RS (n= 116, May 2013-Sep 
2022), SI (n= 322, Mar 1996-Sep 2022), SK (n= 285, Apr 1999-Sep 2022), TR (n= 188, May 2007-Sep 
2022), UK (n= 432, Jan 1985-Dec 2020). 
Source: Estimates by DIW Econ. 

The high correlations of BCS data with GDP are particularly striking when seen in 
the  context  of  their  timely  availability.  As  Figure  2  illustrates,  there  are  alternative 
indicators  which  track  GDP  growth  even  better  (e.g.  industrial  production  in  the  upper 
right quadrant), but they are not as timely as the BCS data. Among the indicators that are, 
like the BCS data, most timely (i.e. closer to the y-axis of the graph), none are as highly 
correlated  to  GDP  as  BCS  data9.  93%  of  the  stakeholders  queried  through  online 
questionnaires agreed that the BCS data are timely enough.  

9 The only exception are HCOB’s Purchasing Managers’ Indices (PMIs), which are comparable to the BCS 
data in terms of their timeliness and correlation with GDP, but have a more limited geographical and 
sectoral coverage. 

12 

 
 
 
 
 
 
 
 
 
 
 
Figure  2:  Publication  lags  and  cross-correlations  with  GDP  growth  for  selected 
indicators 

Note:  Publication  lag  =  Publication  delay  with  respect  to  the  reference  period  of  the  indicator.  For 
example, the publication lag of the producer price index indicates the delay in publication with respect to 
the  month  to  which  the  producer  price  index  refers.  Maximum  absolute  cross-correlation  =  Maximum 
absolute value of monthly data of the current quarter with real GDP growth (y-o-y). Sample size: n = 106 
(Mar 1996 – Sep 2022) except for Flash Estimate correlations (n = 32, Sep 2014 – Jul 2022) 

Source: Estimates by DIW Econ. 

Alternative  computations  of  the  Economic  Sentiment  Indicator  (ESI)  do  not 
significantly  improve  its  capacity  to  track  GDP  growth.  The  ESI  is  a  composite 
indicator  that  combines  the  replies  to  a  fixed  set  of  questions  from  all  four  business 
surveys  and  the  consumer  survey,  weighted  by  the  importance  of  sectors  and,  when  it 
comes  to  the  EU/EA  aggregate  ESI,  the  share  of  each  Member  State  in  EU/EA  gross 
value added. The survey questions used are a mixture of backward- and forward-looking 
questions, as well as appraisals of the current situation (see Figure 3). The external study 
undertook testing for alternative computations of the EU ESI based on different  survey 
questions, with a focus on either expectations or the assessment of the current situation. 
This  work  did  not  yield  significantly  better  results  in  terms  of  correlation  to  economic 
activity (see Figure 3).  

13 

 
 
 
 
 
 
 
Figure 3: Cross-correlations between GDP growth and different ESIs  

Note: Sample size: EU-27 (n = 106, Nov 2013-Sep 2022)  
Source: Estimates by DIW Econ. 

The BCS data are a reliable source to forecast short-term economic developments at 
Member  State  and  Community  level.  In  the  context  of  the  external  study,  the 
contractor  estimated,  for  every  country,  two  forecasting  models10  for  GDP  growth,  one 
with  and one  without survey data.  As shown in  Table 2,  thanks to  the  forward-looking 
information content of the BCS data, the models incorporating survey data in addition to 
other statistical data yield consistently11 lower forecast errors (measured by the root-mean 
squared errors (RMSEs)). Barring Austria and Belgium, the reductions of the RMSEs are 
considerable, reaching up to 43% (Latvia).  

10 The models are similar to the ones used by  Behrens, Pierdzioch, and Risse  (2018) and Kholodilin and 

Michelsen (2019). 

11  France is the only exception among the countries analysed.  

14 

 
 
 
 
 
 
Table 2: RMSEs for forecasting models with and without EU BCS data 

only statistical data  statistical data and BCS data 

forecast improvement12 

AT 
BE 
CZ 
DE 
DK 
EA-19 
EE 
EL 
ES 
FI 
FR 
HU 
IE 
IT 
LU 
LV 
NL 

0.0318 
0.0331 
0.0407 
0.0293 
0.0255 
0.0352 
0.0681 
0.0526 
0.0489 
0.0378 
0.0386 
0.0450 
0.0695 
0.0423 
0.0372 
0.0699 
0.0280 

0.0297 
0.0304 
0.0282 
0.0255 
0.0147 
0.0311 
0.0410 
0.0445 
0.0423 
0.0251 
0.0400 
0.0356 
0.0480 
0.0345 
0.0314 
0.0396 
0.0223 

-6.60% 
-8.16% 
-30.71% 
-12.97% 
-42.35% 
-11.65% 
-39.79% 
-15.40% 
-13.50% 
-33.60% 
3.63% 
-20.89% 
-30.94% 
-18.44% 
-15.59% 
-43.35% 
-20.36% 

Note: Sample sizes: n = 306 (Mar 1997-Sep 2022): DE. n = 170 (Jul 2008-Sep 2022): FR. n = 130 (Nov 
2011-Sep 2022): FI. n = 110 (Jul 2013-Sep 2022): AT, BE, DK, EA19, EE, EL, ES, IE, IT, LU, LV. n = 106 
(Nov 2013-Sep 2022): CZ, HU, NL. Insufficient data for AL, BG, CY, HR, LT, ME, MK, MT, PL, PT, RO, 
RS, SE, SI, SK, TR, UK. 

Source: Estimates by DIW Econ. 

While  the  quality  of  BCS  data  was  not  disputed,  the  replies  to  the  online 
questionnaire  of  partner  institutions  and  data  user  groups  highlighted  room  for 
improvement in data dissemination. 44% of the respondents agreed that EU BCS data 
are  easily  accessible  and  available  in  an  understandable  form  and  only  7%  disagreed. 
Feedback received during the consultation provided a number of concrete suggestions on 
how to improve dissemination on the DG ECFIN website: 

▪  offering the download of the data in other formats than EXCEL to enable 
frequent users of the data to automatically download and process the data, as 
well as offering filter options to locate specific time-series more easily;  

▪  standardising  the  change  log  accompanying  every  downloadable  EXCEL 
file  in  line  with  ‘best  practice’  (changes  should  be  listed  in  reverse 
chronological order and their reasons, as well as their impact on data-quality 
should be explained);  

12 The forecasting performance is measured as the percentage change in the RMSE when the EU BCS data 
are combined with the statistical data compared to when only statistical data are used in the model. A 
negative sign indicates an improvement in the forecasting performance when using the BCS data. 

15 

 
 
 
 
 
 
▪  regularly  updating  the  national  metadata  sheets13  and  questionnaires 

which can be downloaded on DG ECFIN’s BCS website. 

▪  For more inexperienced/infrequent users of the data one could investigate:  

o 

the feasibility  of  providing  more information in the monthly  press 
releases on why certain indicators moved up/down;  

o  making  the  methodological  user  guide  more  user-friendly  by 
providing examples for analyses that can be conducted with the BCS 
data. 

Many users indicated that the Eurostat website offers a more user-friendly approach to 
disseminating EU BCS data. 

Efficiency: 

While  the  benefits  of  the  BCS  Programme  are  difficult  to  quantify  in  financial 
terms, a qualitative assessment suggests that they are substantial, have been stable 
over the evaluation period and, in all likelihood, significantly exceed the annual cost 
of the programme. First of all, 100% of the users consulted via an online questionnaire 
in the context of the external study agreed that the BCS data constitute an essential input 
for monitoring and now-/forecasting economic developments in their country. Secondly, 
the  user-base  is  quite  broad,  spanning  from  institutional  users  such  as  the  ECB,  over 
private  users  (mainly  banks  and  insurances),  to  academia  and  media  representatives. 
Third,  the  programme  provides  harmonised  survey  data  for  a  total  of  32  European 
countries, as well as the EU/EA as a whole. The closest alternative to the BCS data are 
HCOB’s  Purchasing  Managers’  Indices  (PMIs),  which  offer  harmonised  data  only  for 
eight  European  countries and, importantly, with  a more limited sectoral  coverage  (only 
manufacturing and services) and without sub-sectoral breakdowns.   

Over  the  period  2012-21,  the  Commission  paid,  on  average,  5.2  million  EUR  per 
year in grants to the partner institutes participating in the BCS survey14. The grants 
are awarded to partner institutes selected on the basis of a call for proposals which takes 
into account, inter alia, the cost effectiveness of the proposed action. As shown in Figure 
4, the annual amounts remained fairly stable throughout the period observed. At the same 
time, the core output of the programme has remained stable, while benefitting from some 
innovations,  such  as  new  survey  questions  and  indicators  being  added.  Considering  an 
average  year-on-year  inflation  of  1.2%  in  the  EU  over  the  evaluation  period,  this 
altogether  hints  at  some  efficiency  gains  in  the  programme.  These  might  stem  from 
reorganisations in the partner institutes, changes in the deployed survey modes (e.g. more 
use of online surveying), effective ex-ante and ex-post financial controls exercised by the 
Commission, technical support by the Commission, etc.  

13 The national metadata sheets provide information on how the BCS surveys are conducted in the different 
countries  (e.g.  whether  the  surveys  are  conducted  by  phone,  online,  or  differently,  how  many 
responses are collected each month, whether the answers of the respondents are weighted to optimise 
their representativeness). 

14 193,000 EUR thereof were spent on the centralised Financial Services Sector Survey.  

16 

 
 
 
 
Figure 4: Annual nominal cost (grants) of the BCS Programme 2012-21 

Source: Compiled by DIW Econ based on cost data for the period 2012-2021 provided by DG ECFIN 

A literature review by the external contractor, in combination with the results of the 
stakeholder  consultation,  concluded  that  innovative  and  cheaper  approaches  to 
understand  the  business  cycle  based  on  big  data  come  with  too  many  technical 
limitations  and  could  therefore  not  replace  survey  data.  First  of  all,  these  data  are 
very  often  incomplete  in  the  sense  that  they  do  not  cover  the  entire  economy.  Online 
information on prices, for instance, is only available for a sub-set of goods and services 
offered  in  an  economy.  Second,  the  data  tend  to  be  very  volatile  which  obscures  the 
economically relevant signals contained in the data. Third, the availability and quality of 
online  information  differs  across  Member  States,  making  it  hard  to  develop  indicators 
allowing  for  meaningful  cross-country  comparisons.  Finally,  the  data  tend  to  be 
backward-looking,  i.e.  allowing  to  understand  past  economic  developments  but  not  to 
predict future ones.   

For  consumer  surveys,  the  external  study  concluded  that  no  unequivocal 
recommendation could be made for partner institutes to replace  interviewer-based 
survey  modes  (currently  deployed  by  94%  of  them)  by  cheaper  self-administered 
techniques,  such  as  online  questionnaires.  While  the  available  literature  shows  the 

17 

 
 
 
 
 
social desirability bias to be smaller in web-surveys15, the opposite seems to apply to non-
response bias16. Furthermore, there is conflicting evidence with respect to whether web-
surveys or interviewer-based surveys produce higher response rates17. In the stakeholder 
consultation,  a  number  of  partner  institutes  expressed  their  reservation  against  web 
surveys,  invoking  inter  alia  a  possible  decline  in  response  rates,  and  thus  lower 
representativeness of survey results.   

In  terms  of  the  financial  administration  of  the  programme,  the  stakeholder 
consultation  showed  divided  views.  While  a  number  of  partner  institutes  seem  to 
appreciate  recent  changes  aimed  at  simplification,  such  as  the  submission  of  a  budget 
based on average unit costs per staff category and a flat-rate for administrative costs and, 
as  a  consequence,  lighter  reimbursement  procedures,  several  partner  institutes  reported 
problems.  As  the  amounts  indicated  in  the  simplified  budget  must  be  based  on  costs 
incurred in the past, they cannot incorporate inflation, which is particularly problematic 
in the current high-inflation context. Furthermore, partners regret that, whenever they add 
a  non-harmonised  survey  question  to  their  questionnaire,  the  share  of  the  survey  costs 
reimbursed  by  the  Commission  decreases,  since  the  Commission  applies  a  ratio  to 
partners’ total survey costs to ensure that only costs related to the harmonised questions 
benefit from EU co-funding. While showing awareness of the limitations set by the EU 
Financial  Regulation,  several  of  the  Commission’s  partner  institutes  were  calling  for 
(further) simplification of cost estimation and paperwork and improving the efficiency of 
communication  between  the  Commission  and  its  partners  on  financial/administrative 
matters, e.g. through the use of a centralised communication platform for the exchange of 
knowledge on financial-administrative aspects of the programme.  

Coherence: 

The  BCS  Programme  has  a  high  degree  of  internal  coherence.  First  of  all,  across 
countries, a detailed analysis of a representative sample of national questionnaires by the 
external  contractor  showed  the  harmonised  survey  questions  to  be  correctly  translated 
into  the  national  languages.  The  few  deviations  found  are  for  idiomatic  reasons. 
Secondly, across the four sectoral surveys, the set of harmonised survey questions is not 
identical, but has a sufficient degree of coherence. All sectoral surveys feature a question 
on  price  and  on  employment  expectations,  as  well  as  developments  in  output.  The 
presence of some sector-specific questions (e.g. the stock of finished products in industry 
or  the  expected  orders  placed  with  suppliers  in  retail  trade)  is  justified  by  the  different 
ways in which the sectors operate.  

The  BCS  Programme  is  also  externally  coherent,  i.e.  with  respect  to  other  EU 
survey  programmes,  such  as  the  ECB’s  Consumer  Expectations  Survey  (CES),  as 

15  See  Felderer,  Kirchner,  &  Kreuter  (2019),  Kreuter,  Presser,  &  Tourangeau  (2008),  Braunsberger, 

Wybenga, & Gates (2007). 

16 See Felderer, Kirchner, & Kreuter (2019), Mackeben & Sakshaug (2022). 

17 Higher response rates for interviewer-based surveys are found by United Nations (2015), Jäckle, Lynn, 
&  Burton  (2015),  Felderer,  Kirchner,  &  Kreuter  (2019),  while  lower  response  rates  by  Olson,  et  al. 
(2021), Mackeben & Sakshaug (2022). 

18 

 
 
 
 
well as comparable private survey programmes.  The CES is  a monthly online panel 
survey of consumers, which was piloted in 2020. As the CES only covers a sub-set of EA 
countries,  results  from  the  Commission’s  consumer  survey  are  a  valuable  complement, 
this even more so as the CES has a focus on quantitative and the Commission survey on 
qualitative questions. When compared to the Purchasing Managers’ Index (PMI), which 
is one of the most widely used indicators of business sentiment, the BCS Programme is, 
again,  a  useful  complement,  offering  full  coverage  of  EU  Member  States,  a  broader 
sectoral  scope,  which  includes  retail  trade,  construction  and,  importantly,  sentiment 
among  consumers,  sub-sectoral  breakdowns  of  the  survey  results,  as  well  as  different 
time-horizons  of  the  survey  questions18.  By  contrast,  the  Commission’s  Financial 
Services  Sector  Survey  (FSSS)  turned  out  to  provide  little  additional  information 
compared to the ECB’s bank lending survey and was therefore, also with a view to the 
external coherence of the BCS Programme, discontinued in March 2023 (see section 4.3. 
for additional reasons for the termination of the FSSS). 

When effectiveness, efficiency and coherence are all taken into account, the overall 
assessment of the BCS Programme over the period 2012-21 is positive. The analysed 
evidence  shows  that  the  BCS  Programme  provides  timely,  frequently  updated  and 
harmonised  data  for  economic  surveillance  in  the  EU  Member  States  and  candidate 
countries,  as  well  as  the  EU  and  EA  as  a  whole.  It  could,  however,  profit  from  a  few 
changes aiming to make the data-dissemination more user-friendly. The BCS Programme 
is  assessed  as  efficient  in  terms  of  value  for  money.  The  grants  are  awarded  to  partner 
institutes selected on the basis of a call for proposals which takes into account, inter alia, 
the cost effectiveness of the proposed action. Over the period 2012-21, the annual grants 
paid  by  the  Commission  to  the  partner  institutes  in  exchange  for  collecting  the  survey 
data have remained broadly unchanged, implying that they even decreased in real terms. 
This  hints  at  further  efficiency  gains  in  the  programme.  While  the  financial 
administration  of  the  programme  is  perceived  as  cumbersome  by  a  number  of  partner 
institutes, there is widespread understanding of the constraints posed by the EU Financial 
Regulation.  Furthermore,  the  BCS  Programme  is  externally  coherent,  serving  as  a 
valuable complement to  similar EU and private survey programmes. While the internal 
coherence analysis shows some potential limitations, the evidence suggests that the level 
of cross-sectoral comparability is fully sufficient.   

18 The PMI asks how the indicators have changed this month compared to the previous month, while the 
BCS  survey  questions  focus  on  the  last/next  3  months,  the  last/next  12  months  and  the  current 
situation.  

19 

 
 
 
 
 
 
 
 
 
4.2.  How did the EU intervention make a difference and to whom? 

Main conclusions of this section 

The main added value of the BCS Programme is that it ensures 

▪ 
the presence of business and consumer surveys in all EU Member States 
▪  a high degree of harmonisation of the surveys across countries, allowing for 
meaningful cross-country comparisons and generation of EU/EA aggregates  

The main value added of an EU-funded BCS Programme is that (i) it ensures there 
are business and consumer surveys in all EU Member States and (ii) there is a high 
degree  of  harmonisation  in  surveys  across  countries  which  allows  for  meaningful 
cross-country  comparisons  and  the  generation  of  EU/EA  aggregates.  Indeed,  as 
regards  harmonisation,  the  preceding  analysis  on  internal  coherence  has  concluded  that 
the programme ensures high cross-country comparability of the resulting survey results.  
It  acts as an important tool for monitoring the current state of the economy, covering a 
wide  geographic  spread  in  a  consistent  manner.  The  surveys  thus  provide  a  timely, 
robust,  reliable  and  comparable  source  of  data  to  track  and  forecast  economic 
developments  across  the  EU,  euro  area,  as  well  as  in  individual  Member  States  and 
candidate countries.  

The added value of the programme accrues to a variety of different actors, namely 
the  economic  press  and,  in  particular,  news  agencies  specialised  in  economic  news, 
economists in public institutions such as the ECB and in the private sector using the data 
to  track  and  forecast  economic  developments,  as  well  as  academia,  which  harness  the 
data  for  economic  research.  Ultimately,  the  data  also  benefit  policy-makers  whose 
decisions are informed by the above-mentioned forecasts and research. 

As shown in the stakeholder consultation conducted for the external study, the vast 
majority  of  the  users  of  BCS  data  believe  they  result  from  an  EU  rather  than  a 
national  effort  (70%  vs.  15%).  The  BCS  Programme  thus  has  a  high  degree  of  EU 
visibility.  The  perception  seems  to  be  largely  determined  by  the  way  users  source  the 
data: if they are downloaded from DG ECFIN’s or EUROSTAT’s website, they are taken 
for ‘EU’data, the opposite holding true if they are taken from the website of the national 
partner institutes.     

20 

 
 
 
 
 
 
 
 
 
 
 
 
 
4.3. 

Is the intervention still relevant? 

Main conclusions of this section 

The BCS Programme continues to be relevant since 

the survey questions focus on relevant economic concepts 
it produces new data at sufficient frequency (monthly) 

▪ 
▪ 
▪  competing  data  sources  are  inferior  in  terms  of  frequency,  timeliness  or 

(sub)sectoral/country coverage  

The BCS Programme could be rendered more relevant by 

▪  discontinuing the FSSS which is not widely used (Commission abolished the 

survey in March 2023) 

▪  occasionally including harmonised ad hoc questions on topical issues in the 

surveys 
reporting the survey results along additional classifications 

▪ 

In  spite  of  a  number  of  potentially  competing  data-sources  having  emerged  in 
recent  years,  the  BCS  Programme  remains  unique  and  highly  relevant.  First, 
EUROSTAT’s  introduction  in  2016  of  a  preliminary  flash  estimate  of  (EU/EA)  GDP, 
which gets released already 30 days after the reference quarter, did not render the BCS 
data  obsolete.  The  BCS  data  are  still  published  more  frequently  (monthly  instead  of 
quarterly) and in a timelier manner (by the end of the respective reference month) than 
national  accounts  data.  Hence, 
indications  about  economic 
developments in a given quarter as of the end of its first month, when the release of the 
preliminary  flash  is  still three  months  away.  Importantly,  the  survey  information  is  not 
only available at total, but also at sectoral and sub-sectoral level, while the preliminary 
flash GDP estimate is only available at the aggregate level. Second, the relevance of the 
BCS data is unaffected by the presence of alternative cross-sectoral survey programmes 
and, in particular, the well-known Purchasing Managers’ Index (PMI). In contrast to the 
latter,  the  BCS  Programme  has  a  complete  coverage  of  the  EU  Member  States  and 
provides  information  on  additional  sectors,  namely  construction,  retail  trade  and 
consumers.  

they  provide  first 

The  continued  relevance  of  the  BCS  data  also  showed  in  the  online  consultation 
conducted in the context of the external study. 100% of the participating data-users, as 
well as 91% of the partner institutes agreed/strongly agreed that “the programme is still 
relevant in the light of recent progress in accelerating the release of important statistical 
data,  such  as  EU/euro  area  GDP  (preliminary  flash  estimate)  and  the  availability  of 
alternative short-term indicators (e.g. big data)”. 

Deviating  from  the  core  of  the  BCS  Programme,  the  Financial  Services  Sector 
Survey (FSSS) was found to be redundant. During the stakeholder consultation, none 
of the interviewed users of the BCS data could provide feedback on the FSSS, since they 
did  not  use  FSSS  data  at  all.  This  finding  was  corroborated  by  DG  ECFIN  when 
consulting  with  colleagues  working  on  monetary/financial  matters  in-house,  in  DG 
FISMA  and  the  ECB  for  their  awareness  and  use  of  the  FSSS.  Several  central  and 

21 

 
 
 
 
 
 
 
commercial  banks  have  now  developed  their  own  financial  sector  surveys,  which  are 
tailored  to  their  specific  needs,  and,  hence,  reported  to  have  no  use  for  the  FSSS.  The 
same  stakeholders  also  highlighted  that  information  provided  by  the  financial  sector  is 
highly sensitive and firms operating in the sector are, in general, reluctant to share it with 
external  parties.  Given  its  apparent  ineffectiveness,  the  Commission  decided  to 
discontinue the FSSS. The survey was for the last time conducted in March 2023. 

According  to  the  online  consultation,  there  is  broad  agreement  among  data-users 
(89%)  and  the  partner  institutes  (96%)  that  the  harmonised  survey  questions 
featuring in the BCS Programme focus on relevant economic concepts. The addition 
of a harmonised question on capacity utilisation in services (2012), as well as a question 
on the perceived level of uncertainty in all surveyed sectors (2021) seem to have further 
enhanced  the  relevance  of  the  programme.  However,  the  partner  institutes  consulted 
highlighted  the  trade-off  between  the  additional  information  brought  about  by  new 
survey questions and a possible dampening effect of a longer questionnaire on response 
rates.  If  a  question  was  to  be  added  in  the  current  economic  context,  the  stakeholder 
consultation  indicates  that  it  should  be  a  follow-up  to  the  question  on  factors  limiting 
production/activity  in  industry/construction  which  would  allow  to  explore  whether 
reported  shortages  of  materials  are  due  to  disruptions  in  the  supply  chains  or  other 
reasons.  

The surveying frequency of the BCS Programme, with the bulk of questions asked 
monthly  (and  a  few  additional  ones  every  three  months  or  bi-annually),  appears 
appropriate  in  the  light  of  the  United  Nations  Handbook  on  Economic  Tendency 
Surveys. The online survey conducted in preparation of the external study showed 100% 
of the users and 87% of the partner institutes to be satisfied with the surveying frequency. 
Stakeholder interviews furthermore highlight that increasing the frequency of the surveys 
would increase the response burden on participants and, hence, risk lower response rates. 
Furthermore, a higher frequency is associated with a likely increase in volatility, which 
makes the data harder to interpret. The consultation of media representatives, moreover, 
indicates that the press only has limited interest  in more frequent updates of the survey 
data. The only frequency increase advocated by some of the partner institutes is to render 
the (few) harmonised quarterly questions monthly, the rationale being a simplification of 
the survey process with only one version of the questionnaire, rather than different ones, 
depending on the month.  

The  sectoral  reporting  of  the  business  survey  data  (industry,  services,  retail  trade 
and construction) meets users’ needs. 96% of the consulted users consider the current 
classification of results appropriate. Importantly, many users cautioned that replacing the 
current  classification  scheme  by  a  new  one  should  be  avoided  since  it  would  require 
overhauling the existing forecasting models. Nevertheless, a few stakeholders argued that 
the current classification could be complemented by an additional one following the EC 
industrial ecosystem approach. The idea of the latter is to group players operating in the 
same value chain, which implies, inter alia, that service providers and suppliers may be 

22 

 
 
 
mixed19.  As  advanced  by  the  proponents  of  such  an  additional  classification  scheme,  it 
would  not  only  allow  for  different  economic  analyses,  but  might  also  be  more 
intuitive/understandable to the general public.  

The  level  of  disaggregation  at  which  the  survey  results  can  be  accessed20  is 
considered  appropriate  by  89%  of  the  participants  in  the  online  consultation.  As 
regards  the  business  surveys,  a  common  view  is  that  a  finer  breakdown  of  the  results 
would result in very small samples (i.e. there would only be very few responding firms 
per  category)  and,  as  a  consequence,  very  volatile  time-series  and  statistically  non-
significant  results.  Additionally,  a  more  detailed  breakdown  could  raise  confidentiality 
issues  as  the  totality  of  the  information  (country,  activity  sub-sector,  company  size) 
might make it possible to identify the responses of individual firms.      

Allowing  for  ad  hoc  questions  on  topical  issues  to  be  added  on  a  temporary  basis 
could  make  the  BCS  Programme  more  relevant.  As  highlighted  by  the  stakeholder 
consultation, ad hoc questions are currently a non-harmonised activity (i.e. conducted in 
individual countries only, on the initiative of individual partner institutes and without any 
EU  co-funding).  The  current  framework  partnership  agreements  already  allow  for  EU-
funded ad hoc questions. The decision whether to ask harmonised ad hoc questions or not 
and, in particular, their number should take into account the trade-offs highlighted above, 
between  additional  relevant  information  and  the  potential  negative  effect  of  a  longer 
questionnaire on response rates.  

The relevance of the BCS Programme is underscored by the fact that it has become 
an  international  reference  for  survey  programmes  around  the  world,  as  evidenced 
by  Table  3  which  lists  countries  having,  at  least  partially,  adopted  the  methodology 
underlying the BCS Programme. 

19  “Mobility,  transport,  automotive”  is  an  example  of  such  an  ecosystem,  combining,  inter  alia,  logistics 
enterprises and car manufacturers. More information on the EC ecosystem approach can be found in 
the Commission’s 2020 Communication on a New Industrial Strategy for Europe. 

20 For the business surveys: two-digit level of the EC classification of economic activities (NACE, Rev. 2); 
for the consumer survey: along a number of socio-demographic characteristics such as age, income or 
gender.  

23 

 
 
 
 
 
 
 
 
 
 
 
Table  3:  Overview  of  adoption  of  the  European  Commission's  business  and 
consumer survey methodology, in whole or in part 

Region 

Europe  

Countries  

EU-27  &  UK,  EU  candidate  countries  (Albania, 
Montenegro,  Republic  of  North  Macedonia,  Serbia, 
Türkiye),  Georgia,  Kosovo,  Norway,  Russia, 
Switzerland, Ukraine   

Americas 

Brazil, Canada, US, Chile 

Africa 

Asia  

South Africa  

China, Indonesia, Israel, India, Japan, South Korea 

Asia & Oceania 

Australia, New Zealand  

Source: Deloitte and DIW Econ 

24 

 
 
 
 
 
5.  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? 

5.1.  Conclusions 

The  overall  assessment  of  the  BCS  Programme  along  the  criteria  of  effectiveness, 
efficiency, coherence, value added and relevance is very positive. 

The  programme  has  so  far  been  effective  in  providing  timely,  frequently  updated 
and  harmonised  data  for  economic  surveillance  in  the  EU  Member  States  and 
candidate countries, as well as the EU and EA as a whole. The data are widely used 
by  academic  researchers  and  professional  economists  to  monitor  and  now-/forecast 
economic  developments.  They  are  also  frequently  cited  in  the  media.  The  survey  data 
generated  by  the  programme  are  highly  correlated  with  GDP  and  provide  a  significant 
benefit  in  forecasting  models.  While  the  quality  of  the  data  generated  by  the  BCS 
Programme  is  undisputed,  there  is  room  for  improvement  in  respect  of  their 
dissemination,  namely  in  terms  of  enhanced  documentation  of  the  programme’s 
methodology and the ways in which the survey data can be downloaded. 

The BCS Programme has been efficient with its substantial benefits to a wide user-
base  in  all likelihood  significantly  exceeding  the  annual  cost  of  the programme.  In 
the  online  consultation  conducted  by  the  external  contractor,  100%  of  the  participating 
users  agreed  that  the  BCS  data  constitute  an  essential  input  for  monitoring  and  now-
/forecasting economic developments in their country. There is no scope for replacing the 
survey data by cheaper indicators based on big data approaches, as the latter come with 
too  many  technical  limitations,  such  as  an  incomplete  coverage  of  the  economy, 
substantial  short-term  variation  obscuring  the  economically  relevant  signal,  etc. 
Similarly, in the context of the consumer surveys, moving from the currently dominating 
survey modes (phone and face-to-face) to cheaper online surveying is not advisable since 
online  surveys  are  associated  with  a  higher  non-response  bias  and  there  is  conflicting 
evidence  in  the  literature  on  whether  they  produce  higher  or  lower  response  rates  than 
interviewer-based surveys. In the stakeholder consultation, a number of partner institutes 
expressed their reservation against web surveys due to the risk of lower response rates. 
At  the  level  of  the  financial  administration  of  the  BCS  Programme,  the  Commission’s 
partner  institutes  showed  awareness  of  the  limitations  set  by  the  EU  Financial 
Regulation. At the same time, there seems to be room to further simplify cost estimation 
and paperwork and improve the  efficiency of communication between the Commission 
and  its  partners  on  financial/administrative  matters,  which  should  be  further  reflected 
upon. 

The  programme  has  a high  degree  of  internal  coherence,  i.e.  across  countries  and 
economic actors surveyed. The programme is also externally coherent, i.e. with respect 
to other EU as well as private survey programmes, capturing complementary aspects of 
economic  developments  and  filling  geographic  and  sectoral  gaps  in  the  other 
programmes.  

The EU value added of the programme is such that it allows for meaningful cross-
country comparisons and the generation of EU/EA aggregates. The surveys provide a 
timely,  robust,  reliable  and  comparable  source  of  data  to  track  and  forecast  economic 

25 

 
 
 
developments  across  the  EU,  euro  area,  as  well  as  in  individual  Member  States  and 
candidate countries.  

The  BCS  Programme  remains  relevant  in  spite  of  a  number  of  potentially 
competing data-sources having emerged in recent years. In particular, EUROSTAT’s 
introduction of a preliminary flash estimate of EU/EA GDP, which gets released 30 days 
after  the  reference  quarter,  did  not  render  the  BCS  data  obsolete,  as  the  latter  continue 
being  more  timely  and  provide  a  (sub-)sectoral  breakdown.  By  contrast,  the  financial 
services sector survey turned out to be little known to potential users and was therefore 
discontinued in March 2023. According to the stakeholder consultation, the harmonised 
survey questions of the BCS Programme focus on the most relevant economic concepts 
and  the  monthly  frequency  of  the  programme  is  appropriate.  To  render  the  programme 
even  more  relevant,  the  Commission  might  consider  additional  classifications  along 
which to report the survey results as well as ways of facilitating the occasional inclusion 
of harmonised ad hoc questions on topical issues in the surveys.   

5.2.  Lessons learned 

The  above-mentioned  conclusions  provide  a  basis  for  further  reflection,  but  there 
are  also  a  number  of  concrete  ‘lessons  learned’  based  on  feedback  received  to 
stakeholder consultation which could be acted upon in the future. 

1.  Data  dissemination:  To  improve  the  effectiveness  of  the  data  dissemination  on 

the DG ECFIN website, a variety of improvements could be considered:  

a.  offer the download of the data in other formats than EXCEL  to enable 
frequent users to automatically download and process the data, as well as 
offering filter options to locate specific time-series more easily; 

b.  regularly update the national metadata sheets and questionnaires which 

can be downloaded on DG ECFIN’s BCS website; 

c.  standardise  the  change  log  accompanying  every  downloadable  data-file 
to bring it in line with ‘best practice’ (changes should be listed in reverse 
chronological  order  and  their  reasons,  as  well  as  their  impact  on  data-
quality should be explained); 

d.  make the methodological user guide available on the BCS website more 
user-friendly  by  providing  examples  of  analyses  that  can  be  conducted 
with the BCS data; 

e.  inquire  the  feasibility  of  providing  more  information  in  the  monthly 

press releases on why certain indicators moved up/down. 

2.  Easing  the  administrative  burden  of  the  programme:  To  lower  the  time  and 
effort  that  partners  have  to  invest  to  ensure  compliance  of  their  annual  budgets 
and final financial statements with the programme’s financial guidelines and the 
EU Financial Regulation, a variety of adjustments could be considered, including:  
a.  create  a  centralised  communication  platform  for  the  exchange  of 

knowledge on financial-administrative aspects of the programme; 

b.  further simplify cost estimation and paperwork for the administration of 

the programme. 

26 

 
 
 
 
3.  Coverage of the programme21: To ensure the BCS Programme remains relevant, 

the following adjustments could be considered: 

a.  fostering  the  use  of  harmonised  ad-hoc  survey  questions  on  topical 
issues,  taking  into  account  the  trade-off  between  additional  relevant 
information and the potential negative effect of a longer questionnaire on 
response rates; 

b.  introducing  a  supplementary  classification  scheme  of  business  survey 
results  based  on  the  Commission’s  ecosystem  approach,  which  groups 
together players operating in the same value chain.  

21  Based  on  the  results  of  the  stakeholder  consultation  conducted  by  the  external  contractor,  DG  ECFIN 

decided to discontinue the FSSS. The survey was for the last time conducted in March 2023. 

27 

 
 
 
 
 
ANNEX I.   PROCEDURAL INFORMATION 

In  2021,  the  Commission’s  DG  ECFIN  took  the  initiative  for  a  new  evaluation  of  the 
Joint  Harmonised  EU  Programme  of  Business  and  Consumer  Surveys  (BCS 
Programme).  The  initiative  aimed  to  fulfil  the  Commission’s  commitment  to  evaluate 
proportionately all EU spending and non-spending activities that are intended to have an 
impact  on  society  or  the  economy  with  a  view  to  supporting  organisational  learning, 
transparency  and  accountability  and  the  efficient  allocation  of  resources.  With  the  last 
evaluation  completed 
the 
implementation of the BCS Programme from 2012 to 2021. The Decide planning entry 
for  the  evaluation  is  PLAN/2021/11500,  with  the  evaluation  roadmap  running  between 
30 July 2021 and 27 August 2021, to seek wider feedback.  

the  new  evaluation  focussed  on 

in  January  2012, 

In  line  with  the  requirements  set  out  in  the  2021  Commission  Better  Regulation 
Guidelines, the evaluation considered the criteria of effectiveness, efficiency, coherence 
(both internally and with respect to comparable EU and private survey programmes), EU 
value  added  and  relevance.  The  exercise  relies  primarily  on  an  external  independent 
study  commissioned  by  the  Commission  in  May  2022  and  concluded  in  May  2023.  In 
order to ensure validity, the analysis and conclusions of the external study are based on 
the  evidence  obtained  using  several  evaluation  methods  (desk  research  focussing  on  a 
wide variety of relevant documents, a targeted stakeholder consultation using both semi-
structured  interviews  and  online  questionnaires,  a  quantitative  analysis  to  evaluate  the 
quality of the survey data generated by the programme, as well as thorough triangulation 
of the evidence gathered via the different evaluation methods). A number of limitations 
were experienced during the preparation and completion of the study. Annex II provides 
a  detailed  overview  of  these  shortcomings,  together  with  the  mitigation  strategies 
adopted. 

The lead DG to carry out and manage the evaluation has been DG ECFIN. DG ECFIN 
also  chaired  the  inter-service  group  (ISG)  that  was  set  up  to  manage  the  external 
evaluation.  Apart  from  DG  ECFIN,  the  ISG  comprised  representatives  of  other 
Commission  services  (the  Secretariat  General,  the  Directorate-General  for  Internal 
Market, Industry, Entrepreneurship and SMEs, as well as EUROSTAT). The ISG had the 
responsibility to:  

a.  Establish the Terms of Reference (ToR);  
b.  Facilitate the evaluator’s access to the information needed;  
c.  Advise, monitor and comment on the work undertaken by the external contractor. 

The call for tenders was launched in December 2021 and five applications were received. 
Following the verdict of an evaluation committee, the specific contract to undertake the 
external  evaluation  was  awarded  to  a  consortium  led  by  Deloitte  under  Framework 
Contract  number  2018/RTD/A2/OP/PP-07001-2018  -  Lot  2.  Besides  Deloitte,  the 
consortium included DIW ECON. The total amount paid under this specific contract was 
EUR 146,800.00 covering all tasks executed.   

A kick-off meeting, where the ISG and the external contractor discussed the deliverables 
and  the  evaluation  methods,  took  place  in  May  2022.  Meetings  on  the  inception  and 
interim reports were held in, respectively, June 2022 and September 2022.  

28 

 
 
 
Throughout the evaluation process, unit A3 of DG ECFIN, which is in charge of the BCS 
Programme,  provided  support  to  the  contractor  by  responding  to  requests  for 
clarification,  facilitating  access  to  relevant  documents,  etc.  and  closely  monitored  the 
quality of the information and analysis used to inform the external report. 

The draft final report was first submitted in October 2022. Subsequently, the report was 
revised seven times, before a final version was approved in May 2023.  

At the conclusion of the  external  study, the  ISG completed  a quality  assessment  of the 
final report. According to the assessment, the study addressed all key issues as required 
in the Terms of Reference, although some of the underlying analysis lacked depth (This 
was  addressed  in  the  SWD).  The  evaluator  provided  all  requested  deliverables  in  line 
with the work plan. Some of the deliverables required several iterations before the quality 
was  acceptable  to  the  ISG.  The  assessment  concludes  that  the  report  is  broadly 
satisfactory and can be approved on that basis. 

29 

 
 
 
 
 
 
 
ANNEX II.   METHODOLOGY AND ANALYTICAL MODELS USED 

This  annex  presents  the  methodological  approach  to  the  meta-evaluation  study  which 
underpins the SWD. It describes the design of the methodology, the tools used for  data 
and  information  gathering  and  the  results  obtained.  It  also  provides  insights  on  the 
limitations encountered during the study and the mitigation strategies adopted. 

Evaluation design  

The methodology  of the evaluation  study was designed to  respond to  (i) the evaluation 
questions  detailed  in  the  Terms  of  Reference  (ToR)  for  the  evaluation  and  (ii)  the 
evaluation  criteria  as  laid  down  in  the  Better  Regulation  Guidelines.  It  rested  on  three 
pillars:  

1.  Participatory  and  inclusive  data  collection  and  analysis  through  a  stakeholder 
consultation (deploying semi-structured interviews and an online questionnaire);  
2.  A  mixture  of  qualitative  input  (obtained  mainly  through  a  desk  research  of 
relevant  literature/publications  and  stakeholder  input)  and  quantitative  input 
(gained  through  econometric  analyses  assessing  the  quality  of  the  survey  data 
generated by the programme); and  

3.  Triangulation,  i.e.  the  information  and  data  collected  from  a  range  of  different 
sources using a range of methods collectively provides answers to DG ECFIN’s 
evaluation questions. 

Tools  for  information  gathering,  results  obtained,  limitations  and  mitigation 
strategies 

This section presents the methodological tools used by the external contractor to collect 
the information and data required to produce the external study. It furthermore details, for 
each methodological tool, the limitations which were experienced during the completion 
of the independent study and, hence, the preparation of the Commission’s evaluation. 

Desk-research 

Methodology 

A  comprehensive  literature  review  was  conducted  to  obtain  an  overview  of  the  current 
state  of  discussion  and  research  on  the  BCS  Programme.  The  different  steps  of  the 
process are described below. 

First,  based  on  the  evaluation  questions  defined  in  relation  to  the  evaluation  criteria  of 
effectiveness, efficiency, coherence, EU added value and relevance, a range of literature 
databases were identified that might provide relevant literature:  

•  Google Scholar is an internet search engine that indexes the full text or metadata 

of scientific literature in a variety of publication formats and disciplines;  

30 

 
 
 
 
 
• 

IDEAS is one of the largest bibliographic databases on economic research topics 
covering more than 4 200 000 research articles;  

•  EconBiz is a research portal for economics and is operated by the ZBW - Leibniz 

Information Centre for Economics. 

In order additionally  to obtain a representative overview of the grey literature22 and the 
opinions and assessments of politicians and the media on the programme and on specific 
evaluation  questions,  the  search  was  extended  to  general  search  engines  (Google  in 
particular).  Subsequently  different  keywords  relating  to  the  BCS  Programme  and  the 
evaluation  criteria  were  used  to  screen  the  relevant  databases  and  sources.  The  list  of 
keywords  was  extended  using  buzzwords  from  the  specific  evaluation  questions.  The 
resulting  list  of  documents  was  further  extended  with  Commission  documents  on  the 
BCS Programme, including papers and presentations from the EC workshops on current 
developments in business and consumer surveys as well as specific working papers and 
studies, all of which were available on DG ECFIN's website. 

Next,  duplicates  and  documents  that  could  not  be  assigned  to  an  author  or  institution 
were removed from the list of documents. In total, some 120 EC documents (workshop 
documents, working papers, reports and user guides) and about 200 external publications 
(peer-reviewed  articles,  external  assessments,  grey  literature)  were  included  in  the 
subsequent in-depth analysis.  

Finally, the results of the literature review were cross-checked with the findings from the 
stakeholder interviews, the online questionnaire and the quantitative analysis. 

Robustness and Limitations 

There were several potential risks and limitations to the robustness of the results derived 
from the review of the EU BCS documentation, evaluation reports, and EC and external 
research. These – and corresponding mitigation actions – are listed in the Table A2.1.  

22  Grey  literature  refers  to  materials  and  research  produced  by  organisations  outside  of  the  traditional 
commercial or academic publishing. It includes reports, working papers, government documents and 
evaluations. 

31 

 
 
 
 
 
 
 
 
 
 
Table A2.1: Risks and mitigation actions during the desk research 

Type of 
literature 

Risk 

Potential 
Impact 

Occurred 

Mitigation measure 

EC & external 
sources  

Lack of available reliable 
literature 

High 

Yes 

EC & external 
sources 

Drawing wrong 
conclusions due to equal 
weighting of information 
obtained  

Medium 

Yes 

External 
sources  

False conclusions due to 
biased literature selection 
and lack of 
representativeness of the 
information collected 

High 

No 

•  Using several bibliographic 
databases  (Google  Scholar, 
IDEAS,  EconBiz) 
and 
different search terms. 

•  Extending  the  scope  of  the 
research 
multi-
to 
disciplinary  literature  with 
similar  questions/problems 
and to grey literature 

•  Subjecting 

the 

findings 
obtained from the literature 
to  critical  evaluation  using 
the 
the 
stakeholder consultations. 

results 

of 

•  Repeatedly  discussing  the 
insights  gained  within  the 
team. 

•  Using 

several 
databases 
bibliographic 
(Google  Scholar,  IDEAS, 
EconBiz) 
different 
and 
search terms. 

•  Establishing  clear  research 
questions  early  on  and 
developing  a  detailed  plan 
on  how  to  answer  each 
question.  

•  Subjecting 

the 

findings 
obtained from the literature 
to  critical  evaluation  using 
the 
the 
stakeholder consultations. 

results 

of 

Source: Deloitte and DIW Econ 

Two particular risks did materialise in the course of the desk research for the evaluation 
of the BCS Programme.  

First, due to the specific nature of some of the evaluation questions to be addressed, the 
body  of  available  literature  was  sparse  for  some  questions.  For  example,  the  literature 
research did not provide much evidence on the impact on the effectiveness and relevance 
of the BCS Programme of changes in the methodology. In addition, the limited amount 
of  empirical  evidence  on  the  cost  reduction  potential  of  different  survey  methods  is 
ambiguous.  As  a  consequence  of  a  limited  range  of  available  sources,  there  was  a  risk 
that  false  conclusions  would  be  drawn  or  that  no  conclusive  assessment  would  be 
possible. To mitigate these risks, the scope of the literature review was broadened to the 

32 

 
 
 
 
grey literature, such as working papers, evaluations and summary papers (white papers) 
to screen for additional suitable evidence. Lastly, the triangulation with the results of the 
stakeholder interviews, the online questionnaire and the quantitative analysis served as a 
mitigation measure that made it possible to assess the findings from the literature review 
critically and close the gaps in the literature available.  

Second, the evidence was ambiguous for some evaluation questions. For example, some 
country-specific  studies  report  a  weak  BCS  forecasting  performance  for  specific 
countries,  while  other  studies  emphasise  the  added  value  of  the  EU  BCS  data  for 
forecasting  economic  developments  in  the  EU.  This  poses  the  risk  of  drawing  false 
conclusions or of being unable to draw any definite conclusions. To address these risks, 
ambiguous  findings  from  the  literature  were  discussed  within  the  project  team.  In 
addition, the results of the stakeholder online questionnaire and the quantitative analysis 
served  as  a  reference  point  for  validating  the  findings  from  the  literature.  Where  the 
ambiguity  of  the  results  could  not  be  resolved,  the  contradictory  findings  were 
expounded transparently and potential avenues for future research were outlined.  

In conclusion, although for some of the evaluation questions, the relevant literature was 
limited  or  ambiguous,  an  extended  search  and  triangulation  with  the  results  from  the 
stakeholder  consultations  and  quantitative  analysis  were  able  to  mitigate  the  risk  of 
drawing  false  conclusions.  The  results  derived  from  the  desk  research  can  thus  be 
considered robust. 

Stakeholder Consultations 

One of the main challenges of the stakeholder consultations was to be sure of covering 
the  wide  geographic  coverage  of  the  programme  and  obtaining  feedback  from  all 
stakeholder groups.  

When establishing the consultation strategy, several elements were identified as potential 
risks  for  interviews  and  surveys  as  data  collection  tools.  These  risks  fell  into  two 
categories:  

▪  stakeholders  might  not  want  to  participate  in  data  collection  activities  and, 

▪ 

therefore;  
the data obtained might be biased (i.e. the results would only capture the insights 
of  stakeholder  groups  that  were  more  likely  to  participate)  and  might  not  cover 
the geographical scope. 

To  mitigate  these  risks,  the  project  team  established  a  dynamic  and  flexible  process  to 
contact  and  schedule  interviews  with  all  targeted  stakeholder  groups.  This  process  was 
based on three main pillars: 

•  First,  a  thorough  stakeholder  mapping  was  conducted,  which  was  enriched  by 
inputs  provided  by  interviewees  (during  interviews  stakeholders  were  asked  to 
identify potential interviewees  who were aware  of the  programme and might  be 
willing to contribute to the consultation process).  

•  Second,  a  tracking  system  was  put  in  place  to  monitor  reminders  to  be  sent  to 
different  stakeholders  every  two  weeks  and  to  keep  track  of  their  responses, 
especially for the least represented stakeholder groups.  

33 

 
 
 
•  Finally, for the interviews, the project team adapted the interview content, format 

and duration to the availability and needs of stakeholders.  

To  capitalise  on  every  interaction  with  stakeholders,  online  questionnaires  were 
systematically  sent  as  a  follow-up  to  the  interviews.  As  a  result,  the  project  team 
managed  to  extract  as  much  information  as  possible  from  each  interview  and  gathered 
quantitatively comparable answers from all stakeholder groups (51% of the interviewees 
completed the survey).  

Table A2.2: Risks and mitigation actions by data collection tool 

Data 
collection 
tools 

Risk 

Potential 
Impact 

Occurred 

Mitigation measure 

Low response rate to 
survey 

High 

Yes 

Online 
questionnaire 

Online 
questionnaire 
and 
interviews 

Low or lack of 
stakeholder 
engagement in the data 
collection activities 

High 

Yes 

•  Extending 

consultation 
the 
period  to  ensure  a  sufficient 
number of replies. 

•  Monitoring  the  response  rate 
and sending reminders to boost 
participation.  

•  Receiving  feedback  from  all 
groups  and  verifying  that  the 
responses  and  concerns  were 
consistent  across  stakeholder 
groups. 

•  Using  the  accreditation  letter 
from  DG  ECFIN  to  increase 
the  chance  of 
stakeholder 
response. 

•  Asking  interviewees  and  DG 
ECFIN  to  provide  the  team 
contacts  who  were 
with 
already 
the 
aware 
programme  and  would  be 
interested in collaborating with 
this evaluation.  

of 

Source: Deloitte and DIW Econ 

Table A2.3: Risks and mitigation actions by stakeholder group 

Stakeholders 

Risk 

Potential 
impact 

Occurred 

Mitigation measure 

Low or lack of 
engagement of 
stakeholder group in 
the data collection 
activities 

Private sector 

in  place, 

•  Putting  a 
process 
individual 
agreements 
companies to participate.  

full  confidentiality 
including 
non-disclosure 
encourage 

to 

   Yes 

•  Making  use  of 

the  Deloitte 

34 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
 
Stakeholders 

Risk 

companies and 
media 

Potential 
impact 

     High 

Occurred 

Mitigation measure 

network,  when  possible 
encourage 
participate.  

companies 

to 
to 

•  Verifying that feedback had been 
received  from  different  private 
companies to obtain a variety of 
following  up  on 
responses, 
contacts  proactively 
there 
appeared to be gaps. 

if 

•  Remaining 

flexible 

and 
proposing  several  different  slots 
to  accommodate  the  diaries  of 
the different stakeholders.  

All 

Limited time to 
participate in 
interviews 

Medium 

Yes 

•  Extending 

consultation 
the 
period  to  collect  feedback  from 
groups 
under-represented 
(economic  press)  and  always 
proposing  that  they  send  the 
team  their  answers  in  writing 
and completing the survey when 
they  were  not  available  for  an 
interview. 

All 

Contacting someone 
not in a position to 
provide information 

High 

Yes 

All 

Limited knowledge 
of the topic 

High 

Yes 

that 

•  Liaising  with  DG  ECFIN  to 
right 
ensure 
representatives  of  stakeholder 
groups were contacted for both the 
surveys and interviews.  

the 

•  Ensuring  that  when  the  contact 
was  not  the  right  one,  the  team 
always  managed  to  get  in  touch 
with 
the 
someone  within 
organisation  who  could  answer 
our questions. 

•  Ensuring  our  contact  was  the 
right  person  to  reply  to  our 
questions.  

•  Sharing 

the 

interview  guides 
with  stakeholders  before 
the 
interviews to allow them to have 
available 
necessary 
the 
information  on  the  objectives  of 
the  interview  and  the  type  of 
insights  the  team  needed  from 
them. 

•  Stakeholder  answers  have  been 
weighted  giving  more  relevance 
stakeholders  with  higher 
to 

35 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Stakeholders 

Risk 

Potential 
impact 

Occurred 

Mitigation measure 

Limited ability to 
provide relevant data 
or information 

All 

High 

Yes 

experience using the  programme 
data 
the 
programme. 

or  working 

in 

•  Preparing 

interview 
targeted 
guides  allowed  asking  the  right 
questions to the right people. 

•  When  not  able 

to  provide 
relevant information to a specific 
question, 
stakeholders  were 
asked  to  tell  the  team  what 
prevented  them  from  answering 
the  question  and,  if  possible,  to 
provide 
the 
contact  details  of  a  colleague 
who could answer the question. 

team  with 

the 

• 

stakeholders 

about 
Inquiring 
qualitative 
or 
narratives,  as  a  substitute  for 
data or hard information. 

information 

Source: Deloitte and DIW Econ 

Despite all the above, the level of stakeholder engagement with the data collection was 
low or very low, for some stakeholder groups. The stakeholder response rates were low 
for interviews (34% of all stakeholders contacted agreed to  an interview) and very low 
for  the  online  questionnaire  (15%  of  the  stakeholders  only  contacted  for  the  survey 
completed  the  online  questionnaire  and  52%  of  the  stakeholders  responded  the  online 
questionnaire after the interview)).23  

Despite the low participation rates, the project team managed to cover all the stakeholder 
groups even though partner institutes are the stakeholder group most represented. Almost 
50%  of  the  feedback  and  data  obtained  from  the  interviews  and  online  questionnaire 
represented  the  point  of  view  of  the  partner  institutes,  with  the  remaining  50% 
representing the point of view of a range of EU BCS data user groups (i.e. private sector 
companies, institutional users, academics and economic press). This representation of the 
stakeholders was not by design, as participation in these activities was purely voluntary. 
A  higher  representation  of  partner  institutes  is  the  result  of  the  higher  interest  of  this 
stakeholder group in participating in this evaluation.24 The main limitation of the data is 

23  One  factor  that  could  explain  the  low  stakeholder  participation  rate  is  the  project  timeline  i.e.  the 

consultation took place mainly during the summer (i.e. June, July, and August).  

24  To  obtain  more  participation  from  under-represented  users  (e.g.  media  and  private  sector  companies 
from  sectors  other  than  the  financial  sector)  the  consultation  period  was  extended  until  the  end  of 
October for these groups, but this nevertheless did not bring adequate representation of these groups. 

36 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
the  under-representation  of  private  sector  companies  outside  the  financial  sector  and 
especially of users from the economic press. 

Figure 5: Coverage of interviews and online questionnaires, by stakeholder groups 

Source: Deloitte and DIW Econ 

In  terms  of  geographic  coverage,  28  countries  were  covered  with  both  data  collection 
tools: 24 of the 27 Member States were covered (Bulgaria, Cyprus and Portugal were not 
covered);  only  two  of  the  five  candidate  countries  were  covered  (Türkiye  and  Serbia). 
The UK and Switzerland were also covered. 

The  online  questionnaire  allowed  the  study  team  to  collect  quantitatively  comparable 
answers and some qualitative responses to open questions which allowed to complement 
the  qualitative  insights  gathered  during  the  interviews.  To  that  end,  two  separate 
interview guides and online surveys were prepared for partner institutes and users. Users 
were mainly asked how they use the programme data. Partner institutes were asked about 
operational  aspects  of  the  programme  and  also  replied  to  questions  on  the  use  of  the 
programme data, as some of them use the EU BCS data as research institutes or national 
statistical agencies, and/or could provide inputs on behalf of national users who buy their 
EU  BCS  national  data  or  have  reached  out  to  them  with  questions  on  the  programme 
data. 

To address any issue of bias when analysing the information gathered, the stakeholders’ 
responses  were  triangulated  with  the  results  of  the  desk  research  and  literature  review, 
and  weighted  taking  into  account  question  topics  and  stakeholder  experience  with  the 
data/  programme.  In  other  words:  1)  interview  and  survey  responses  were  triangulated 
and contrasted with the results of the literature review and quantitative analysis, and 2) 
for each question, the stakeholders’ experience with the programme and data as well as 
their knowledge of the issue were taken into account. By knowing the background of the 

37 

46%20%16%12%6%49%21%16%9%5%Partner InstitutesAcademicsInstitutional usersPrivate Sector CompaniesEconomic PressInterviewsOnline Questionnaires 
 
 
 
respondent, it was easy to assess, especially for the technical questions, which feedback 
was more relevant.  

In  summary,  despite  the  low  participation  rate  of  some  stakeholder  groups  in  both 
interviews and surveys, information from all stakeholder groups was collected to match 
the  geographical  coverage  of  the  programme.  The  results  and  conclusions  of  the 
stakeholder  consultation  were  therefore  considered  to  be  robust  as,  on  the  one  hand, 
responses  were  triangulated  to  avoid  and  clarify  any  discrepancies,  and  on  the  other, 
answers  were  weighted  taking  into  account  the  stakeholder’s  experience  with  the 
programme/data and their knowledge of the topic of the questions. 

Quantitative Analysis 

Methodology  
The quantitative analysis was structured into the following activities:  

•  Activity 1: Determining the aim and scope of the analysis 
•  Activity 2: Data preparation 
•  Activity 3: Bivariate analysis 
•  Activity 4: Multivariate analysis 
•  Activity 5: Meta-analysis and other quantitative evaluations 
•  Activity 6: Alternative aggregation procedures of the data 
•  Activity 7: Summary and visualisation.  

Activity  1  determined  the  aim  and  scope  of  the  analysis,  which  was  followed  by  data 
preparation  (Activity  2).  While  Activities  3  and  4  focused  on  the  predictive  and 
explanatory  power  of  the  EU  BCS  data  relative  to  other  leading  indicators,  Activity  5 
used  the  results  of  the  previous  steps  to  find  correlates,  e.g.  between  certain 
methodological  features  of  the  surveys  and  the  quality  of  the  data  generated  by  the 
surveys.  Activity  6  specifically  corresponded  to  Question  3  of  the  evaluation  matrix 
(Could  the  ability  of  the  data  to  capture  economic  developments  be  enhanced  through 
different  aggregation  techniques?).  Activity  7  summarised  and  comprehensively 
visualised the findings.  

Activity 1: Determining the aim and scope of the analysis 
The main goal of the quantitative analysis was to assess whether the EU BCS survey data 
accurately  captures  economic  developments  in  the  EU  Member  States  and  candidate 
countries.  There  was  a  particular  focus  on  whether  the  survey  data  can  be  used  to 
nowcast and forecast real GDP growth and HICP-based inflation. However, that left the 
question of which other variables should be included in the nowcasting and forecasting 
models. The study team therefore scanned the now- and forecasting literature to identify 
current best practices. Due to the increased use of machine learning methods in now- and 
forecasting,  the  team  also  scanned  the  literature  for  applications  of  machine  learning 
methods,  such  as  ensemble  and  shrinkage  estimators,  to  the  forecasting  of  (economic) 
time  series  data.  The  review  of  the  literature  focussed  on  numerical  forecasting  since 
these  are  the  forecasting  methods  most  commonly  used  by  academics  and  professional 
economists. The literature on turning point and recession forecasts was thus beyond the 
scope  of  the  quantitative  analysis.  Table  A2.4  summarises  the  meta  information  of  the 
literature reviewed.  

38 

 
 
 
Table A2.4: Literature search: Literature on now- and forecasting 

Number of papers reviewed 

Areas 

Models used 

Data used 

Key aspects of the literature 

Source: Deloitte and DIW Econ 

• 

• 
• 
• 

129 

Papers on forecasting: 84 
Papers on nowcasting: 59 
Papers using survey-based indicators: 101 

•  Dynamic Factor models 
•  VAR, VEC, BVAR 
•  MIDAS 
•  DSGE models 
•  Machine learning models (e.g. Random forest, LASSO) 

• 

Papers usually use data for the US, UK, the euro area and Japan. 
US  data  are  by  far  the  most  often  used,  due  in  part  to  the  data 
availability.  

•  Data sets use up to 400 variables depending on data availability  

•  Most studies on now- and forecasting use dynamic factor models 
due  to  the  short  time  series.  The  benefits  of  mixed  frequency 
BVAR models are often highlighted in this context. 

•  The  number  of  papers  using  machine  learning  methods  is 

growing. 

Activity 2: Data preparation 
The baseline data set for the evaluation of the EU BCS survey data consisted of two key 
variables  of  interest:  quarterly  real  GDP  and  monthly  HICP  inflation.  The  data  was 
complemented by all available data produced by the programme at a monthly frequency. 
In addition, all macroeconomic data available at the monthly frequency was added. The 
data spanned all 27 Member States plus the five candidate countries. For the majority of 
countries,  the  EU  BCS  series  were  available  from  January  1985  onwards.  Data  on 
economic sentiment, industrial and construction confidence for DE, DK, FR, IT, LU, NL 
were  available  from  January  1980  onwards.  The  variable  selection  for  the  data  set 
followed  the  standard  practice  in  the  literature  (Carriero,  Galvao,  &  Kapetanios,  2019; 
Angelini, Camba-Mendez, Giannone, Reichlin, & Rünstler, 2011). Data were taken from 
Eurostat  and  downloaded  via  Eurostat’s  API  using  R’s  Eurostat  library.  EU  BCS  data 
that is not available via Eurostat, such as the sub-sector EU BCS data, was downloaded 
manually from the ECFIN website in June 2022. The seasonally adjusted data series were 
used;  the  financial  services  sector  data  were  the  only  exception  since  they  only  exist 
unadjusted due to the short time series. However, to fit the rest of the data set  the team 
used  a  simple  AR-based  de-seasonalisation  procedure  to  avoid  data  loss.  In  addition  to 
the  data  available  on  Eurostat,  the  team  used  the  RWI  Container  Throughput  Index 
(RWI, 2022); national stock market indices were taken from OECD. For the countries for 
which such data was available, price indices from The Billion Prices Project (Cavallo & 
Rigobon,  2016)were used as  a real-time measure for inflation. Since the data is  by and 

39 

 
 
 
 
large non-stationary, the study team generated year-on-year (log-) differences of the data 
where appropriate. Table A2.5 summarises the data sets and the relevant transformations.  

Table A2.5: Data series and their transformations 

Differenced  Log  Frequency  Source 

X 

X 

x 

x 

Variable 

Real GDP 

Harmonised consumer price index  

Construction confidence indicator 

Construction development of orders 

Construction employment expectations next 3 months 

Construction price expectations in 3 months 

Building activity development over the past 3 months  

Factors limiting construction activity – demand 

Factors limiting construction activity – labour 

Factors limiting construction activity – financial 

Factors limiting construction activity – material 

Factors limiting construction activity – none 

Factors limiting construction activity – other 

Factors limiting construction activity – weather 

Industry confidence indicator 

Industry assessment of export order-book levels 

Industry assessment of order-book levels 

Industry assessment of the current level of stocks of finished products 

Industry production development observed over the past 3 months 

Industry employment expectations over the next 3 months 

Industry selling price expectations over the next 3 months 

Industry production expectations over the next 3 months 

Services confidence indicator 

Services business situation development over the past 3 
months 

Services evolution of demand over the past 3 months 

Services evolution of employment over the past 3 months 

Services expectation of the demand over the next 3 months 

Expectations of employment over the next 3 months 

Services expectations of prices over the next 3 months 

Consumer confidence indicator 

Consumer statement on financial situation of household 

Consumer major purchases at present 

40 

Q 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Variable 

Consumer savings at present 

Consumer general economic situation over last 12 months 

Consumer general economic situation over next 12 months 

Consumer financial situation over next 12 months 

Consumer major purchases over next 12 months 

Consumer price trends over next 12 months 

Consumer savings over next 12 months 

Consumer financial situation over last 12 months 

Consumer price trends over last 12 months 

Consumer unemployment expectations over next 12 months 

Retail confidence indicator 

Retail current stocks 

Retail business expectations over the next 3 months 

Retail employment expectations over the next 3 months 

Retail expected order over the next 3 months 

Retail expected sales over the next 3 months 

Other financial services business situation previous 3 
months 

Other financial services confidence indicator 

Other financial services demand previous 3 months 

Other financial services employment previous 3 months 

Other financial services expected demand next 3 months 

Other financial services employment expectations next 3 
months 

Financial services business situation previous 3 months 

Financial services confidence indicator 

Financial services demand previous 3 months 

Financial services employment previous 3 months 

Financial services expected demand next 3 months 

Financial services employment expectations next 3 months 

Financial intermediation business situation previous 3 
months 

Financial intermediation confidence indicator 

Financial intermediation demand previous 3 months 

Financial intermediation employment previous 3 months 

Financial intermediation expected demand next 3 months 

Financial intermediation employment expectations next 3 

41 

Differenced  Log  Frequency  Source 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Variable 

months 

Insurance business situation previous 3 months 

Insurance confidence indicator 

Insurance demand previous 3 months 

Insurance employment previous 3 months 

Insurance expected demand next 3 months 

Insurance employment expectations next 3 months 

Economic sentiment indicator  

Gross wages capital goods sector 

Hours worked capital goods sector 

Employment capital goods sector 

Production capital goods sector 

Domestic turnover capital goods sector 

Foreign turnover capital goods sector 

Total turnover capital goods sector 

Gross wages consumer goods sector 

Hours worked consumer goods sector 

Employment consumer goods sector 

Production capital goods sector 

Domestic turnover consumer goods sector 

Foreign turnover consumer goods sector 

Total turnover consumer goods sector 

Gross wages durable goods sector 

Hours worked durable goods sector 

Employment durable goods sector 

Production durable goods sector 

Domestic turnover durable goods sector 

Foreign turnover durable goods sector 

Total turnover durable goods sector 

Gross wages gas/electricity sector 

Hours worked gas/electricity sector 

Employment gas/electricity sector 

Production gas/electricity sector 

Gross wages energy sector w/o gas and electricity 

Hours worked energy sector w/o gas and electricity 

Employment energy sector w/o gas and electricity 

42 

Differenced  Log  Frequency  Source 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Variable 

Differenced  Log  Frequency  Source 

Production energy sector w/o gas and electricity 

Domestic turnover energy sector w/o gas and electricity 

Foreign turnover energy sector w/o gas and electricity 

Total turnover energy sector w/o gas and electricity 

Production energy sector w/o gas and electricity 

Gross wages intermediate goods sector 

Hours worked intermediate goods sector 

Employment intermediate goods sector 

Production intermediate goods sector 

Domestic turnover intermediate goods sector 

Foreign turnover intermediate goods sector 

Total turnover intermediate goods sector 

Gross wages manufacturing 

Hours worked manufacturing 

Employment manufacturing 

Production manufacturing 

Domestic turnover manufacturing 

Foreign turnover manufacturing 

Total turnover manufacturing 

Gross wages mining 

Hours worked mining 

Employment mining 

Production mining 

Domestic turnover mining 

Foreign turnover mining 

Total turnover mining 

Gross wages non-durable goods sector 

Hours worked non-durable goods sector 

Employment non-durable goods sector 

Production non-durable goods sector 

Domestic turnover non-durable goods sector 

Foreign turnover non-durable goods sector 

Total turnover non-durable goods sector 

Gross wages water industry 

Hours worked water industry 

Employment water industry 

43 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

 
 
 
Variable 

Price index capital goods 

Price index consumer goods 

Price index durable goods 

Price index electricity 

Price index gas 

Price index energy w/o electricity and gas 

Price index high-tech manufacturing 

Price index intermediate goods 

Price index low-tech manufacturing 

Price index manufacturing 

Price index mid-tech manufacturing 

Price index mining 

Price index non-durable consumer goods 

Producer price index industry 

Number of construction starts 

Production index construction 

Production index civil engineering 

Gross wages construction 

Hours worked construction 

Employment construction 

Production construction 

Number of construction starts – residential 

Number of construction starts excl. community housing 

Cost index construction 

Input prices construction 

Labour cost construction 

Output prices construction 

Turnover hospitality 

IT/communication turnover 

Turnover real estate sector 

Turnover transportation 

Hospitality Services 

Gross wages hospitality 

Hours worked hospitality 

Employment hospitality 

Gross wages IT/communication 

44 

Differenced  Log  Frequency  Source 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

 
 
 
Variable 

Differenced  Log  Frequency  Source 

Hours worked IT/communication 

Employment IT/communication 

Gross wages real estate sector 

Hours worked real estate sector 

Employment real estate sector 

Gross wages transportation 

Hours worked transportation 

Employment transportation 

Retail turnover 

Gross wages retail 

Hours worked retail 

Employment retail 

Consumption brown coal 

Consumption diesel 

Consumption electricity 

Consumption fuel 

Consumption kerosene 

Consumption motor spirit 

Consumption natural gas 

Imports crude oil 

Imports electricity 

Imports natural gas 

Production diesel 

Production electricity 

Production motor spirit 

Production natural gas 

Energy total supply 

Stock market index 

Crude oil prices (Brent) 

Extra-euro area exports capital goods 

Extra-euro area exports consumer goods 

Extra-euro area exports intermediate goods 

Extra-euro area exports total 

Extra-euro area imports capital goods 

Extra-euro area imports consumer goods 

Extra-euro area imports intermediate goods 

45 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

OECD 

EIA 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

Eurostat 

 
 
 
Variable 

Differenced  Log  Frequency  Source 

Extra-euro area imports total 

Government bond yield (10 years) 

Money market rate (3 months) 

Overnight money market rate 

Unemployment total 

Real effective exchange rate (broad concept) 

Industrial import price index 

Container throughput 

Industrial production 

Number of air passengers carried 

Number of commercial flights 

Source: Deloitte and DIW Econ 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

X 

x 

x 

x 

x 

x 

X 

X 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

Eurostat 

ECB 

ECB 

ECB 

Eurostat 

Eurostat 

Eurostat 

RWI 

Eurostat 

Eurostat 

Eurostat 

Activity 3: Bivariate analysis 
The bivariate analysis of the EU BCS data proceeded in two steps: 

1.  First, the team analysed the dynamic cross-correlations of the Economic Sentiment 
Indicator  with  log  differences  in  real  GDP  and  of  12-months-ahead  price 
expectations with the log differences of the HICP.  

2.  Second,  the  team  ran  bivariate  Granger  causality  tests  between  the  real  GDP  and 

HICP variables and the relevant EU BCS series.  

In  both  cases,  the  monthly  EU  BCS  data,  monthly  HICP  data  and  quarterly  real  GDP 
data were used. For the cross-correlation analyses, quarterly real GDP data was treated as 
a monthly variable in the last month of the quarter in order to conduct the analysis at the 
monthly frequency. For example, quarterly real GDP in the first quarter of the year was 
used as an observation for real GDP in March. The next observation of real GDP in June 
is  then  GDP  in  the  second  quarter.  For  the  Granger  causality  tests,  the  analysis  was 
performed at  the quarterly  frequency, so quarterly averages  were calculated for the EU 
BCS data. 

The  key  performance  indicators  for  these  two  procedures  are  the  magnitude  of  the 
correlation coefficient between real GDP and lagged values of the EU BCS data, and a 
significant  F-test  for  the  Granger  causality  test.  Both  tests  were  conducted  for  all 
available series of the EU BCS data across all countries and their corresponding EA and 
EU aggregates.  

Activity 4: Multivariate analysis 
For  the  multivariate  evaluation  of  the  quality  of  the  EU  BCS  data,  the  study  team 
estimated a forecasting model using all available data. The analysis was conducted at the 
quarterly  frequency,  with  quarterly  averages  calculated  for  the  monthly  EU  BCS  data. 
The models were run separately for all available countries and the EA. A random forest 
model was chosen for this exercise. While MIDAS or dynamic factor models are used by 
most  authors  for  this  type  of  estimation,  recent  studies  have  shown  that  ensemble 
methods  or  shrinkage  estimators  perform  equally  well  in  terms  of  their  forecasting 

46 

 
 
 
 
 
 
 
performance.  Moreover,  using  machine  learning  models  allows  the  researcher  to  be 
agnostic about the variables that enter the model without the risk of over-fitting. 

To evaluate the forecasting power of the EU BCS data, the study team first ran a model 
for  each  country  using  all  available  data  and  secondly  a  model  excluding  the  EU  BCS 
data.  To  evaluate  the  predictive  power,  the  root-mean-squared  errors  (RMSEs)  of  both 
models  were  compared.  Hence,  if  the  RSME  of  the  model  using  the  EU  BCS  data  is 
smaller than the RMSE of the model without the EU BCS data, then the EU BCS data 
adds valuable information to the model. 

Activity 5: Meta-analysis 
This step used the metadata to explain differences in the forecasting performance across 
countries and sectors. The approach used was to take the Granger causality tests and the 
contemporaneous  correlations  between  the  sectoral  confidence  indicators  and  log  real 
GDP  growth  for  each  country.  In  the  case  of  the  Granger  causality  tests,  a  dummy 
variable  was  constructed  that  is  1  if  the  particular  confidence  indicator  Granger  causes 
real  GDP  growth  and 0  otherwise and then used the classified metadata  as explanatory 
variables  of  the  model.  The  variables  created  from  the  model  and  their  data 
transformations  are  summarised  in  Table  A2.6.  The  model  was  estimated  using  OLS 
including country fixed effects. The evaluation metric here was whether or not a specific 
factor, such as sample size or sampling procedure had a significant positive or negative 
impact on the Granger causality.  

Table A2.6: Metadata tested as explanatory variables for forecasting performance 

Variable 

Number of weighting factors 

Sampling method 

Type of partner institute 

Non-response treatment 

Sample size 

Population size 

Fieldwork period 

Type of interview 

Source: Deloitte and DIW Econ 

Transformation 

Type 

Numeric 

List of dummy variables 

List of dummy variables 

List of dummy variables 

Numeric 

Numeric 

Numeric 

Log 

Log 

Log 

List of dummy variables 

Activity 6: Alternative aggregation procedures of the data 
This  step  explored  alternative  ways  of  aggregating  the  EU  BCS  survey  data  and 
constructing  confidence  indicators  and  evaluated  their  impact  on  the  forecasting 
performance of the BCS indicators.  

First,  composite  indicators  across  sectors  were  compared  qualitatively  based  on  their 
proportion of forward/backward-looking and contemporaneous questions to see how far 
deviations  between  sectoral  indicators  could  be  explained  by  the  different  focus  of  the 
questions.  Second,  alternatives  to  simple  unweighted  averages  to  construct  the  sectoral 
confidence indicators were investigated.  

47 

 
 
 
 
 
 
 
  
In addition, based on the information given in the programme User Guide, the study team 
constructed  two  alternative  versions  of  the  ESI,  namely  one  that  was  only  forward-
looking  and  one  that  was  only  backward-looking,  and  then  compared  their  cross-
correlations with log real GDP growth. 

Activity 7: Summary and Visualisations 
In  the  last  activity  of  the  quantitative  analysis,  the  results  were  summarised  and 
visualised so that the main results of the analysis could easily be understood by the reader 
of the evaluation report.  

Robustness and limitations 

In  the  quantitative  analysis,  there  was  the  possibility  of  robustness  issues  arising  from 
spurious results, model misspecification and data limitations.  

The  risk  of  spurious  results  may  particularly  arise  for  cross-correlations  and  Granger 
causality  tests.  For  example,  the  observed  strong  correlation  between  real  GDP  growth 
and the ESI may be spurious if both variables are correlated with a third factor that drives 
the  observed  correlation,  even  though  the  true  correlation  between  the  two  variables  is 
zero.  However,  this  is  a highly  unlikely  case  since  the  ESI  and  the  sectoral  confidence 
indicators aim to capture expectations about the observed current state of the economy. 
Standard macroeconomic models constantly highlight the importance of expectations for 
macroeconomic outcomes. Therefore, the results are considered to be reliable.  

In  transforming  the  variables  for  the  model  specifications,  the  study  team  followed 
closely  standard  practices  in  the  academic  literature.  A  potential  issue  that  remained 
regarding  the  bivariate  models  was  that  the  relationship  between  e.g.  real  GDP  growth 
and  economic  sentiment  might  be  non-linear  and  thus  simple  linear  models  might  not 
suffice  to  capture  the  relationship  between  economic  sentiment  or  confidence  and  the 
outcome variable. This could be a potential explanation for why the relationship between 
real GDP growth and economic sentiment in a few countries is rather weak. On the other 
hand, the meta-analysis showed that several structural factors of the programme are able 
to explain significant parts of the differences. Moreover, the multivariate random forecast 
models  were  able  to  address  potential  non-linearities  between  EU  BCS-based  variables 
and real GDP growth and inflation respectively.  

In  terms  of  sampling,  the  team  conducted  robustness  tests,  such  as  excluding 
recessionary periods and the COVID-19 crisis where necessary. This applies particularly 
to  the  results  of  the  multivariate  forecasting  models.  In  general,  by  virtue  of  their 
construction, multivariate random forecast results can be considered more robust than the 
bivariate models. Similarly, one of the important features of random forecasts is that the 
model  and  variable  selection  are  purely  data-driven  and  consist  of  the  “most  optimal 
specification”  given  the  training  sample.  Therefore,  the  results  are  considered  as  more 
reliable than models in which the specification of the model is chosen subjectively.  

In terms of data limitations,  it needs to  be pointed out  that the covariates for the meta-
regressions were constructed from the 2016 metadata publicly available on the EU BCS 
website  and  augmented  by  changes  documented  in  the  downloadable  Excel  data  sets. 
These observations constitute a snapshot of the current state of play at a particular point 
in  time,  while  the  dependent  variables  are  the  outcome  of  bivariate  time  series  models 
over  a  longer  horizon.  Hence  changes  in  the  collection  of  the  EU  BCS  data,  and  thus 

48 

 
 
 
changes in the relationship between EU BCS data and economic aggregates may not be 
fully reflected in  the metadata on the right-hand side of the model. Furthermore, in  the 
multivariate analysis of the forecasting performance, several countries were omitted from 
the estimation of the forecasting model because there were not enough data points for a 
consistent estimation. 

While some evaluation questions, such as ‘Have changes to the programme enhanced its 
effectiveness?’,  would  have  lent  themselves  to  quantitative  investigation,  this  could  not 
be done due to the lack of data from before the changes. Similarly, answers to questions 
such  as  the  ‘Coherence  of  the  survey  across  countries’  had  to  be  based  on  potentially 
outdated questionnaires shown on the DG ECFIN website. 

In  summary,  the  results  of  the  quantitative  analysis  can  be  considered  robust  in  most 
cases. In the case of a few questions, however, no data were available or the data were 
insufficient,  implying  a  need  for  cautious  interpretation  of  the  empirical  findings  as 
highlighted in the corresponding sections of the report. Table A2.7 illustrates the risks in 
the  quantitative  analysis  of  the  data  collection  tools  and  the  mitigation  measures  that 
were taken. 

Table A2.7: Risks and mitigation actions during the quantitative analysis 

Data collection 
tools 

Correlation 
Analysis  

Risk 

Potential 
Impact 

Occurred 

Mitigation measure 

 Spurious correlation 

High 

No 

•  Comparing 

correlation 
results  to  studies  from  the 
literature  and  triangulating 
bivariate 
with 
multivariate analysis.  

results 

•  Choosing 

approaches 
that  reflect 
current state of research. 

modelling 
the 

Multivariate 
Analysis  

Model mis-
specification & 
sampling issues  

High 

No 

•  Applying 

a 

data-driven 
specification 
variable 
selection to rule out subjective 
specification errors. 

and 

•  Comparing  and 
quantitative 
results  from 
where available. 

testing  the 
results  against 
literature, 

the 

•  Conducting  different  model 
to  provide  a 
the 

check  of 

specifications 
robustness 
findings. 

Multivariate 
Analysis 

Lack of data or 
poor data quality 

Medium 

Yes 

Source: Deloitte and DIW Econ 

•  Estimating forecasting model 
only for those countries for 
which sufficient data are 
available for consistent 
estimation 

49 

 
 
 
ANNEX III.   EVALUATION MATRIX AND, WHERE RELEVANT, DETAILS ON ANSWERS TO  THE EVALUATION QUESTIONS (BY CRITERION) 

This  annex  provides  a  detailed  overview  of  the  evaluation  matrix  used  for  the  evaluation  study.  It  was  developed  by  the  external  contractor,  as  a 
framework  for  the  independent  assessment  in  line  with  the  ToR.  The  matrix  assigns  each  evaluation  question  outlined  in  the  ToR  to  the  relevant 
evaluation criteria and serves as general framework for the applied methodology. 

Table A3.1: Evaluation questions matrix 

Question 

Judgement Criteria 

Indicators 

Q1. How successful was the programme in 
achieving the objective of providing data for 
economic surveillance in the European Union 
enabling to compare business cycles between 
Member States and giving an overall view of 
the business cycle in the Union? 

Q2. How accurately and reliably do the survey 
data collected by the programme capture 
economic developments in the Member States 
and candidate countries? 

• Extent to which the BCS data is used to 
nowcast and forecast the business cycle     
• Usefulness of the BCS data for monitoring 
and predicting business cycle fluctuations 
• Accuracy and reliability of BCS data for 
capturing economic developments in Member 
States and Candidate Countries 

• Cross Correlations between BCS indicators and real 
GDP growth and HICP-based inflation  
• Use cases and evaluation of the BCS data in the 
literature           
• Survey question: "The BCS data are an essential 
input for the monitoring and now/forecasting of 
economic developments in our country?" 

Information/Data Source 

• Desk research (academic 
literature, policy literature) 
• Quantitative analysis25  
• Stakeholder interviews  
• Online questionnaire 

Q3. Could the ability of the data to capture 
economic developments be enhanced through 
different aggregation techniques? 

• Extent to which alternative aggregation 
methods have better tracking performance of 
economic developments in EU and candidate 
countries 

• Comparison of cross-correlations of existing 
indicators and alternative indicators (based on 
alternative aggregation) with real GDP growth 
• Evaluation of alternative aggregation methods in the 
literature 

• Desk Research (User guide, 
Academic Literature, 
Documentation of other Survey 
Institutions) 
• Quantitative analysis 

25 A quantitative analysis of the use of BCS data based on counts of citations was not meaningful because observed citations are a noisy measure of actual use of the data and because of 

a lack of a meaningful benchmark. 

50 

 
 
 
 
 
 
 
Q4. Are the data timely enough? 

• Perception of timeliness of data by users 
• Timeliness of BCS data compared to other 
indicators 

• Survey question: "Are the EU BCS data timely 
enough?"  
• Evaluation of timeliness in literature 
• Comparison of publication lag and cross-
correlations between alternative indicators / BCS data 
and real GDP growth/HICP inflation 

• Desk research 
• Quantitative analysis  
• Stakeholder interviews 
• Online survey 

Q5. How useful are they for 
nowcasting/forecasting relevant economic 
variables? 

• Accuracy of nowcasts/forecasts for 
predicting economic developments 

Q6. Are data disseminated in a clear and 
understandable form? 

• The degree to which BCS data are accessible 
and understandable for the users 
• Availability of relevant information for 
expert users 

• Granger Causality Test between BCS ESI indicator 
and real GDP  
• Comparison of root mean squared forecasting errors 
(RMSE) of multivariate forecasting model with and 
without BCS indicator 
• Use of BCS data for forecasting as reported in 
stakeholder consultation 
• Evaluation of forecasting accuracy of BCS data in 
academic literature 

• Survey questions: "Do you think the data from the 
BCS Programme is easily accessible and presented in 
an understandable way?" "How do you usually access 
the survey data related to your country?" & "How do 
you usually access the survey data?" 
• Comparison to best practices of data dissemination  

Q7. Are there sufficient supporting metadata 
and guidance for users? 

• The degree to which metadata are 
comprehensive, up-to-date and easily 
accessible 

• Survey Question “Are the metadata and guidance 
for users sufficient and understandable?" 
• Comparison to best practices for metadata 

• Desk research (academic 
literature) 
• Quantitative analysis  
• Stakeholder interviews  

• Desk research (Press Releases, 
programme Webpage) 
• Stakeholder interviews 
• Online questionnaire 

• Desk research (User Guide, 
Methodological guidelines, 
Reference metadata on 
methodology and quality, 
programme Webpage) 
• Stakeholder interviews  
• Online questionnaire 

Q8. Has the programme enhanced the 
capabilities of partner institutes, for example 

• Partner institutions' assessment of whether 
the BCS Programme has improved their 

• Documented perception of the partner institutes in 
the literature  

• Desk research (Workshop 
documents) 

51 

 
 
 
 
 
 
through knowledge sharing? 

capabilities. 

• The degree to which BCS methodology is 
referenced in other trend surveys and best 
practice examples. 

• Impact of programme changes on the 
effectiveness of the BCS Programme. 

Q9. Has the programme created 
methodological spillovers? 

Q10. Have changes to the programme's 
methodology and coverage enhanced its 
effectiveness? 

Q11. To what extent was the design, 
implementation, and financing of the 
programme appropriate? Have changes 
improved its appropriateness? 

• Survey Questions " In your opinion does the BCS 
Programme enhance partner institutes’ capabilities 
(e.g., reporting) and contribute to the development of 
new indices and products?" & "Which capabilities 
would you say are enhanced?" 

• Stakeholder interviews 
• Online questionnaire 

• Documented references to the methodology of the 
BCS Programme  

• Desk research (academic 
literature, policy literature) 

• Impact assessments in the literature 
• Assessments of specific changes by different 
stakeholders 

• Desk research (academic 
literature, workshops documents) 
• Stakeholder interviews 

• programme design, implementation and 
financing in line with best practices 
• programme design, implementation and 
financing well-justified and adaptive to issues 
encountered 
• programme changes had a discernible impact 

• Evaluative judgment of the programme’s budget 
and administration 
• Comparison of the programme’s design and 
implementation with best practices (e.g. UN/OECD 
Handbook on BCS) 

• Desk research (academic 
literature, policy literature) 
• Quantitative analysis  
• Stakeholder interviews 
• Online questionnaire 

Q12. What are the cost and benefits of the 
BCS for different stakeholders? 

• Quantitative or qualitative figures/estimation 
of costs and benefits 

Q13. What is the simplification, cost and 
burden reduction potential? 

• Qualitative assessment of potential cost or 
burden reductions 

• Evolution of the costs and benefits of the BCS 
Programme over time  
• Evaluation of Cost & Benefits from users and 
Partner Institutes 

• Self-assessment of stakeholders 
• Stakeholders’ narrative of potential reductions 
• Feasibility of potentially cost reducing methods 
according to the academic literature 
• Cost reduction when compared to best practice 
implementation 

• Desk research (previous 
programme evaluation, Financial 
Transparency System, Budget 
figures provided by DG ECFIN) 
• Stakeholder interviews  

• Desk research (academic 
literature, programme meta data 
on methodology and quality)  
• Stakeholder interviews 
• Online questionnaire 

52 

 
 
 
 
 
 
  
Q14. Could alternative approaches to 
monitoring the economy in (quasi) real-time, 
such as big data analysis, have achieved the 
same benefits at less cost, or greater benefits 
at the same cost? 

Q15. To what extent is the financial 
administration of the programme, namely 
through the annual award of grant agreements 
and reimbursement based on incurred costs 
under multi-year framework partnership 
agreements efficient? 

• Costs and benefits of alternative indicators 
for predicting economic developments 

• Evaluation of alternative indicators in the literature  
• Stakeholders' assessment of alternative indicators  

• Desk research (academic 
literature, policy literature) 
• Stakeholder interviews  

• The extent to which the financial 
administration of the programme is considered 
efficient. 

• Evaluative judgment of the programme 
administration 

• Desk research (academic 
literature, policy literature) 
• Stakeholder interviews 

Q16. To what extent is the programme still 
relevant? 

• Extent to which the BCS surveys offers 
unique benefits compared to other surveys and 
indicators 

• Timeliness of alternative indicators 
• Geographical coverage of the alternative indicators 
• Stakeholders' assessment of the relevance of the 
BCS Programme 
• Survey Question: "The programme is still relevant 
in the light of recent progress in accelerating the 
release of important statistical data, such as EU/euro-
area GDP (preliminary flash estimate), and the 
availability of alternative short-term indicators (e.g., 
big data).” 

• Desk research (academic 
literature, press releases, web 
pages) 
• Stakeholder interviews  
• Online questionnaire 

• Programme has no gaps with regards to what 
would be useful for stakeholders 
• Programme has no unused parts with regards 
to the needs of stakeholders 

• Comparison of programme content with the needs 
of stakeholders 

• Desk research (academic 
literature, policy literature) 
• Stakeholder interviews 
• Online questionnaire 

Q19. Is the surveying frequency (monthly for 

• Assessment whether the surveying frequency 

• Survey Question: “Would you say the surveying 

• Desk research (academic 

53 

Q17. Given the programme’s aim to provide 
quasi real time information on the state of the 
EU and EU candidate economies, is the 
programme still relevant in the light of recent 
progress in accelerating the release of 
important statistical data, such as EU/euro-
area GDP (preliminary flash estimate), and the 
availability of alternative short-term 
indicators? 

Q18. Do the survey questions used in the BCS 
Programme focus on the most relevant 
economic issues or could the programme 
benefit from additional or modified questions 
and could some of the questions be dropped 
from the survey? 

 
 
 
 
most questions, quarterly for some) 
appropriate? 

is appropriate 

Q19a. In your view, does the Financial 
Services Sector Survey complement other data 
provided by other financial services surveys? 

• Assessment of the usefulness of the 
Financial Services Sector Survey and its 
complementary value in view of other 
financial services surveys 

frequency (monthly for most questions, quarterly for 
some) is appropriate?” 
• Users’ assessment of survey frequency in interviews 

literature, policy literature) 
• Stakeholder interviews 
• Online questionnaire 

• Survey Question: “The Financial Services Sector 
Survey is complementary to other data (e.g.., other 
financial services surveys /indicators or additional 
financial services forecast)?” 
• Users’ assessment of usefulness in interviews 

• Stakeholder interviews 
• Online questionnaire 

Q20. Does the sectoral aggregation of the 
survey results meet users’ needs, or should 
different aggregates be introduced? 

• Assessment of whether the current sectoral 
aggregation of the results is appropriate for 
users' needs 
• Assessment of whether other aggregates 
would be more useful 

• Survey Question: “Does the sectoral aggregation of 
the survey results (industry, services, retail trade, 
construction, consumers) meet user needs?” 
• Users’ assessment of sectoral aggregation in 
interviews  

• Stakeholder interviews 
• Online questionnaire 

Q21. Is the disaggregation of the results in 
terms of sub-sectors and consumer categories 
sufficient/appropriate? 

• Extent to which disaggregation is 
appropriate for research / analytical purposes 
and for monitoring economic developments 

Q22. In particular, are microdata on individual 
businesses’ responses needed for up-to-date 
statistical analysis? 

• Extent to which microdata are needed for up-
to-date statistical analysis 

• Survey Question: “Is the disaggregation of survey 
results in terms of sub-sectors and consumer 
categories sufficient?” 
• Users’ assessment of disaggregation in interviews 
• Survey Question: “Would public access to the 
microdata on individual businesses’ responses 
significantly improve up-to-date statistical analysis of 
economic developments?” 

Q23. Is there capacity to adapt to very specific 
needs in particular moments, as was the case 
during the COVID pandemic? Is there a way 
in which ad-hoc survey questions could be 
introduced more rapidly / with less 
administrative burden? 

• Extent to which the programme has adapted 
to specific needs, for example, by adjusting 
the survey collection methodology, 
introducing ad-hoc survey questions, or by 
producing ad-hoc reports on specific issues 
• Availability of new mechanism for swift 
introduction of ad-hoc survey questions in 
surveys 

• Survey Question: “How do you evaluate the 
programme capacity to adapt to very specific needs 
in particular moments, as was the case during the 
COVID-19 pandemic?“ 
• Partner institutes’ assessment in interviews 

• Stakeholder interviews 
• Online questionnaire 

• Desk research (programme 
documentation) 
• Stakeholder interviews 
• Online questionnaire 

• Stakeholder interviews 
• Online questionnaire 

54 

 
 
 
 
 
 
 
 
 
 
 
 
Q24. Has the programme or survey 
methodology been adopted in third countries? 

• Adoption of the BCS methodology in third 
countries 

• Documented references to the methodology of the 
BCS Programme in tendency surveys in third 
countries 

• Desk research 

Q25. Have changes to the programme's 
methodology and coverage enhanced its 
relevance? 

• Degree to which changes in the programme’s 
methodology and coverage enhanced its 
relevance 

Q26. To what extent is the programme 
coherent internally, i.e., between the different 
sectoral surveys and between the different 
countries? Are possibly identified 
incoherencies justified? 

• Coherence between of surveys across 
countries and across sectors.   

Q27. How appropriate are differences between 
the sectoral surveys in terms of the survey 
questions asked and the selection of the 
questions entering a sector’s overarching 
confidence indicator?  

• Extent to which the differences across 
sectors are appropriate 

• Desk research 
• Stakeholder interviews 
• Online questionnaire 

• Desk research (national 
questionnaires, User Guide) 
• Stakeholder interviews 
• Online questionnaire 

• Desk research (national 
questionnaires, User Guide) 
• Stakeholder interviews 
• Online questionnaire 

• Survey Question: “The introduction of the 
Employment expectations Indicator (2020) and 
Economic Uncertainty indicator (2021) have had a 
discernible impact on the programme and its 
outputs?” 

• Differences in (national) survey questionnaires 
across countries 
• Differences between sectoral surveys and sectoral 
confidence indicators in terms of the time horizon to 
which the survey questions refer 
• Assessment of differences by partner institutes in 
interviews 

• Survey question: "The differences between the 
sectoral surveys in terms of the survey questions 
asked (e.g. expected demand in services has no 
matching question in industry survey) and the 
selection of the questions entering a sector’s 
overarching confidence indicator (e.g. services 
confidence includes a question on past developments, 
while industry confidence features only questions 
reg. current/expected situation) are appropriate" 
• Comparison of sectoral developments between 
existing sectoral indicators and alternative indicators 

55 

 
 
 
 
 
 
  
Q28. What is the degree of complementarity 
of the BCS Programme with other EU survey 
programmes, for instance with the ECB’s 
Consumer Expectations Survey and the Bank 
Lending Surveys, as well as comparable 
private/national surveys? 

Q29. What is the additional value of a 
harmonised EU survey programme compared 
to existing national economic tendency 
surveys? 

Q30. How prevalent are the data generated by 
the BCS Programme in discussions and 
analyses of short-term economic 
developments in the EU/euro area and other 
(cross-sectoral) economic analyses? 

Q31. Is the connection between national and 
EU level survey results accurately perceived 
by stakeholders? Is the survey perceived as a 
national or an EU effort in member states or 
candidate countries? 

• Assessment of the usefulness of the BCS 
Programme and its complementary value in 
view of other EU survey programmes 

which use the same survey questions in all surveyed 
sectors26 

• Evaluative judgment of the complementarities 
• Survey question: “The BCS Programme is 
complementary with other EU survey programmes, 
for instance with the ECB’s Consumer Expectations 
Survey and the Bank Lending Surveys, as well as 
comparable private/national surveys?” 

• Desk research (academic 
literature, policy literature) 
• Stakeholder interviews 
• Online questionnaire 

• Benefits of harmonised BCS Programme 
compared to other existing tendency surveys 

• Assessment of benefits of BCS Programme and 
other surveys in the literature 
• Assessment of benefits by users in interviews  

• Desk research (academic 
literature, policy literature) 
• Stakeholder interviews 

• Assessment of the prevalence of the EU BCS 
data for the analysis of short-term economic 
developments 

• Prevalence in the media measured by Google search 
results 
• Citations in academia measured by Google Scholar 
citations 
• Use of BCS data in the private sector as reported in 
interviews 

• Desk research (academic 
literature, policy literature) 
• Stakeholder interviews 

• Extent to which stakeholders are fully aware 
of the EU-level programme and how national 
surveys are part of it 

• Survey Question: “In your opinion, is the survey 
perceived as a national or an EU effort in member 
states or candidate countries?” 

• Desk research (academic 
literature, policy literature) 
• Stakeholder interviews 
• Online questionnaire 

26 Another way of addressing the question would be to impose the structure of one sectoral indicator (e.g., Industry) on another (e.g., Services) and then see whether any differences in 
the development of sectoral indicators become smaller. Unfortunately, this test is not feasible because, for example, there are no current-looking questions in the monthly survey in 
the service sector. 

56 

 
 
 
  
 
 
 
 
 
 
 
Answers to the evaluation questions  

Effectiveness 

Q1/Q2: The frequent use of the BCS data in the media and in academic research as well 
as  the  quantitative  results  indicate  that  the  BCS  Programme  provided  robust  data  that 
allowed for reliable monitoring of economic developments and business cycles in the EU 
Member  States  and  candidate  countries.  This  assessment  was  also  shared  by 
stakeholders,  who  indicated  that  EU  BCS  data  made  an  important  contribution  to 
monitoring and forecasting economic developments in their country. 

Q3:  According  to  the  literature  review,  the  ability  of  the  EU  BCS  data  to  capture 
economic  developments  could  potentially  be  improved  by  data-driven  aggregation 
methods  and  machine-learning  techniques  in  a  statistically  significant  way.  However, 
these methods would lead to frequent changes in the weights; the resulting retrospective 
changes in the time series of the indicators would be difficult to communicate to users. 
Moreover, the Economic Sentiment indicator  already provides valuable information  for 
monitoring  economic  developments  in  the  EU,  as  evidenced  by  the  strong  correlations 
between the ESI and real GDP growth shown above. 

Q4:  The  available  academic  literature,  as  well  as  the  results  of  the  stakeholder 
consultation  and  quantitative  analysis,  concur  that  the  EU  BCS  data  have  been  more 
timely than other leading indicators of GDP, while offering a high degree of correlation 
with the target variable. This combination made them particularly useful for monitoring 
the  real  economy  in  the  EU-27.  In  the  case  of  inflation,  by  contrast,  the  advantage  of 
timeliness was less pronounced for the EU BCS data on price expectations because of the 
early availability of flash estimates of inflation. 

Q5: The academic literature, stakeholders and our own quantitative analysis all conclude 
that  the  EU  BCS  data  have  been  useful  for  nowcasting  and  forecasting  real  economic 
variables. However, both the academic literature and our own quantitative analysis show 
that  there  has  been  some  heterogeneity  in  the  predictive  power  of  the  EU  BCS  data 
across EU Member States. 

Q6/Q7: Stakeholders provided mixed feedback concerning the dissemination of the BCS 
data. Less than half the users (44%) stated that the EU BCS data were easily accessible 
and available in an understandable form, while only 8% of users explicitly disagreed with 
this statement. Users suggested that dissemination could be made more user-friendly by 
offering  search  and  filter  options  and  additional  download  formats  other  than  Excel. 
Furthermore, the huge majority of stakeholders (85%) stated that the supporting material 
of  the  BCS  Programme  was  understandable  and  sufficient.  Nevertheless,  there  is  still 
room for improvement in the supporting material, including more didactic information in 
the User Guide, better reporting of changes and updating metadata and questionnaires. 

Q8:  The  stakeholder  consultations  and  the  review  of  relevant  supplementary  material 
suggest that the intervention has led to an exchange of knowledge between DG ECFIN 
and  the  partner  institutes  as  well  as  knowledge  transfer  among  the  partner  institutes 
themselves.  This  could,  however,  be  rendered  even  more  effective  if  the  annual 
workshops focussed more on the practical aspects of running survey programmes. 

57 

 
 
 
 
Q9:  The  programme  has  created  substantial  methodological  spillovers  and  will  most 
likely continue to serve as a methodological benchmark for other business and consumer 
surveys around the world. 

Q10:  The  changes  to  the  BCS  Programme  in  2012-2021  should  improve  the  BCS 
Programme  in  terms  of  its  effectiveness  in  tracking  and  monitoring  economic 
developments in EU countries. First, a number of changes were made that are justified by 
the previous evaluation of the BCS Programme. Second, several changes at the EU level 
-  which  were  not  based  on  the  previous  evaluation  -  were  made  only  recently,  so  their 
effectiveness cannot yet be assessed. However, the fact that these changes were based on 
thorough ex ante evaluations and were positively assessed by stakeholders suggests that 
these  changes  should  improve  the  effectiveness  of  the  BCS  Programme.  Third,  several 
country-level  changes  (e.g.  the  switch  to  online  survey  methods)  may  have  had 
unintended  negative  consequences  (e.g.  on  response  behaviour),  but  were  necessary  in 
response to demographic change and to save costs. 

Efficiency 

Q11: Taking into account findings from the desk research  and partner institutes’ views 
collected in  the stakeholder consultation,  the design,  implementation  and financing  can 
be  considered  appropriate  overall.  While  the  changes  to  the  design  have  increased  the 
programme’s effectiveness (as evidenced by ex post evaluations), a concern shared by a 
significant  number  of  partner  institutes  relates  to  the  administrative  procedures 
surrounding  the  co-funding  of  the  programme.  Several  institutes  said  that  they  would 
appreciate  if  those  procedures  could  be  simplified,  while  acknowledging  that  complex 
procedures were necessary to ensure full transparency vis-à-vis the taxpayer. 

Q12: The evidence shows that EU spending on the BCS Programme was overall stable in 
recent  years.  At  the  national  level,  however,  some  differences  can  be  observed.  While 
grants  increased  for  some  countries,  other  countries  received  less  funding  from  the  EC 
over time. The reasons behind these trends (such as cost reduction due to reorganisation, 
technical  progress,  enhanced financial  control)  are beyond the scope of this  evaluation. 
Although the benefits of the BCS Programme are hard to quantify, this evaluation finds 
that the BCS Programme has offered substantial benefits to its users in the media, private 
sector, academia and among policy-makers across Europe. The EU BCS data have been 
an essential input to monitoring and forecasting economic developments across European 
countries.  Compared  to  other  existing  indicators  and  surveys,  the  BCS  Programme 
offered  the  unique  advantage  of  a  broader  sectoral  and  geographic  scope.  In  addition, 
publication  of  EU  BCS  data  was  more  timely  and  more  frequent  than  other  indicators, 
which  made  them  particularly  useful  for  policy-makers  who  need  information  on 
economic developments before official data are available. 

Q13:  The  analysis  shows  that  the  digitalisation  of  survey  data  collection  processes  by 
using  the  CAWI  method  offers  cost  reduction  potential.  However,  there  is  still 
conflicting  evidence  on  the  effects  of  CAWI  methods  on  response  rates,  as  well  as 
measurement  error  and  non-response  bias,  which  seems  to  be  one  of  the  reasons  why 
only  14 %  of  institutes  currently  deploy  CAWI  methods  (fully  or  partially  in  a  mixed 
method) in the consumer surveys. Arguably, a clearer verdict by the academic literature 

58 

 
 
 
 
in favour of web surveys would be a precondition for a recommendation to shift towards 
that  data  collection  mode.  In  the  administrative  sphere,  several  partner  institutes 
identified  cost  reduction  potential,  e.g.  by  reducing  the  number  of  cost  factors  to  be 
reported. 

Q14: Big data approaches offer promising new avenues for tracking the economy in real-
time. However, big data approaches cannot offer the same benefits as the EU BCS data 
(e.g. they cannot cover the whole economy, while at the same time providing sector- and 
activity-specific breakdowns) and should thus be considered as a complement to the EU 
BCS data rather than a substitute. 

Q15:  Although  the  current  financial  administration  of  the  programme  received  mixed 
feedback from partner institutes during the interviews, partner institutes understand that 
DG  ECFIN  does  not  have  much  leeway  to  modify  administrative  processes.  However, 
they  believe  that  more  efficient  communication  through  a  centralised  platform  for  the 
exchange of information on administrative aspects of the programme would help lighten 
the administrative burden. A shared platform would allow partner institutes to learn from 
each  other's  experiences  (i.e.  to  see  the  EC's  answers  to  other  partner  institutes)  and  to 
exchange knowledge, methodologies and information with other partner institutes. 

Coherence 

Q26:  The  EU  BCS  surveys  have  been  implemented  coherently  across  countries  and 
differences identified in the questionnaires across countries have been of minor concern. 
This  assessment  was  also  shared  by  partner  institutes  in  the  stakeholder  consultations. 
The  different  uses  of  past-,  present-  and  future-oriented  questions  in  the  sectoral 
confidence indicators have a potential impact on the predictive power of those indicators 
and  on  their  comparability,  which,  however,  turns  out  to  be  minor  (see  answer  to 
question Q27).  

Q27:  Based  on  the  judgment  of  the  users  and  partner  institutes  as  well  as  theoretical 
arguments  from  the  literature,  it  can  be  concluded  that  the  differences  in  the  questions 
asked  in  sectoral  surveys  and  selected  for  the  sectoral  confidence  indicators  are 
appropriate. While the overall development  of the sectoral  indicators  is not  affected by 
the selection of different questions across sectors, they account to only a small extent for 
the differences observed in the development of the sectoral indicators in recent decades. 
However, since economists and policy makers are not per se interested in the differences 
in  the long-term  averages of the sectoral  indicators, but  rather in  the movements of the 
indicators  over  time,  the  effect  identified  here  on  the  differences  between  sectoral 
indicators  is  irrelevant,  and  it  is  reassuring  that  the  correlations  between  the  actual  and 
alternative indicators are very strong, showing that they move closely together. 

Q28:  The  BCS  Programme  complements  other  survey  programmes.  The  programme 
complements the ECB bank lending survey by covering more countries in Europe and by 
providing  a  broader  overview  of  the  current  situation  in  the  financial  services  sector, 
which goes beyond bank lending conditions. The EU BCS surveys complement the ECB 
Consumer  Expectations  Survey  by  providing  long  historic  time  series  on  qualitative 

59 

 
 
 
 
 
consumer  sentiment  that  can  be  usefully  combined  with  the  much  shorter  quantitative 
time series from the CES. The EU BCS surveys also complement the PMI in terms of the 
time  horizons  of  the  survey  questions  and  in  terms  of  the  geographic  coverage,  as  the 
programme  covers  more  countries  in  Europe  while  the  PMI  covers  countries  on  other 
continents. In addition, the PMI is better suited for tracking quarter-on-quarter economic 
growth, while the ESI is best suited for tracking the development of economic activity on 
a year-on-year basis. 

Q29: According to the stakeholder consultation, the BCS Programme, in comparison to 
existing  national  economic  tendency  surveys,  brings  additional  value  to  the  market, 
namely the comparability of its data between countries and sectors, the consistency of its 
data, and its ability to provide harmonised input on the future of the economy at the EU 
level. 

EU added value 

Q30: The EU BCS data have been used widely  by the press,  academia and the private 
sector to monitor, discuss and analyse short-term economic developments.  

Q31:  The  stakeholder  consultation  results  provide  evidence  that  among  users  the 
predominant perception is that the data is the result of an EU-wide effort. Evidence from 
partner institutes is  that EU endorsement is  key to positive perceptions. In particular, it 
seems that the EU-funded nature of the programme lends extra credibility to the data. 

Relevance 

Q16/Q17:  The  EU  BCS  data  remains  relevant  as  an  early  indicator  of  economic 
developments  in  the  EU  Member  States.  Compared  to  flash  estimates,  the  programme 
data has offered the advantages of timeliness, a higher frequency and a broader sectoral 
coverage.  The  broader  sectoral  coverage  as  well  as  a  broader  geographic  coverage  has 
also been the main advantage of the EU BCS data compared to the PMI. 

Q18:  The  feedback  obtained  from  the  stakeholder  consultation  supports  the  hypothesis 
that the BCS Programme focuses on the most important economic questions. While both 
partner institutes and users would appreciate additional questions on specific topics, best 
practice recommends having questionnaires that are as user-friendly and short as possible 
in order not to increase the response burden, which in turn might compromise the quality 
of the data. While deleting questions to make room for new survey questions could be a 
solution, the stakeholder consultation highlighted that this would need to be based on a 
careful  analysis,  as  questions  currently  appearing  irrelevant  might  gain  importance  in  a 
changing economic environment.  

Q19: In the light of the results of the stakeholder consultation, as well as the impact that 
a  higher  surveying  frequency  could  have  on  the  quality  of  the  data  and  the  general 
public’s  interest  in  the  EU BCS  data,  the  conclusion  can  be  drawn  that  the  current 
surveying  frequency is  appropriate.  A higher surveying  frequency  would not  add much 
value to analyses.  Instead, private sector users would appreciate the quarterly questions 
being  moved  to  monthly  frequency.  Some  partner  institutes  also  agree  with  this  as  it 
would  decrease  their  burden  of  having  different  versions  of  the  questionnaires;  while 

60 

 
 
 
 
they do not foresee a risk of this compromising response rates and the quality of the data, 
it would however imply a higher response burden. 

Q19a: It is  not  possible to  determine whether the FSSS complements data provided by 
other financial services surveys. The feedback received from the BCS Programme users 
shows that the FSSS is not relevant to them as it is not integrated in most users’ analyses. 
Potential users of the FSSS acknowledged that they prefer to construct their own indices 
and  use  the  other  sectoral  BCS  Programme  surveys  to  provide  further  input/context  to 
their financial stability analyses. Given the low usage and the users' preference for other 
similar surveys, discontinuation of the FSSS could be considered. 

Q20:  The  current  sectoral  aggregation  of  the  survey  results  meets  users’  needs.  Some 
suggestions  for  improvements  were  voiced  in  the  stakeholder  consultations,  such  as 
taking  an  ecosystem  approach  to  the  aggregation  of  the  EU  BCS  data.  However,  this 
would  pose  a  risk  to  the  integrity  of  the  data  and  time  series  as  current  ones  would  be 
replaced by the new ones. There might be a case for coverage of all NACE sectors across 
the surveys. This would ensure full coverage of all sectors, while users interested in an 
ecosystem  approach  would  build  ecosystem  indicators  on  their  own,  based  on  their 
needs. 

Q21: For the vast majority of users, the current level of disaggregation of the results in 
terms  of  sub-sectors  (i.e.  2-digit  level)  and  consumer  categories  (income,  age,  etc.)  is 
appropriate. 

Q22: The information gathered from the stakeholder consultation showed that microdata 
on individual businesses’ responses would open the door to research topics of interest to 
academics,  private  sector  users  and  institutional  users.  Interested  researchers  can  in 
principle turn to the national data-collecting partner institutes and negotiate the terms of 
the usage of the data as the EC does not have access to partner institutes’ business survey 
microdata.  The  majority  of  partner  institutes  requested  additional  compensation  should 
the EC want to have access to those survey results for centralised dissemination. The risk 
that  sharing  business  microdata  might  pose  to  response  rates,  as  explained  by  partner 
institutes,  might  be  mitigated  by  publishing  the  microdata  with  a  two-  or  three-year 
delay, even though  this might  not  provide relevant  information  for private sector users. 
Considering the unwillingness of some partner institutes to share business microdata and 
the  expected  budgetary  impact  of  any  decision  to  share  microdata  as  part  of  the 
programme, a change to the current contractual set up is not considered appropriate. 

Q23:  Most  partner  institutes  considered  that  they  have  the  capacity  to  adapt  to  very 
specific needs at particular moments. For the particular case of ad hoc questions, it seems 
that  partner  institutes  are  unaware  of  the  existence  of  a  specific  clause  on  ad  hoc 
questions in the applicable Framework Partnership Agreements. The Commission should 
raise awareness of this clause so that it could, jointly with the partner institutes, explore 
the  practical  details  of  the  provision,  inter  alia  how  to  coordinate  the  content  of  the 
(harmonised) ad hoc questions and how to reimburse the partner institutes. 

Q24: The BCS Programme was initially launched by the EC to harmonise business and 
consumer  surveys  across  the  EU  Member  States  to  be  able  to  monitor  and  forecast 
economic developments across Member States and at the aggregate EU level. Over time, 

61 

 
 
 
the programme became a reference for business and consumer surveys around the world. 
The OECD and EU have supported other countries in adopting the programme system. 

Q25:  Tentative  evidence  from  the  literature  and  the  stakeholder  consultation  suggests 
that the introduction of the EUI has improved the relevance of the BCS Programme by 
responding to the increasing need to monitor developments in the perceived uncertainty 
of consumers and businesses. 

62 

 
 
 
 
 
 
ANNEX IV.   OVERVIEW OF BENEFITS AND COSTS AND, WHERE RELEVANT, TABLE ON SIMPLIFICATION AND BURDEN REDUCTION 

Table A4.1: Overview of costs and benefits identified in the evaluation 

Citizens/Consumers 

Businesses 

Administrations 

Partner institutions 

Quantitative  Comment 

Quantitative  Comment  Quantitative 

Comment  

Quantitative 

Comment 

Direct 
compliance 
costs  

Recurrent  

N/A 

N/A 

Costs  

The EC granted 
€ 5 159 222 in total 
on average per year. 

• 

€ 4 966 667 for 
grants for the 
BCS 
Programme.  
€ 192 556 for the 
FSSS.  
•  The amount 

• 

ranged across 
countries from 
€ 30 442 to 
€ 386 468. 

Benefits  

The Commission 
supported the activities 
of the partner 
institutions with action 
grants, which were 
limited to a maximum 
of 50% of the total cost 
of the surveys.  

The figures are average 
values for the period 
2012/13-2020/21. 

The annual costs for 
the implementation of 
the BCS Programme 
amounted to 
€ 10 718 662 for the 
national partner 
institutes. 

Due to country-
specific factors, 
there are 
considerable 
differences between 
country-specific 
costs. 

The costs are 
average values for 
the period 2005-
2010 and are based 
on the last 
evaluation of the 
BCS programme 
from 2012. 

Direct & Indirect 
benefits 

Although it is difficult to quantify the benefits of the programme in monetary terms due to the fact that there is no comparable fee-based programme, 
this evaluation found that the BCS Programme offers substantial qualitative benefits to its users in the media, the private sector (e.g. banks), academia 
and among policy-makers. Firstly, the online survey showed that 100% of the surveyed users considered the EU BCS data to be an essential input for 

63 

 
 
 
       
 
 
 
 
 
monitoring and now-/forecasting of economic developments in their country. Secondly, the EU BCS data are frequently used by academic researchers 
to monitor and forecast GDP and inflation as well as for other analytical purposes. The usefulness of the EU BCS data for monitoring and forecasting 
GDP and inflation was also confirmed in the quantitative analysis conducted by the external contractor. Thirdly, the BCS Programme offers important 
advantages compared to other existing indicators: the publication of the EU BCS data is more timely and more frequent (monthly). Furthermore, when 
compared  to  other  survey  programmes,  the  BCS  Programme  has  a  broader  sectoral  and  geographic  scope,  thereby providing  a  comprehensive  and 
comparable  overview  of  business  cycle  developments  across  European  countries  and  sectors.  It  is  therefore  an  essential  point  of  reference  for 
economic policy in the EU and beyond. 

64 

 
 
 
 
Table A4.2: Potential simplification and burden reduction (savings) 

Citizens/Consumers/Workers 

Businesses 

Administrations 

Partner institutions 

Quantitative  

Comment 

Quantitative  

Comment 

Quantitative 

Comment   Quantitative 

Comment 

Digitalisation of the data collection process  

Type: recurrent 

Reducing administrative burdens in the financing process  

Increasing the digitalisation of the data collection process 
offers cost reduction potential. In order to quantify precise 
savings effects, further research is needed on differences in 
data quality and response rates of different data collection 
methods. 

Suggested improvements include a reduction in the 
number of cost factors that need to be reported, e.g. by 
eliminating separate reporting for each survey, and 
standardising certain cost factors that are only adjusted in 
the event of methodological changes. 

In order to quantify the exact savings effects, further 
research is needed that addresses specific accounting 
issues. 

65 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ANNEX V.   STAKEHOLDERS CONSULTATION - SYNOPSIS REPORT 

This annex presents an outline of the consultation strategy adopted for the external study. 
The  annex  details  the  objectives  of  the  strategy  adopted,  the  stakeholders  and  tools 
identified, and its final dissemination. 

Objectives 

The consultation strategy followed the indications provided by the Commission’s Better 
Regulation Tool #54. 

The specific goals of the consultation activities were to:  

1.  Collect additional data on the implementation of the BCS Programme at the level 
of the Member States and candidate countries, as well as the use of the data by the 
various types of stakeholders; 

2.  Engage  with  stakeholders  involved  in  the  collection  of  the  survey  data  (i.e.  the 
Commission’s  partner  institutes)  and  the  various  stakeholders  using  the  data  in 
their daily work; 

3.  Cross-check and validate the preliminary findings based on the desk-research and 

quantitative analysis. 

In order to ensure transparency and engagement, the process followed the standards and 
methods set out in the Better Regulation Guidelines. The various consultations complied 
with the Terms of Reference (ToR) and the following evaluation roadmap27. 

Identified stakeholders and tools 

The  stakeholders  targeted  by  the  consultation  activities  were,  on  the  one  hand,  the 
Commission’s  partner  institutes  both  in  their  role  as  data  producers  (they  do  the  actual 
survey  work  and  send  the  collected  data  to  the  Commission  for  further  processing  and 
dissemination) and as users (a number of partner institutes use the BCS data themselves 
to  inform  economic  forecasts,  conduct  topical  economic  analyses,  etc.).  On  the  other 
hand, the consultation targeted a wide array of different users of the data, namely:  

▪ 

Institutional  users,  i.e.  public  EU  bodies  (e.g.  the  European  Central  Bank, 
specific  units  in  DG  ECFIN,  as  well  as  other  Directorates-General  of  the 
Commission);  

▪  Private  sector  users  harnessing  the  data  for  macro-economic  forecasting  and 
economic analyses (mainly the research departments of commercial banks) or to 
inform  an  industry-specific  outlook  (e.g.  business  associations  representing  a 
specific sector of the economy);  

▪  Academics using the data for economic research;  
▪  The (economic) press regularly reporting on releases of BCS data (represented by 

both news agencies and newspapers). 

27 The evaluation roadmap ran between 30 July 2021 and 27 August 2021. 

66 

 
 
 
 
 
The consultation strategy employed two different consultation tools, following the ToR 
provisions and agreement with DG ECFIN, namely stakeholder interviews and an online 
questionnaire. 

Stakeholder interviews 

The stakeholder interviews were semi-structured in nature, meaning that guidelines and a 
set of the questions were produced for the interviewers to follow but a level of flexibility 
was  maintained  to  allow  new  ideas  to  be  discussed  depending  on  the  interlocutor.  The 
interview guides were sent to the respective interviewees before the scheduled interview 
to contextualise the call and give the interviewees the possibility to prepare the feedback 
they  wanted  to  highlight.  The  interviews  were  conducted  using  videoconferencing 
software,  such  as  Zoom,  Skype  for  Business  or  Microsoft  Teams.  The  interviews  were 
conducted over a period of 15 weeks (11 July-21 October 2022). 

Online questionnaire 

The  purpose  of  the  online  questionnaire  was  twofold.  First  of  all,  as  time  was  limited 
during the above-mentioned interviews (one hour) and some interviewees provided very 
detailed  feedback,  some  interviews  did  not  cover  all  the  questions  included  in  the 
interview  guide  and  all  the  aspects  of  the  programme.  In  such  cases,  the  online 
questionnaire served as a follow-up which interview participants were asked to fill in in 
order  to  capture  their  answers  to  questions  which  could  not  be  raised  in  the  interview. 
Second, by asking questions like “Do you agree that…”, the online questionnaire allowed 
to capture quantitative data on the appraisal of the BCS Programme which could usefully 
complement  the  qualitative  data  collected  via  the  interviews.  The  bulk  of  the  online 
questionnaires  sent  were  destined  to  the  stakeholders  having  completed  a  stakeholder 
interview, but some invitations to participate in the online survey were also sent to other 
stakeholders.  The  online  consultation  was  conducted  over  the  same  period  as  the 
stakeholder interviews, i.e. over 15 weeks (11 July-21 October 2022). 

Final outreach and delivery of the consultation 

A.  Stakeholder interviews 

A  total  of  269  stakeholders,  across  all  groups,  were  contacted  for  an  interview.  The 
overall  response  rate  was  just  under  40%  (including  negative  responses).  A  total  of  91 
interviews  were  conducted  with  private  sector  users,  institutional  users,  press  and 
academics, and partner institutes.  

Table A5.1: Stakeholder interview key performance indicators  

Partner 
Institutes 

Private 
sector 
users 

Institutional 
users 

Press  Academics    Total 

68 

7 

72 

10 

66 

7 

20 

10 

43 

7 

269 

N/A 

Stakeholders 
contacted 

Weekly 
reminders sent 

67 

 
 
 
 
Partner 
Institutes 

78% 

15% 

Private 
sector 
users 

40% 

37% 

Institutional 
users 

Press  Academics    Total 

42% 

43% 

25% 

20% 

42% 

56% 

39% 

40% 

45 

19 

15 

4 

8 

91 

Response rate28 

Negative 
responses as 
share of 
responses 

Interviews 
conducted29 

Source: Deloitte and DIW Econ 

An overview of the stakeholders which participated in the interviews is provided in Table 
A5.3. 

B.  Online questionnaire 

A  total  of  111  stakeholders  across  all  groups  were  contacted  to  complete  the  online 
questionnaire. The overall response rate was 45%, i.e. a total of 50 respondents provided 
feedback through the online questionnaire, across all stakeholder groups. 

Table A5.2: Online questionnaire key performance indicators  

Partner 
Institutes 

Private sector 
users 

Institutional 
users 

 Press 

Academia    Total 

2 

0 

0 

0 

15 

3 

0 

0 

3 

0 

20 

3 

0% 

0% 

33% 

0% 

0% 

15% 

7 

10 

7 

10 

7 

N/A 

Stakeholders 
contacted only for 
the online 
questionnaire 

No. of responses 
from online 
questionnaire-
only stakeholders 

Response rate of 
online 
questionnaire-
only stakeholders  

Weekly 
reminders sent 

28 Includes negative responses 
29 Includes stakeholders who sent their feedback in writing due  to unavailability for a live interview. The 
number  of  interviews  does  not  correspond  to  the  number of  interviewed  partner  institutes  but  to  the 
number  of  interviewed  stakeholders,  i.e.  in  some  instances  multiple  stakeholders  were  interviewed 
within  the  same  organisation  to  cover  different  aspects  and  elements  of  the  BCS  Programme. 
Stakeholders  from  different  departments  of  the  same  organisation  had  different  uses  for the  data.  In 
some  countries,  there  were  two  partner  institutes  carrying  out  different  surveys  of  the  BCS 
Programme. In the case of the economic press, journalists from the same newspaper were interviewed 
as they were covering the BCS Programme in different countries. 

68 

 
 
 
 
 
Partner 
Institutes 

Private sector 
users 

Institutional 
users 

 Press 

Academia    Total 

45 

23 

19 

10 

15 

5 

4 

3 

8 

6 

91 

47 

51% 

53% 

33% 

75% 

75% 

52% 

23 

10 

9 

3 

6 

51 

49% 

53% 

27% 

75% 

55% 

45% 

Online 
questionnaire sent 
after interviews  

Number of 
responses from 
stakeholders after 
interviews  

Response rate of 
stakeholders after 
interviews  

Total number of 
responses to the 
online 
questionnaire 

Overall response 
rate 

Source: Deloitte and DIW Econ 

Table  A5.3  lists  the  stakeholders  which  filled  in  the  online  questionnaire  and/or 
participated in the stakeholder interviews.  

Table A5.3: Organisations participating in the consultation 

Organisation name 

Interview (Y/N) 

Online questionnaire 
(Y/N) 

CIRET 

Research Institute of Industrial 
Economics (Sweden) 

KOF Swiss Economic Institute 

Netherlands Bureau for Economic 
Policy Analysis 

The Vienna Institute for 
International Economic Studies 

University of Zagreb 

DG ECFIN Country Desk 

Croatia, Spain 

DG ECFIN Country Desk  

Estonia, Latvia, Lithuania, 
Netherlands 

DG ECFIN Country Desk Ireland 

Academics 

Y 

Y 

Y 

Y 

Y 

Y 

Institutional User 

N 

Y 

Y 

69 

N 

N 

Y 

Y 

N 

Y 

Y 

Y 

Y 

 
 
 
DG ECFIN Country Desk Portugal 

DG ECFIN Country Desk 
Romania 

DG ECFIN Unit B3 

DG EMPL 

DG GROW 

ECB 

Eurostat 

Bank of Finland 

Bank of Latvia 

Croatian National Bank 

National Bank of Belgium 

Bank of Ireland 

Central Statistical Bureau of 
Latvia 

Confederation of Finnish 
Industries 

Czech Statistical Office 

Data Collect s.r.o. 

EMCS Malta 

Foundation for Economic and 
Industrial Research, Greece 

GfK SE 

GfK Spółka z ograniczoną 
odpowiedzialnością, Poland 

GKI Economic Research, Hungary 

Institut National de la Statistique et 
des Etudes Économiques, France 

Ipsos GmbH, Germany 

Ipsos Market, Media and Public 
opinion research, Ltd, Croatia 

Ipsos, Belgium 

Istituto Nazionale di Statistica 
(ISTAT), Italy 

Latvian Facts, Ltd. 

Leibniz Institut für 
Wirtschaftsforschung an der 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Central Banks 

Y 

Y 

Y 

Y 

Partner Institutes 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

70 

Y 

Y 

N 

Y 

N 

Y 

N 

N 

Y 

N 

N 

N 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

N 

Y 

Y 

N 

Y 

N 

N 

 
Universität München e.V., 
Germany 

Lithuanian Department of 
Statistics 

National Institute of Statistics of 
Romania 

Österreichisches Institut für 
Wirtschaftsforschung, Austria 

Simple Lógica Investigación, S.A., 
Spain 

Statistics Denmark 

Statistical Office of the Republic 
of Serbia 

Statistical Office of the Republic 
of Slovenia 

Statistical Office of the Slovak 
Republic 

Statistics Finland 

Statistics Poland 

The Malta Chamber of Commerce, 
Enterprise and Industry 

Turkish Statistical Institute 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Economic press and news agencies 

DG ECFIN Unit A4  

Reuters 

Berenberg Economics 

BNP Paribas 

Czech Banking Association 

German Association of Machinery 
Producers 

ING Think 

JP Morgan 

Luminor Group 

Union investment 

Source: Deloitte and DIW Econ 

Y 

Y 

Private Sector 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

71 

Y 

Y 

N 

N 

N 

Y 

Y 

Y 

Y 

N 

N 

N 

Y 

Y 

N 

N 

N 

N 

Y 

Y 

Y 

N 

 
 
 
Results of the consultation  

This section presents the results of the completed consultation activities. 

•  Do  you  think  EU  BCS  data  is  timely  enough?  Is  the  surveying  frequency  (monthly  for  most 

questions, quarterly for some) appropriate? 

INTERVIEWS 

Users 

Partner Institutes 

• 

• 

The data is timely enough as it is available online 
immediately when a press release comes out. The 
survey  frequency  is  good  for  most  respondents. 
However,  some  users  indicated  that  speeding  up 
publication would be good. 
Some  remarks  were  made  on  frequency  during  the 
interviews: even though the monthly frequency is 
fine  under  normal  circumstances,  a  significant 
number  of  users  said  that  the  current  times  of 
crisis  require  a  higher  frequency.  At  such  times, 
weekly frequency could be better; however, there is 
a  general  concern  that  this  could  compromise  the 
quality of answers and response rates.  

•  A  majority  of  respondents  said  that  quarterly 
questions  should  be  moved  to  a  monthly  basis  if 
possible,  especially  due  to  heightened  uncertainty 
since 2020.  

•  Users from the press did not request additional data 
or  a  higher  frequency  as  too  much  data  could  be 
overwhelming and the interest from users would not 
be high enough to have more frequent reports on the 
surveys. 
Some  users  remarked  that  the  publication  time  is 
irregular from one month to the other. 

• 

• 

The  questionnaires  are  very  short  and  most  of  the 
respondents  are  used  to  them.  Consequently,  most 
partner  institutes  consider  that  the  frequency  is 
appropriate  as 
the 
respondents’ perspective.  

is,  both  from 

their  and 

it 

the  response  rate 

•  A  higher  frequency  than  monthly  would  impose 
too  much  of  a  burden  on  the  respondents  and 
would  compromise 
(i.e. 
respondents  need  to  understand  why  they  have  to 
answer  frequently,  and  it  would  be  complicated  to 
justify  a  shorter  responding  period  from  the  partner 
institutes’ perspective). 
Some  partner  institutes  have  decided  to  ask  all 
questions  every  month  to  render  the  processes 
easier rather than having different versions of the 
survey  for  different  times  of  the  year.  Since  there 
are  not  many  quarterly  questions,  this  does  not  add 
much of a burden for respondents. 

• 

ONLINE QUESTIONNAIRE 

• 

• 

93%  of  respondents  said  that  the  EU  BCS  data  is 
timely enough while 7% indicated that it is not.  
100%  of  users  said  that  the  surveying  frequency  is 
appropriate. 

Source: Deloitte and DIW Econ 

• 

87% of partner institutes said that the surveying 
frequency  is  appropriate  while  13%  did  not 
agree. 

•  Do you think the EU BCS data are disseminated in a clear and understandable form? Are there 

sufficient supporting metadata and guidance for users? 

INTERVIEWS 

Users 

Partner Institutes 

• 

The dissemination is clear and straightforward enough 
for  expert  users.  A  majority  of  users  pointed  out 
particularly that the communication of the  programme is 
overall 
survey 
compared 
programmes.  

satisfactory 

other 

to 

•  Academics and other expert users pointed out that it is not 
possible  to  serve  all  target  groups.  In  particular,  non-
expert users might need more guidance. In this instance, a 
more  practical 
the  current 
methodological  focus  would  be  helpful  as  well  (e.g. 
provide examples of how to use the data). 
Private  companies  have  data  providers,  mostly 
DataStream, Ava and MacroBond. Most users access the 

focus, 

rather 

than 

• 

72 

• 

• 

it 

The  programme  follows  a  standard  dissemination 
strategy, which is satisfactory.  
The  clarity  of  the  information  might  depend  on  the 
target  audience  as 
largely  understood  by 
is 
researchers.  However,  the  general  media  does  not 
usually  deep  dive  into  the  data  as  the  raw 
presentation of the data might be more challenging 
for  them.  Many  partner  institutes  pointed  out  that 
guidelines and supporting documentation are targeted 
at stakeholders that are already users of the data. The 
general public might need more detailed guidance. 
•  A  majority  of  partner  institutes  pointed  out  that  the 
visibility of  the  data and  the  programme  could be 

 
 
 
 
 
data from the DG ECFIN website or Eurostat website.  
•  Many users across all stakeholder groups pointed out that 
the  format  of  the  data  could  be  improved  on  the  DG 
ECFIN  website  to  enable  easier  automisation  and 
updating  of  data  sets.  The  current  format  requires  a  lot 
of manual work and is not very user-friendly. 

•  Most  expert  and  academic  users  suggested  that  key 
features to be integrated would be a search function and 
filters  to  avoid  having  to  search  for  specific  data 
manually.  Many  users  said  that  the  Eurostat  website  is 
more user-friendly in its presentation and that it would be 
beneficial if the two websites could be harmonised for a 
better  user  experience  that  would  require  knowledge  of 
only  one  set  of  functionalities.  Others  suggested  that 
Eurostat  disseminate  the  data  directly  for  more  efficient 
communication.  
Integrating  all  the  data  and  aggregates  in  the  Eurostat 
datasets was also mentioned as a potential improvement. 
To  get  around  this,  a  few  users  access  the  data  through 
data aggregator websites which provide the data in better 
format  for  free  (for  example  DB  Nomics).  A  few  users 
also  receive  the  data  directly  from  national  statistical 
offices. 
Some expert users pointed out that the metadata has not 
been  updated  since  2016.  Adapting  metadata  to  have  a 
more  practical  approach  to  the  data  would  be  much 
appreciated by users.  

• 

• 

improved by communicating more on the practical 
use of the data sets. Making the platform more user-
friendly would also help in that regard.  

•  One  partner  institute  said  that  the  PMI  (Purchasing 
Managers’ Index) has a better communication strategy 
than  the  BCS  Programme,  which  means  that  it  also 
has better visibility for the general public and for use 
and promotion of its data. 
Some  partner  institutes  suggested  that  the  Excel 
database  should  remain  available  but  a  more  user-
friendly  presentation  should  also  be  provided.  In 
particular,  filters  and  a  search  bar  to  easily  and 
quickly  find  information  on  a  specific  indicator  or 
variable would be appreciated.  

• 

•  Multiple partner  institutes  mentioned  that  some  users 
might need more guidance, as it depends on the target 
group. 
Partner  institutes  remarked  that  on  the  DG  ECFIN 
website,  metadata  at  country  level  is  outdated  (last 
update was 2016). 

• 

•  Users from the press suggested that the press releases do 
not  provide  much-added  value  as  they  merely  describe 
what  is  in  the  data.  It  would  be  more  interesting  for 
these  users  to  have  more  context  and  for  the  press 
releases  to  be  usable  by  journalists  as  quotes.  In 
particular,  more  context  as  to  why  an  indicator  has 
changed or evolved would be needed.  

•  Overall,  a  significant  number  of  users  across  all 
the  visibility  of  the 

that 

stakeholder  groups  said 
programme could be improved. 

•  Users provided some improvement suggestions: 

Improve the presentation of the data 

• 
•  make  a  search  function  available  for  particular  data 

series 
have all the data synchronised in Eurostat 

• 
•  make the User Guide more practical in how the data 

can be used 

•  make use of AI to create a chatbox to help users find 

their way. 

ONLINE QUESTIONNAIRE 

• 

This  question  was  not  asked  in  the  partner  institutes’ 
online questionnaire. 

• 

• 

• 

44%  of  respondents  agreed  the  data  from  the  BCS 
Programme  is  easily  accessible  and  presented  in  an 
understandable form, while 48% provided no answer and 
7% disagreed. 
11%  of  respondents  indicated  that  they  access  the  data 
directly through the national data provider, 52% access it 
through  the  EU  BCS  website  of  DG  ECFIN  and  22% 
through  the  Eurostat  website.  Others  indicated  that  they 
access 
the  ECB  Statistical  Data 
Warehouse  or  private  data  providers  such  as  Haver  or 
Macrobond. 
85%  of users  indicated  that the metadata  and  supporting 
documentation  provided  are  sufficient  and  appropriate, 
while 11% disagreed and 4% did not provide an answer. 

the  data 

through 

Source: Deloitte and DIW Econ 

73 

 
 
 
• 

In your opinion does the EU BCS enhance partner institutes’ capabilities and contribute to the 
development  of  new  indices  and  products?  Which  capabilities  were  enhanced?  How  could  the 
BCS Programme be modified to improve knowledge sharing? 

INTERVIEWS 

Users 

Partner institutes 

N/A 

• 

• 

• 

• 

institutes  pointed  out 

For a majority of partner institutes, being part of this 
programme  is  a  good  credential,  which  allows 
them  to  have  more  visibility  in  the  market.  This 
can help them secure new contracts as well.  
Several  partner  institutes  indicated  that  the  BCS 
Programme 
is  not  particularly  demanding 
methodologically  so  in  that  sense  it  does  not 
enhance  their  capabilities  (i.e.  usual  processes  for 
other surveys).  
the 
Some  partner 
methodology  and  required  coverage  of  the 
programme  is  quite  rigid  and  does  not  allow  for 
much  innovation,  but  they  do  not  perceive  it  as  a 
barrier.  Instead,  the  rigidity  allows  harmonisation 
across  geographies  and  stability  for  long  time 
series,  and  these  are  the  most  important  aspects  of 
the programme. 
The diversity of partner institutes participating in 
this  programme  is  an  added  value  in  learning 
different uses of the data and sharing experience 
on technical issues. This is particularly appreciated 
during the annual workshop. 

that 

•  Multiple  partner  institutes  indicated  that  having  a 
centralised  platform  at  the  EC  level  would  help 
knowledge  sharing.  This  should  be  an  interface 
through  which  the  EC  and  partner  institutes  could 
communicate,  for  example  on  changes  of 
the 
methodology  or  addition  of  questions.  Even  though 
the annual workshop is very helpful, it remains very 
academic  and  the  number  of  participants  prevents 
sharing on a more practical level. Such a platform, or 
more regular workshops, would help in this regard. 

ONLINE QUESTIONNAIRE 

• 

This  question  was  not  asked  in  the  users’  online 
questionnaire. 

• 

Source: Deloitte and DIW Econ 

indicated 

respondents 

95%  of 
the  BCS 
Programme  enhances  partner  institutes’  capabilities 
and  contributes  to  the  development  of  new  indices 
and products, while 5% indicated that it does not. 

that 

•  To  what  extent  was  the  design,  implementation  and  financing  of  the  programme  appropriate? 

Have changes improved its appropriateness and efficiency? 

INTERVIEWS 

Users 

Partner institutes 

N/A 

• 

Some countries have specific teams who take care of 
the administrative and financial parts, so the financing 
and  design  of  the  programme  is  not  a  problem  for 
them.  Some  countries  see  some  improvements  from 
the  past,  especially  in  terms  of  the  paperwork 
burden.  

•  Multiple  countries  identified  an  issue  in  the  move 
from  “traditional”  financing  (i.e.  via  a  call  for 
proposals  through  which  DG  ECFIN  selected  one 
partner institute and then cover the full amount) to the 

74 

 
 
 
 
 
current  partnership  agreement  (i.e.  annual  award 
through  a  grant  agreement)  where,  since  2021,  there 
has been a unit cost per staff category which did not 
allow  for  correction  for  any  labour  cost  inflation 
suffered  after  the  COVID-19  pandemic  in  2020  as 
unit  costs  are  calculated  on  the  basis  of  historic 
payroll  data  and  thus  refer  to  staff  costs  incurred 
during  the  reference  year  2020  (i.e.  the  financial 
management  was  considered 
the 
summary  of  costs  did  not  reflect  the  actual  costs 
incurred,  but  rather  those  that  would  have  been 
incurred if salaries had not changed since 2020.  
Several countries pointed out that the administrative 
paperwork  is  very  time-consuming  and  was  more 
efficient in the past. 

inefficient  as 

• 

•  All  partner  institutes  are  now  requested  to  estimate 
the costs in a very detailed manner, in terms of labour 
costs,  telephone,  etc.  This  has  to  be  based  on  a 
previous  reference  year 
to  develop  a  detailed 
estimation  of  costs.  At  the  end  of  the  period,  they 
have  to  justify  the  actual  working  time  spent  by  the 
different  staff  categories.  This  works  when  the 
economy is stable, with low inflation, as there are no 
major  changes  in  salaries.  However,  the  costs  are 
currently  significantly  affected  by  salary  increases 
and high inflation rates.  

•  Overall, the recent changes have been criticised by 
several  partner  institutes.  The  financial  design  is 
rigid  and  does  not  help  their  innovative  capacity. 
Indeed,  when  questions  are  added,  the  funding  grant 
goes down.  

•  Multiple  partner  institutes  indicated  that  having  a 
centralised  platform  at  the  EC  level  would  help  the 
programme  management  be  more  efficient.  An 
interface in which the EC and partner institutes could 
communicate,  for  example  on  changes 
the 
methodology or the addition of questions would help 
communication between the partners (see above) and 
could also be used for administrative purposes.  

in 

ONLINE QUESTIONNAIRE 

• 

This  question  was  not  asked  in  the  users’  online 
questionnaire. 

• 

52% of respondents answered that the changes in the 
design of the programme have improved its efficiency 
while 48% said that they have not.  

Source: Deloitte and DIW Econ 

•  Do you have a use for the FSSS? If yes, in your view, does the Financial Services Sector Survey 

complement other data provided by other financial services surveys?  

INTERVIEWS 

Users 

Partner institutes 

•  None of the users interviewed through stakeholder 
consultations were able to provide feedback on the 
FSSS as they did not have a use for it. Stakeholders 
consulted  on  this  question  included  national  central 
banks,  private  sector  companies,  academics  and 
researchers as well as institutional users.  
Some  central  banks  and  commercial  banks  pointed 
out  that  they  have  developed  their  own  internal 
financial  surveys  tailored  to  their  specific  needs, 
them. 
which  rendered 
However,  they  mentioned  that  their  analysis  is 

the  FSSS  redundant  for 

• 

• 

• 

The partner institute responding to this survey did not 
have  visibility  on  the  users  of  the  data,  but  shares 
the two reports produced each year with the panel of 
respondents  to  keep  them  engaged.  This  respondent  
mentioned  that  giving  more  visibility  to  the  data 
would provide encouragement to respondents.  
Partner institute is aware of reports produced and sold 
by  private  companies  based  on  the  data.  Partner 
institutes  does  not  have  visibility  on  similar 
potential competing products on the market. 

75 

 
 
 
• 

complemented by the five other surveys in the BCS 
Programme  which  provide  a  forward-looking 
perspective to their financial stability analysis and 
help  them  assess  the  risks  from  the  macro 
environment and sectoral credit risks. 
These  stakeholders  also  pointed  out  that  information 
provided  by  the  financial  sector  is  highly  sensitive 
and  that  financial  companies  are  hesitant  about 
sharing 
full  control  and 
visibility  of  the  context  of  the  survey,  the  questions 
asked  and  the  profiles  of  the  respondents  were 
identified as key elements for making the data reliable 
enough  for  users’  needs,  and  having  all  the  relevant 
information available for accurate analysis. 

information.  Having 

ONLINE QUESTIONNAIRE 

to 

•  Responses to the online questionnaire confirmed these 
insights  as  37%  of  respondents  answered  “Do  not 
know” 
the  FSSS 
the  question  of  whether 
complements  other  data  provided  by  other  financial 
services  surveys,    while  18%  “Agreed”  or  “Strongly 
Agreed”.  44%  of  respondents  did  not  answer  as  the 
question was included in the survey after it had been 
launched.  

Source: Deloitte and DIW Econ 

• 

This  question  was  not  asked  in  the  partner 
institutes’ online survey. 

•  Does the sectoral aggregation of the results meet your (users) needs? Should different aggregates 

be introduced? If that’s the case which ones do you consider the most useful? 

INTERVIEWS 

Users 

Partner Institutes 

• 

• 

from 

the  official  classification 

The  sectoral  aggregation  is  good  as  such,  as  it 
follows 
the 
European statistical offices.  
Some  users  voiced  specific  opinions  during  the 
interviews: some questions are not asked across all 
sectors (e.g. capacity utilisation is asked for industries 
and services but not construction), which can limit the 
coverage  of  the  questions.  A  few  users  did  not 
consider these differences justified. 

•  Most  academics  and  other  expert  users  said  that  the 
sectoral  aggregation  should  not  be  changed  too 
quickly  as  most  models  are  based  on  this  type  of 
structure  and  it  would  add  a  burden  for  researchers 
and other users.  
Some  users  pointed  out  that  classifying  industries 
by product types is not always the most useful for 
economic interpretation. 

• 

•  Most  users  said  that  if  any  changes  were  to  be 
implemented,  sample  size  should  be  taken  into 
consideration,  especially  for  small  countries,  as  a 
small sample size would compromise the results.  

•  A  large  majority  of  partner  institutes  said  that  the 
sectoral aggregation is good as such, as it follows the 
official  classification  from  the  European  statistical 
offices.  
Some  partner  institutes  pointed  out  that  the  services 
aggregate could be further broken down as it is too 
broad.  

• 

•  One partner institute pointed out that some industries 
are misclassified: 45-2 (Casting of steel/Maintenance 
and  repair  of  motor  vehicles/Maintenance  and  repair 
light 
of  motor  vehicles)  and  45-3  (Casting  of 
metals/sales 
and 
motor 
accessories/wholesale trade of motor vehicle parts and 
accessories/retail  of  motor  vehicle  parts  and 
accessories). 

vehicles 

of 

ONLINE QUESTIONNAIRE 

• 

96% of respondents said that the sectoral aggregation 
of the survey results is appropriate, the remaining 4% 
did not provide an answer. 

• 

This question was not asked in the partner institutes’ 
online questionnaire. 

76 

 
 
 
 
 
 
 
• 

Is  the  disaggregation  of  the  results  in  terms  of  sub-sectors  and  consumer  categories 
sufficient/appropriate? 

INTERVIEWS 

Users 

Partner Institutes 

Partner institutes had received  no negative feedback 
from  users  so  far.  They  had  received  requests  from 
clients  for  more  regional  disaggregation,  which  they 
cannot provide.  
Some  partner 
classifying 
aggregates  at  the  letter  level  rather  than  at  a 
double-digit level. 
Partner  institutes  in  most  countries  said  that 
disaggregating results below the 2-digit level would 
not  be  useful  as  it  would  increase  noise  and  the 
sample size would be too small.  

suggested 

institutes 

• 
• 

• 

• 

• 

• 

• 

The disaggregation is enough as it is for forecasting.  
Expert users voiced an interest in having sub-sectoral 
data with disaggregation that goes beyond the NACE 
two-digit level but they understood that this would be 
difficult to implement in practice.  
Expert users and academics said that microdata 
would be of more interest than further 
disaggregation by ECFIN so that each user could 
aggregate/disaggregate the data as needed.  
Some  expert  users,  mainly  from  the  private  sector, 
asked  for  specific  changes,  such  as  separation  of  the 
shortage  of  materials  and  equipment  as  factors 
limiting  building  activity  in  the  construction  and 
industry  surveys,  further  differentiation  between 
traditional  and  e-commerce,  and  separation  of  SME 
and  MNE  data.  In  particular,  services  is  too  broad  a 
categorisation  and  should  be  further  broken  down. 
Some  users  pointed  out  that  some  services  (e.g. 
hospitality  or  banking)  are  prevalent 
in  some 
countries  but  not  in  others.  A  further  breakdown  of 
services would therefore be useful.  

•  Other  users  remarked  that  further  disaggregation  of 
investment  and  expected 
investment  would  be 
interesting  for  forecasting.  The  interest  is  mainly  in 
knowing where the expected investments are directed 
(internal (e.g. staff, machinery, R&D) / external (e.g. 
acquisition of other companies)) 

ONLINE QUESTIONNAIRE 

• 

of 

respondents 

the 
89% 
disaggregation  of  survey  results  is  sufficient  while 
11%  indicated  that  further  disaggregation  would  be 
needed.  

considered 

that 

• 

This question was not asked in the partner institutes’ 
online questionnaire. 

Source: Deloitte and DIW Econ 

•  How far is access to the microdata of EU BCS needed for up-to-date statistical analysis? In light 
of  repeated  requests  from  researchers  to  get  access  to  the  micro-data  underlying  the  business 
survey, would you be willing to share that microdata with the Commission for dissemination? 

INTERVIEWS 

Users 

Partner Institutes 

•  A  majority  of  users  considered 

the 
programme should be as open as possible with its 
data  collection  and  data  publication  to  enrich 
research  projects.  This  includes  the  publication  of 
microdata. 

that 

•  Growing  uncertainty  increases  the  need  and  case  for 
access  to  microdata.  If  access  to  microdata  cannot 
be  granted,  academic  users  said  that  3-digit  level 
data would be very valuable. 

•  Microdata  would  be  useful  to:  study  how  the  price 
expectation/inflation  expectation  is  created  (i.e. 
how  expectations  are  formed);  analyse  shortages  in 
the  labour  market  for  skilled workers;  pinpoint  a 
moment  and  the  direct  impact  of  a  shock  (i.e. 
impact  of  business  behaviour);  build  experimental 
models  (regression  type  models  and  multivariable 

77 

• 

Some  partner  institutes  do  not  want  to  share  the 
microdata  as 
the 
respondents not to share that information. 

they  have  agreements  with 

•  A  majority  of  partner  institutes  said  that  if  access  to 
microdata  were  to  be  granted,  the  contract  between 
partner  institutes  and  the  European  Commission 
should  state  that  and  the  partner  institutes  should 
be compensated for providing it.  

•  A  significant  number  of  partner  institutes  formulated 
some  concerns  about  this  and  made  the  following 
points: 

•  Access  and  use  should  be  granted  only 

to 

researchers 

• 

The  data  provided  by  the  survey  is  sensitive  as  it 
into  companies’  economic 
provides 

insights 

 
 
 
• 

analysis).  
Some  users  suggested  that  anonymised  microdata 
would  not  be  useful  as  it  defeats  the  purpose  of 
having  access  to  microdata.  Indeed  microdata  would 
be most useful if it were complemented by the context 
of the respondents. Anonymised data would therefore 
not be as useful. However, users understand that non-
anonymised data cannot be provided. 

situation,  their  expectations  and  indirect  insight 
into their strategy. Providing access to microdata 
could  affect  response  rates  as  companies  could 
become reluctant to provide that information. 

•  All partner institutes said that any microdata should 
be  anonymised,  but  many  pointed  out  that  this 
might not be  enough,  especially in  certain  smaller 
sectors  and  countries,  where  it  would  still  be 
possible to identify respondents. 

•  One  partner  institute  suggested  publishing  the 
microdata with a time lag of 2-3 years rather than 
current microdata to avoid the pitfalls cited above. 

• 

Partner  institutes  in  some  countries  have  already 
implemented  an  application  process  for  users  to 
request access to microdata. 

ONLINE QUESTIONNAIRE 

• 

59%  of  respondents  “Agreed”  or  “Strongly  Agreed” 
that  access  to  microdata  would  improve  up-to-date 
statistical analysis of economic developments. 19% of 
respondents  “Disagreed”  or  “Strongly  Disagreed” 
while 22% answered “Do not know”. 

Source: Deloitte and DIW Econ 

• 

This  question  was  not  asked  in  the  partner  institutes’ 
online questionnaire. 

• 

Is there a capacity to adapt to very specific needs in particular moments, as was the case during 
the COVID-19 pandemic? Is there a way in which ad-hoc survey questions could be introduced 
more  rapidly/  with  less  administrative  burden?  Would  you  have  a  concrete  idea  how  what  an 
easy system for including ad hoc questions could look like? 

INTERVIEWS 

Users 

Partner institutes 

•  N/A 

• 

• 

Partner  institutes’  answers  to  this  question  were  quite 
divided and diverse. 
Some partner institutions expressed the view that there 
is  no  capacity  for  this  as  it  is  too  complex.  It  implies 
new requirements and agreements in the contract as the 
current  administrative  and  financial  burdens  are  too 
rigid to allow this flexibility.  

•  Obtaining  reliable  information  would  be  more 
difficult  as  new  questions  would  need  to  be  tested 
first  and  respondents  might  not  respond  given  the 
unexpected additional burden. If the set of questions 
were 
this 
compromising the response rate. Any change would be 
costly  to  implement  and  the  quality  of  the  questions 
and their insights would have to take priority to ensure 
the  integrity  of  the  data  rather  than  adding  a  new 
question rapidly. 

there  would  be  a  risk  of 

long, 

too 

• 

• 

•  Others  were  undecided  and  said  that  introducing ad-
hoc  questions  when  surveys  are  conducted  online 
should not be a problem, but more time and resources 
would be needed for printed surveys.  
Finally,  some  partner  institutes did  consider  that  there 
is the capacity to add ad-hoc questions.  
Some countries have already implemented solutions 
for  respondents  to  identify  ad-hoc  factors  that 
influence  their  business  decisions.  One  country  has 
adapted one of the questions on identification of factors 
limiting  economic  activities.  The  question  lists  a  few 
factors  and  respondents  can  also  click  ”other”.  If they 
do  that,  they  have  to  write  down  the  factor.  In  the 
period  leading  up  to  this  study,  many  answered  war, 

78 

 
 
 
 
 
inflation  or  COVID-19.  This  approach  allows  new 
factors to be identified, but does not require addition of 
a new question.  

ONLINE QUESTIONNAIRE 

• 

This  question  was  not  asked  in  the  users’  online 
questionnaire. 

• 

61% of respondents answered that it would be possible 
to adapt rather quickly with little administrative burden 
or  very  quickly  with  almost no  administrative burden. 
The  remaining  39%  indicated  that  adaptation  would 
either be rather slow or very slow, with a considerable 
to very large administrative burden. 

Source: Deloitte and DIW Econ 

•  What  is  your  perception  about  the effort  needed  to  take  the  programme  forward? Is  the 
connection between national and EU survey results accurately perceived by stakeholders? Is the 
survey perceived as a national or an EU effort in member states or candidate countries? 

INTERVIEWS 

Users 

Partner institutes 

• 

The  users’  perception  of  the  programme  largely 
depends on how they access the data. If they receive 
the  data  directly  from  national  statistical  offices  and 
only use their national data, they view the surveys as a 
national  effort.  When  users  access  the  data  through 
the  DG  ECFIN  website  or  Eurostat  and  make  use  of 
the entire data to compare data between countries, it is 
clear to them that it is an EU-wide effort. 

• 

• 

•  Where  national  statistics  institutes  did  not  have 
similar  data  before  the  programme,  it  is  clearer 
that  it  is  a  European  Commission  programme.  In 
some  countries,  such  as  Germany,  where  similar 
indices  were  already  provided  by  national  statistical 
offices, the perception is that this is now a joint effort. 
•  A  majority  of  users,  across  all  stakeholder  groups, 
highlighted  that  the  strength  of  the  programme  is 
cooperation in data collection and data publication 
which enables comparability at EU scale.  

From  the partner institutes’ point  of  view,  it  does  not 
appear  that  users  pay  attention  to  this.  In  some 
countries,  respondents  to  the  surveys  think  that  it  is  a 
local initiative, even though it is clearly communicated 
that it is funded by the EU.  
In countries where the Business and consumer surveys 
were  already  provided  before  the  BCS  Programme, 
users  might  think  that  there  are  two  separate  surveys 
and  not  understand  that  they  are  part  of  the  same 
programme.  

•  A  number  of  partner  institutes  indicated  the  feedback 
they  received  is  that  when  users  are  aware  that  the 
programme is funded by the EU, they are not interested 
in  knowing  more  as  they  consider  the  EC  a  reliable 
data source, and it is that which matters to them most. 

ONLINE QUESTIONNAIRE 

• 

15%  of  respondents  answered  that  the  survey  is 
perceived as a national effort, while 70% indicated it 
is  perceived  as  an  EU  effort.  The  remaining  15% 
answered “Do not know”. 

• 

43%  of  respondents  answered  that  the  survey  is 
perceived as a national effort, while 43% indicated it is 
perceived  as  an  EU  effort.  The  remaining  13% 
answered “Do not know”.  

Source: Deloitte and DIW Econ 

79 

 
 
 
 
 
ANNEX VI.   CHANGES IN THE BCS PROGRAMME SINCE 2012 

Table A6.1: Overview of changes in the BCS Programme since 2012 

Year  Changes to the BCS survey 

2012  Introduction of new question on capacity utilisation in the services survey. 

2013  The Italian partner institute (ISTAT) introduces some methodological 

improvements in sampling and survey techniques. 

German services data revised to be fully in line with the NACE rev.2 statistical 
classification of economic activities.  

Croatia included in the EU aggregate. Historical values, as well as country 
weights, revised accordingly. 

2014  Latvia joined euro area on 1 January 2014 and was included in euro area 

aggregates. 

2015  Revision of Romanian investment survey data between 2011 and 2015. 

Change of partner institutes in Serbia and Türkiye.  

Lithuania joined the euro area on 1 January 2014 and was included in the 
euro area aggregates. 

Portuguese consumer data based on a new sample. For the back-casting of the 
series, the two samples were collected simultaneously between November 2014 
and October 2015. 

Revision of the Bulgarian data for the five investment surveys conducted between 
March/April 2013 and March/April 2015. 

British partner institute (CBI) updated the sampling weights for the industry, 
investment, retail and services surveys in line with changes in officially available 
data from various UK government sources. 

Historical consumer survey series for Ireland revised from 2003 to 2015 for 
questions 1, 2, 4, 7, 8 and 10). 

2016  Correction of discrepancies between partner institute (ISTAT) and DG ECFIN 

data for the industry, services, building and retail trade surveys. 

  Consumer categories PR0 to PR9 discontinued Europe-wide.  

  Due to a revision of the breakdown by occupation of the respondents as of May 

2016, time series corresponding to consumer categories PR0 to PR9 discontinued 
until further notice. 

  Statistics Portugal publishes the results for the services, construction, industry and 

trade surveys based on new samples and sampling frames. 

  Change of partner institutes in Ireland and Montenegro. 

  Correction of Slovenian investment structure data for the two aggregates Food and 

beverages industry (FOBE) and Consumer goods (CONS) for 2013, 2014 and 
2015. 

  In October 2016, French partner institute (INSEE) modified the industry capacity 

80 

 
  
 
 
 
 
 
 
 
utilisation data (Q13) back to October 2004 to correct a break in the series 
introduced by the questionnaire harmonisation in 2004.  

  Correction of data for Q10 of the Swedish industry survey (at total level). 

  Correction of Dutch data for the industry, retail trade and services surveys 

between January and June 2016. 

  Correction of French data for the services survey between January 2013 and May 
2016, with significant revisions mostly for three NACE2 sub-sectors (56, 68 et 
96). 

  French partner institute (INSEE) introduced a modification of the secondary 

weights used when computing the industry survey balances. 

2017  Revision of the design of the Dutch consumer survey. 

Change in weighting procedure in Türkiye. 

Revision of Italian data for services due to the inclusion of sub-sectors 75 and 90 
to 96.  

2018  Revision of past data for Germany back to 1991, reflecting changes in the 

aggregation of firm-level data and the inclusion of late responses implemented by 
the data provider (Ifo Institute).   

  Revision of question 2 of the construction survey on factors limiting building 

activity in France.  

2019  European aggregate recalculated and UK data excluded after the UK construction 

survey was halted in November 2019. 

Ireland included in the European aggregates.  Historical values, as well as the 
country weights, revised accordingly. Country weights used to calculate the EU 
and the euro area aggregates updated. 

Structural change in the way consumer data is collected in Finland and Germany.  

Change of partner institute in Austria. 

Revision of the Consumer Confidence Indicator. 

Correction of Hungarian data for question 8 of the industry survey on factors 
limiting activity. 

2020  Temporary changes in survey modes due to COVID-19. Containment measures 
resulted in lower response rates than usual. The partner institutions took different 
approaches to dealing with non-response, which led to several revisions. 

  All EU aggregates calculated on the basis of 27 Member States (i.e. excluding the 
UK, which withdrew from the EU on 31 January 2020). Historical values of EU 
series revised accordingly. 

  Structural change in the way consumer data is collected in Sweden.  

  Correction for a change in several questions on order books and stocks in the 
industry, construction and retail trade surveys in Denmark in 2014 in order to 
harmonise them with the BCS Programme guidelines. 

  Starting year of the standardisation window used for the construction of the ESI 

changed to 2000. 

81 

 
 
  
  Launch of an 'Employment Expectations Indicator' (EEI), which condenses 

the employment expectations in industry, services, retail trade and 
construction into a composite indicator. 

2021  Factors limiting production (Q2) revised for Belgium. 

  Revisions of Equipment & Material (F5) and other factors (F6) as factors limiting 

production (question Q2) for Finland and France. 

  Following the introduction of a new sampling method and weighting procedure, 

time-series related to the Turkish services, construction survey back-casted for the 
period until December 2020. 

  Misallocation in Latvian data of survey results from answers to the categories 

BUIL Q2 remedied.  

  Introduction of the survey-based Economic Uncertainty Indicator. 

Country weights used to calculate the EU and the euro area aggregates updated. 

Back-cast conducted of the Slovenian time series most affected by the change in 
the consumer survey collection method in 2016. 

Revision of the weighting scheme of the Italian consumer survey. 

Change from quantitative to qualitative questions in the investment survey 
(DG ECFIN, 2022) 

Revision of factors limiting production (Q8) for Portugal 

Revision of Equipment & Material (F4) as a factor limiting production (question 
Q8) for Belgium, Finland and France. Revision of Other factors (F5) as factors 
limiting production (Q8) for Belgium and Finland. 

Revised weighting scheme introduced for the answers to the financial services 
sector survey.  

2022  Change in seasonal adjustment procedure and revision of affected data. 

Revision of factors limiting production (Q8) for Czechia, Denmark, Italy, 
Hungary, Montenegro and Sweden. 

Correction of the French industry survey results for “shortage of material and/or 
equipment” (Q8-F4) for October 2021. 

2023  Discontinuation of the Financial Services Sector Survey (FSSS) 

Source: Deloitte and DIW Econ 

82