DG ECONOMICS, STATISTICS AND RESEARCH – BANCO DE ESPAÑA

INTERVIEWING METHODS AND DATA QUALITY: THE 

CASE OF THE SPANISH SURVEY OF HOUSEHOLD 

FINANCES (EFF) 2020

Laura Crespo

Sandra García-Uribe

Elena Vozmediano

Survey of Household Finances Unit

Microeconomic Studies Division

Banco de España

2023 FCSM RESEARCH & POLICY CONFERENCE

25 October 2023
DG ECONOMICS, STATISTICS AND RESEARCH-BANCO DE ESPAÑA 2

MAIN IDEA

• The EFF is a nationally representative survey conducted by the BdE since 2002 every three years (every two since

2020) that contains detailed information on household assets, debts, income and consumption for the population

of households living in Spain.

• Given the complexity and the length of the questionnaire, interviews are computed assisted (CAPI) and conducted

by interviewers with specific training.

• However, the health and mobility restrictions imposed by

the COVID-19 pandemic led us to revise some

methodological aspects of the 2020 edition, mainly, to

switch the interviewing mode from CAPI to CATI.

• The goal of this analysis is to estimate the overall impact

of this change for the 2020 edition on the quality of the

data collected.
DG ECONOMICS, STATISTICS AND RESEARCH-BANCO DE ESPAÑA 3

MAIN IDEA: CAUSAL INFERENCE FROM OBSERVATIONAL DATA

• To assess the mode effect we compare data quality indicators based on telephone interviews from the EFF2020

with those based on the personal interviews conducted during its preceding wave, the EFF2017.

• Method’s main idea: we adjust for differences in the covariates between the treated and the control sample units

using a weighting procedure based on the propensity score of being treated.

• The comparison of data quality indicators between the 2020 (treated) sample and the reweighted (control) 2017

sample provides an estimate of the overall impact of the change in the interviewing mode.

• This analysis is useful to:

• Assess the comparability of the EFF2020 data with previous waves.

• Open the possibility of implementing mixed modes in the data collection for subsequent waves.
DG ECONOMICS, STATISTICS AND RESEARCH – BANCO DE ESPAÑA

1. The EFF: purpose and methodological features

2. The EFF2020: a survey in the middle of the pandemic

3. Estimation strategy

4. Average effect on: 

• Interviews length

• Item non-response 

• Respondents’ attitude. 

5. Takeaways

6. 5. 

OUTLINE

4
DG ECONOMICS, STATISTICS AND RESEARCH-BANCO DE ESPAÑA 5

A BRIEF INTRO TO THE EFF

• The EFF is a nationally representative survey conducted by the BdE since 2002 that contains detailed information on

household assets, debts, income and consumption for the population of households living in Spain.

• Frequency: 

• Every three years (2002, 2005, 2008, 2011, 2014, 2017, 2020).

• Every two years since 2020.

• The field for the eight edition (2022) has just finished this June. More to come in 2024, 2026,...

• Sample size: around 6300 HHs interviewed in each wave. Out of those, around 50-60% corresponds to panel 

households (HHs interviewed at least in the previous wave).

• Main characteristics:

• Oversampling of wealthy households based on individual wealth tax files.

• Panel component.

• Full CAPI in the period 2002-2017; full CATI in 2020; mixed mode (CAPI first option) in 2022. 
DG ECONOMICS, STATISTICS AND RESEARCH-BANCO DE ESPAÑA 6

A BRIEF INTRO TO THE EFF

• The questionnaire: the info that we get

1. Demographic characteristics (all members of the HH)

2. Real assets and associated debts 

3. Other debts

4. Financial assets

5. Insurance and pension plans

6. Labour market situation and related individual income (all adult members. Income referring to the moment of the interview 

and to the whole calendar year previous to the interview)

7. Incomes not related with labout market (for the moment of the interview and to the calendar year previous to the interview) 

8. Means of payment

9. Expenditure

10. In the EFF2020 a new section on the impact of the covid-19 crisis on the economic situation of households was included.

• Information is collected at the household level (except demographics and labour market outcomes).

• It is reported by the so-called “reference person of the HH”, i.e., the most knowledgeable HH member about the

household’s finances.
DG ECONOMICS, STATISTICS AND RESEARCH-BANCO DE ESPAÑA 7

EFF 2020: A SURVEY IN THE MIDDLE OF A PANDEMIC

• Seventh edition of the survey, following those of 2002, 2005, 2008, 2011, 2014 and 2017 (biennial frequency from 2020 

on)

• Interviews were collected from mid November 2020 until end of June 2021

• The pandemic mobility and health restrictions led us to change several aspects related to the fieldwork:

• The interviewing mode from CAPI to CATI (most salient change).

• Longer duration of interviewers’ training and more time to practice.

• Close monitoring of interviewers’ performance (e.g., through audio records).

• The number of valid interviews was 6313, 60.7% corresponds to panel households (hhs inteviewed in 2017)
DG ECONOMICS, STATISTICS AND RESEARCH-BANCO DE ESPAÑA 8

ESTIMATING THE EFFECT OF CATI (VS. CAPI) ON DATA QUALITY FROM OBSERVATIONAL DATA

• We study the effect of the telephone mode (as opposed to F2F) on several outcomes of interest:

• Time durations of different sections at the interview level

• Interview completeness ratios: point-value ratio and wide ratio

• Prevalence of item non-response for specific monetary questions

• Respondents’ attitudes when completing the interview.

• Limitations:

• No experimental design or random assignment to treatment.

• The telephone mode was applied to all interviews in the EFF2020 so there is not within sample variation.

• Given the lack of an experimental design, what we do is to use as control the sample of the EFF2017.

• The EFF2017 represents a good benchmark to compare with since it is the nearest wave in time to the EFF2020 

edition and it is similar regarding important dimensions of the fieldwork (e.g, fielwork company and the pool of 

interviewers).
DG ECONOMICS, STATISTICS AND RESEARCH-BANCO DE ESPAÑA 9

ESTIMATING THE EFFECT OF CATI (VS. CAPI) ON DATA QUALITY: MAKING SAMPLES MORE SIMILAR

• In addition, we use those covariates to reweight observations in the control sample (EFF2017) so that the

characteristics of both the treatment and control samples are similar at least in observables.

• Weights are such that they increase the relevance of those observations in the control sample that are more similar in

observables to those in the treated (telephone) sample, where similarity is defined by the predicted probability of

being CATI as computed from a logit model estimated using both samples.

• BASIC IDEA: to make both groups more similar in terms of observables so that we can use the EFF2017 sample as

counterfactual for the treated sample.

• BUT: Interviews in the treatment and control groups belonged to different contexts, samples and fieldworks so that

we adjust for confounding using an extensive set of covariates that measure households’ characteristics, interviewers

characteristics, and municipality size:

• Age, education and gender of Reference Person (RP)

• Number of household members

• Proxy respondent

• Ownership status of main residence

• Municipality size

• Interviewer characteristics: gender, experience in the EFF, years of experience at the survey agency
DG ECONOMICS, STATISTICS AND RESEARCH-BANCO DE ESPAÑA 10

ESTIMATING THE EFFECT OF CATI (VS. CAPI) ON DATA QUALITY: MAKING SAMPLES MORE SIMILAR
DG ECONOMICS, STATISTICS AND RESEARCH-BANCO DE ESPAÑA 11

RESULTS: THE EFFECT ON THE LENGTH OF THE INTERVIEW

• Results show that the average duration of the sections comparable across waves stayed almost unchanged.

• Small increases of around 1 or 2 minutes in some sections when using the telephone.
DG ECONOMICS, STATISTICS AND RESEARCH-BANCO DE ESPAÑA 12

RESULTS: THE EFFECT ON THE INTERVIEW COMPLETENESS

• We observe a moderate decrease in the percentage of monetary questions answered exclusively with point values (0.7

p.p.), going from 91.4% in 2017 to 90.7% in 2020.

• We do not observe changes in the percentage of monetary questions answered (either through point values or intervals)

throughout the interview.

• Putting all together, the evidence suggests that although households were somewhat more likely to respond providing

ranges, item non-response did not increase overall for monetary questions within the interview.

Percentage of monetary questions 

answered with point value

Percentage of monetary questions 

answered with point value or ranges

EFF2020 -0.007*** 0.001

(0.002) (0.001)

Constant 0.914*** 0.962***

(0.002) (0.001)

Observations 12,736 12,736

R-squared 0.001 0.000

Standard errors in parentheses

*** p<0.01, ** p<0.05, * p<0.1

Table 4. Effect on item non-response indicators
DG ECONOMICS, STATISTICS AND RESEARCH-BANCO DE ESPAÑA 13

RESULTS: THE EFFECT ON ITEM NON-RESPONSE FOR MONETARY QUESTIONS

• For most of the questions we do not observe

substantial differences in the prevalence of

item non-response.

• But some degree of heterogeneity across

questions:

• Item non-response decreased for all

questions on real assets and also for

unlisted shares.

• Significant increases in item non-response

for questions on mutual funds and debt

interest rates.
DG ECONOMICS, STATISTICS AND RESEARCH-BANCO DE ESPAÑA 14

RESULTS: THE EFFECT ON RESPONDENTS’ ATTITUDES

• The percentage of respondents who were

“not suspicious at all” increased by 2.4 pp to

almost 82%.

• The distribution of the variable on “how the

respondent understood questions” remained

unchanged with a large proportion of

respondents among the highest values

(63.7%).

• The proportion of interviews with very

reliable answers increased by 2.6 pp to

60.6%.

• The proportion of respondents using

documents as consultation at some point

during the interview increased by 22 pp (to

37%).
DG ECONOMICS, STATISTICS AND RESEARCH-BANCO DE ESPAÑA 15

MAIN TAKEAWAYS

• In this paper, we estimate the overall impact of the change in the interviewing mode from CAPI to CATI on specific

data quality indicators by comparing the values of those indicators in the EFF2020 (CATI) sample with those in the

EFF2017 (CAPI) reweighted sample.

• Our results show that this change did not impact substantially on dimensions such as the length of the interviews,

the prevalence of item non-response, and respondents’ attitudes when completing the interview.

• This evidence is useful when exploring the possibility of implementing mixed modes in the data collection of

subsequent waves.

• However, our results could be driven by the very particular context of the COVID-19 crisis, when social mobility

was very restricted, people stayed at home for longer, had less social life, and were more willing to talk and to

participate in surveys.
DG ECONOMICS, STATISTICS AND RESEARCH – BANCO DE ESPAÑA

More information on the Spanish Survey of Household Finances at:

https://app.bde.es/efs_www/home?lang=ES

To contact us, you can write to:

laura.crespo@bde.es

16
DG ECONOMICS, STATISTICS AND RESEARCH – BANCO DE ESPAÑA

THANK YOU FOR YOUR ATTENTION