U.S. Energy Information Administration Independent Statistics & Analysis www.eia.gov

Analyzing Survey Duration Time and Its 

Relationship to Data Quality for a Household Survey

Federal Committee on Statistical Methodology

October 27, 2022

By

S. Grace Deng, L. Kaili Diamond

Disclaimer: The analysis and conclusions contained in this presentation are those of the authors and do not represent 

the official position of the U.S. Energy Information Administration or the U.S. Department of Energy.
Introduction

• Residential Energy Consumption Survey (RECS)

– Collects energy-related characteristics and energy consumption billing data from the 

Household Survey and the Energy Supplier Survey

– 2020 RECS Household Survey was conducted entirely in self-administered web and mail 

modes: total of 18,496 responding households, 73% via web and 27% via paper

• Survey duration time

– The length of time it takes a respondent to complete the set of questions in the Household 

Survey questionnaire

– Both the total duration time in completing the entire survey and the duration time in 

completing each section were collected as paradata in the web instrument

– 11,328 web cases in this analysis

Grace Deng 

FCSM: Washington DC, Oct 2022 2

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Research questions

1. How is the survey duration time related to respondent demographics and 

housing characteristics? 

– e.g.: housing type, respondent age, respondent education

2. How is the survey duration time related to the data quality of survey 

responses?

– e.g.: percentage of missing questions, batch-edit changes

Grace Deng 

FCSM: Washington DC, Oct 2022 3

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
General linear model was used to check if demographic and housing 

characteristics are correlated with duration time 

Predictors in the model: 

– Housing type (3): mobile, single family (attached and detached), multifamily (apartments at 

least 2 units)

– Total household members (4): 1, 2, 3-4, >=5

– Respondent education (4): High school or below; some college; bachelor; master or above

– Respondent age (5): below 25; 25-34; 45-54; 55-64; 64+

– Respondent sex (2): male; female

– Employment (4): full-time; part-time; retired; not-employed

Grace Deng 

FCSM: Washington DC, Oct 2022 4

Dependent variable duration time was log transformed

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
5 Grace Deng 

FCSM: Washington DC, Oct 2022

Duration time was log-transformed due to its skewed distribution

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Respondent age had the largest effect on the average duration time, 

other effects were relatively small

Grace Deng 

FCSM: Washington DC, Oct 2022 6

Source

Degree 

of 

freedom

Sum of 

Square

Mean 

Square F Value Pr > F

respondent age 5 122.8 24.4 167.6 <.0001

respondent employment 3 10.3 3.4 23.6 <.0001

total household members 3 9.1 3.0 20.9 <.0001

housing type 2 4.1 2.1 14.2 <.0001

respondent education 3 3.8 1.3 8.8 <.0001

respondent sex 1 3.3 3.3 22.6 <.0001

housing type*education 6 2.0 0.3 2.3 0.0294

education*sex 3 1.8 0.6 4.1 0.0065

total member*sex 3 1.5 0.5 3.5 0.0148

If separated by housing type, respondent age still have the largest effect. 

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Overall, the older the age, the longer the duration time it took 

respondents to complete the survey

Grace Deng 

FCSM: Washington DC, Oct 2022 7

24 26

29

31

34

38

0

5

10

15

20

25

30

35

40

45

<25 35-34 35-44 45-54 55-64 >=65

duration time (mins)

age group

Average duration time by age group

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
The age impact had the same systematic trends across all sections

Grace Deng 

FCSM: Washington DC, Oct 2022 8

* Color scheme here is only for section readability

0

5

10

15

20

25

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

all a b c d e f g h i j k

average duration time per question (secs)

section and age group

Age impact on section average duration time (final section not included)

geomean 95%LCL 95%UCL

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
The average time on the supplier information section took 

significantly longer than the rest of the sections

Grace Deng 

FCSM: Washington DC, Oct 2022 9

0

20

40

60

80

100

120

140

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

<25

25-3435-4445-5455-64

>=65

average duration 

all a b c d e f g h i j k l

time per question (secs)

section and age group

Age impact on section average duration time

geomean 95%LCL 95%UCL

* Color scheme here is only for section readability

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Mobile phone users took the shortest average time in completing 

the survey, and device type was highly correlated with age

Grace Deng 

FCSM: Washington DC, Oct 2022 10

28

33 36

0

5

10

15

20

25

30

35

40

Mobile PC Tablet

average duration time (mins)

Average duration time by device type

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Evaluating data quality by three “speed groups”

Grace Deng 

FCSM: Washington DC, Oct 2022 11

Defining baseline group, fast group, and slow group for duration time 

• Fast group: the lower 5th percentile from each housing type category

• Baseline group: between the 2nd and 3rd quantiles from each housing type category

• Slow group: the upper 5th percentile from housing type category

Metrics for data quality evaluation

• Percentage of missing questions

• Percentage of missing energy supplier information

• Percentage of variables with explicit don’t know responses

• Batch-edit changes (automatic corrections to variables)

• Analyst-edit changes (manual corrections to variables)

Statistical analysis

• Generalized linear models/logistic regression models

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Missing questions

Grace Deng 

FCSM: Washington DC, Oct 2022 12

A RECS survey question can include multiple response variables, and the 

number of eligible questions vary among respondents due to skip patterns. 

Missing questions: all of the response variables in a question are missing

• Skipped questions were not counted 

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work

Section

# Total survey 

questions

Total survey 

variables

a - Your Home 39 62

b - Appliance 45 55

c - Electronics 26 59

d - Space heating 21 31

e - Air Conditioning 15 21

f - Thermostats and Temperatures 5 11

g - Water Heating 8 11

h - Lighting 7 11

i - Energy Bills 25 49

j - Household Characteristics 10 18

k - Energy Assistance 14 25

l - Final Questions 4 20

219 373

Example: one question with six response 

variables

In your home, which of the following types of 

computer equipment are used for teleworking or 

working from home? 

Desktop computer? [TLDESKTOP] (Yes/No)

Laptop computer? [TLLAPTOP] (Yes/No)

Tablet? [TLTABLET] (Yes/No)

External monitor? [TLMONITOR] (Yes/No)

Other [TLOTHER, TLOTHER_other] (Yes/No)
Overall, the fast group had significantly higher average percentage of missing 

questions than those of the baseline and slow groups (~2%)

Grace Deng 

FCSM: Washington DC, Oct 2022 13

2.7%

1.9%

1.8%

0.0%

0.5%

1.0%

1.5%

2.0%

2.5%

3.0%

fast group baseline slow group

% of missing questions

Percentage of missing questions by speed group 

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Within housing type, the fast groups in both SF and MF also had higher 

percentage of missing questions

Grace Deng 

FCSM: Washington DC, Oct 2022 14

2.4%

3.6%

1.7%

2.8%

1.6%

2.8%

0.0%

0.5%

1.0%

1.5%

2.0%

2.5%

3.0%

3.5%

4.0%

Single Family Multi Family

% of missing questions

Percentage of missing questions by duration time group and housing type

fast group baseline slow group

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Missing energy supplier information

Grace Deng 

FCSM: Washington DC, Oct 2022 15

Survey asks for supplier information on electricity, natural gas, fuel oil, 

and propane in separate questions, if a respondent provided at least one 

fuel supplier, then it is considered a response. 

Missing energy supplier information: no supplier information was 

provided for any of the fuel questions

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
The fast group had higher percentage of missing energy supplier information 

Grace Deng 

FCSM: Washington DC, Oct 2022 16

39.8%

17.0%

12.0%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

50.0%

fast group baseline slow group

% of missing supplier information

Percentage of missing supplier information by speed group

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Within housing type, the fast groups in both SF and MF had higher percentage 

of missing energy supplier information

Grace Deng 

FCSM: Washington DC, Oct 2022 17

38.1% 44.4%

15.2%

23.9%

10.0%

20.2%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

Single Family Multi Family

% failed at least one batch edit

Percentage of missing supplier information by speed group and housing type

fast group baseline slow group

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Answered explicit “don’t know” 

Grace Deng 

FCSM: Washington DC, Oct 2022 18

Some questions offered explicit “don’t know” responses. There is a total of 49 

response variables (skips were not counted)

Example:

About how many windows does your home have? 

1 1 or 2 windows

2 3 to 5 windows

3 6 to 9 windows

4 10 to 15 windows

5 16 to 19 windows

6 20 to 29 windows

7 30 or more windows

-4 Don’t know

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
The fast group had higher percentage of answering explicit don’t know 

Grace Deng 

FCSM: Washington DC, Oct 2022 19

17.0%

8.8%

8.4%

0.0%

5.0%

10.0%

15.0%

20.0%

fast group baseline slow group

% answered explicit don't know

Percentage of answering explicit don't know by speed group 

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Within housing type, the fast groups in both SF and MF had higher percentage 

of answering explicit don’t know

Grace Deng 

FCSM: Washington DC, Oct 2022 20

12.8%

30.0%

6.2%

19.7%

6.3%

17.9%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

Single Family Multi Family

% answered explicit don't know

Percentage of answering explicit don't know by speed group and housing type

fast group baseline slow group

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Batch-edit changes

Grace Deng 

FCSM: Washington DC, Oct 2022 21

These are automatic corrections to response variables, includes:

• Setting illogical outlier values to missing (e.g., TV is used more than 24 hours 

a day)

• Setting inconsistent responses to missing (e.g., reporting heating equipment 

as portal electric heater but saying the fuel is natural gas)

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work

Analyst-edit changes

These are cases required analyst reviews and making manual corrections
For batch-edit changes, the fast group was not statistically different from the 

baseline group, but the slow group was higher compared to that of baseline 

Grace Deng 

FCSM: Washington DC, Oct 2022 22

51.0%

46.2%

54.6%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

fast group baseline slow group

% failed at least one batch edit

Percentage of failing at least one batch edit by speed group

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Within housing type, the trend in batch-edit changes were the 

same as the overall

Grace Deng 

FCSM: Washington DC, Oct 2022 23

47.6%

57.9%

44.7%

51.8% 52.1%

63.3%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

Single Family Multi Family

% failed at least one batch edit

Failed at least one batch-edit percent by duration time group and housing type

fast group baseline slow group

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Similarly for analyst-edit changes, the fast group was not statistically 

different from the baseline group; but the slow group was higher

Grace Deng 

FCSM: Washington DC, Oct 2022 24

7.0%

7.0%

10.3%

0.0%

3.0%

6.0%

9.0%

12.0%

15.0%

fast group baseline slow group

% failed at least one analyst edit

Percentage of failing at least one analyst edit by speed group 

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Within housing type, no statistical differences were found in analyst-edit 

changes among speed groups

Grace Deng 

FCSM: Washington DC, Oct 2022 25

5.6%

9.5%

6.8% 7.5% 9.3%

13.8%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

Single Family Multi Family

% failed at least one batch edit

Percentage of failing at least one analyst edit by speed group and 

housing type

fast group baseline slow group

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Grace Deng 

FCSM: Washington DC, Oct 2022 26

Conclusion: 

 Among different characteristics, respondent age had the largest impact on 

duration time. The other characteristics effects were relatively small. Device 

type was highly correlated with age, and mobile phone users took shorter 

time than those of the PC or Tablet users. 

 Duration time could also be affected by the nature of the survey questions 

(e.g., reporting supplier information would require longer time).

 The fast group that took shorter time were more likely to leave the questions 

blank and choose explicit don’t know than the baseline or slow groups. 

However, in terms of batch-edit or analyst-edit changes, comparing to the 

baseline, this group did not have more edit changes, but the slow group had 

significant higher edit changes. These would be due to the fast group are 

more likely not to respond when they don’t know the answer.

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Grace Deng 

FCSM: Washington DC, Oct 2022 27

Future work:

• Additional editing from reconciliation between household characteristics and 

energy bills are still ongoing, we are planning to include these edits for future 

analysis.

Introduction Research questions Relationship with demographic & housing characteristics Relationship with data quality Conclusion Future work
Thank You! 

• Grace Deng|Shaofen.deng@eia.gov

• Kaili Diamond|Kaili.diamond@eia.gov

Grace Deng 

FCSM: Washington DC, Oct 2022 28
BACKUP SLIDES

Grace Deng 

FCSM: Washington DC, Oct 2022 29
Grace Deng 

FCSM: Washington DC, Oct 2022 30

Speed group time definition and sample size

Speed group Mobile SF MF Total

Fast group 23 378 126 527

Baseline 238 4372 1099 5709

Slow group 24 451 109 584

Speed group Mobile SF MF

Fast group <=16mins <=17mins <=15mins

Baseline ≥23 & ≤40mins ≥25 & ≤43mins ≥21 & ≤38mins

Slow group ≥64mins ≥65mins ≥64mins