1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Leveraging Paradata to Assess 

Respondent Behavior and Data Quality in 

the CE Online Diary Survey

Graham Jones & Parvati Krishnamurty, BLS

FedCASIC 2023

April 12, 2023
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Agenda

 What is Paradata?

 Background of CE Online Diary

 Use in measuring activity

 Analysis Results:

• Diary Use and Logins

• Devices and Operating systems

• Time spent in the Diary

• Relation to CE Data Quality

 Next Steps for CE Paradata use
3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

What are Paradata?

 Data generated as a by-product of the survey’s a collection process

 Analysis of paradata can lead to...

Improved data collection efficiency

Reduced survey fielding costs

Better understanding of measurement error
4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Online Diary Background

Development of CE Online 

Diary Instrument

Large Scale Feasibility Test of 

Online Diaries (LSF) 10/2019 

– 4/2020

CE Online Diary 

Supplemental Use

6/2020

CE Online Diary in 

Production

7/2022
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Online Diary Paradata

 Events and user actions recorded along with event details in tabular form.

Each event (action) is recorded as its own row in the data

Each event (action) is time-stamped

All rows have a value describing the event “type”

 Event Types:

• Login

• Hyperlink (click)

• Failed Login

• Next Action

• Logout

• Previous Action

• Entry

• Field Change

• Exit

• Error Trigger
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Online Diary Paradata
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Paradata Analysis Results

1. Diary Use and Logins

2. Devices and Operating systems

3. Time spent in the Diary

4. Relation to CE Data Quality
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Online Diary Use

 1,877 unique Consumer Units (CUs) with online diary credentials

1,801 successfully logged in

76 never successfully logged in

CUs with 

Online Diary 

Credentials 

(n=1,877)

CUs that 

Successfully 

Logged In 

(n=1,801)

CUs that 

Never 

Logged In 

(n=76)
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Login Activity

CUs with successful logins 

(n=1,801)

•Accounted for 22,100 logins in 2021

•Had a mean login total of 12.3

•Had a median login total of 10

CUs with at least one 

login failure (n=583)

•Accounted for 3,160 login failures in 

2021

•Had a mean login failure total of 5.4

•Had a median login failure total of 4
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Percentage of Logins by Diary Day in 2021

10.4

7.8

7.3 6.4

5.9 5.4 5.2 5.1 5.8

4.8 4.4 4.7 4.8 5.4

7.1 6.8

2.7

22.4

16.3

8.9

7.1

4.9 3.8 3.0 4.0 4.5

2.0 1.2 2.3 2.4 3.9 4.1

5.2

4.2

0

5

10

15

20

25

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Percentage of Total Logins

CE Diary Day

Logins (n=20,670) Failed Logins (n=2,830)
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Paradata Analysis Results

1. Diary Use and Logins

2. Devices and Operating systems

3. Time spent in the Diary

4. Relation to CE Data Quality
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Paradata Analysis: Device Use in 2021

Desktop Only User

• All actions recorded in the paradata associated with a desktop

• Accounted for 1,162 or 64.5% of CUs

Mobile Only User

• All actions recorded in the paradata associated with a mobile device

• Accounted for 387 or 21.5% of CUs

Mixed Device User

• Actions recorded in the paradata split between desktop and mobile devices. 

• Accounted for 252 or 14.0% of CUs
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Login Activity Comparison by Device Use

12.3

10.7

14.6

10

6

11

0

2

4

6

8

10

12

14

16

Desktop Only (n=1,162) Mobile Only (n=387) Mixed Use (n=252)

Average Total Logins per User

Mean Logins Median Logins
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Paradata Analysis Results

1. Diary Use and Logins

2. Devices and Operating systems

3. Time spent in the Diary

4. Relation to CE Data Quality
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Paradata Analysis: Time Spent in the Diary

 Time Spent in the Diary Per User =

∑tt=1

nn EEEEEEEEEEtt−EEEEEEEEEEtt+1

60

 Action/Event Types:

• Login

• Hyperlink (click)

• Failed Login

• Next Action

• Logout

• Previous Action

• Entry

• Field Change

• Exit

• Error Trigger
16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Paradata Analysis: Time Spent in the Diary

 1,606 CUs had sufficient diary activity for the total time 

calculation in 2021.

Mean of 33.5 minutes

Median of 24.2 minutes
17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Time Spent in the Diary by Device Type

Desktop Only User

• Mean of 36.7 minutes

• Median of 27.5 minutes

Mobile Only User

• Mean of 19.7 minutes

• Median of 10.3 minutes

Mixed Device User

• Mean of 35.7 minutes

• Median of 26.5 minutes
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Median Time Spent in Diary per Diary Day

1.2

3.6

4.2 4.8 4.9 5.2

6.3

6.9 6.5 6.3

7.5

6.7

7.7

9.6 9.0

8.3

9.0

0.0

2.0

4.0

6.0

8.0

10.0

12.0

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Time Spent in Diary in Minutes

Diary Day

2021 CE Production (N=1,606)
19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Paradata Analysis Results

1. Diary Use and Logins

2. Devices and Operating systems

3. Time spent in the Diary

4. Relation to CE Data Quality
20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Login Activity Relation to CE Data Quality

$400.05

28.5

$697.06

37

0 100 200 300 400 500 600 700 800

Median Total Expenditures (in dollars)

Median Expenditure Counts

Logins ≥ 10.5 (n=734) Logins < 10.5 (n=756)
21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Device Use Relation to CE Data Quality

$555.31

34

$559

33

$400.50

27.5

0 100 200 300 400 500 600

Median Total Expenditure (in dollars)

Median Expenditure Count

Mobile Only (n=266) Mixed Device (n=222) Desktop Only (n=1,002)
22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Time Spent in Diary Relation to CE Data Quality

$365.14

24

$838.81

44

0 100 200 300 400 500 600 700 800 900

Median Total Expenditure (in dollars)

Median Expenditure Count

Time greater than or equal to 27 (n=706) Time less than 27 (n=708)
23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

End of Paradata Analysis Results
24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Paradata Analysis Conclusions

 Granular detail of paradata allows researchers to look at diary 

keeping behavior from new perspective.

 Greater time spent in the diary and login activity were 

associated with higher quality data.

 Respondents using non-mobile devices outperformed those only 

using a mobile device.
25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Paradata Analysis Next Steps

 Incorporate average total logins and median total time spent in 

the diary into the CE Data Quality Profile as standard metrics.

 Continue to tweak the calculation of ‘time spent in diary’ for the 

most precise estimate.

 Revaluating values and naming convention for the variables in 

the paradata.

 Encouraging respondents to use non-mobile devices
26 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Contact Information

Gray Jones

Economist

Consumer Expenditure Surveys

jones.graham@bls.gov