Content does not represent BLS policy

Respondent Burden: 

Lessons Learned from a MixedMethods Approach to Assessing 

Measures of Perceived Burden 

Morgan Earp

Scott Fricker

John Dixon
Multi-Method 

Assessment Overview

 Reliability (Internal Consistency)

 Distinguishability of Item Sub Scales

 Dimensionality 

Do the items measure the latent construct well?

Do they measure more than the latent 

construct?

 Linearity

Are the item sub scales linearly related to the 

latent variable?

2
Overview

 Potential sources of measurement error

 Item assessment tools

 Case study 

Internal consistency 

Dimensionality 

Item Sub Scale Performance

– Distinguishability

– Linearity Summary

 Conclusions

 Next steps

3
Research Questions

 1) Are responses to the items used in the 

measure internally consistent? 

4
Research Questions

 2) If not, why do they vary? 

A) Are item relationships higher or lower than 

we would expect? 

5
Research Questions

 3) How are the subscales of my measure 

performing? 

A) Do the different response options 

distinguish between varying levels of latent 

burden? 

B) Are the ordinal response options linearly 

related to each other and latent burden as 

whole? 6
Models & Statistics

 Confirmatory Factor Analysis

Error Variance (internal consistency)

 Item Response Theory

Chi Square (internal consistency)

LD Chi Square (dimensionality)

Trace-lines (distinguishability of sub-scales)

Mean Square Error (linearity of sub-scales)

7
Assessed 9 Burden Items

 Burden 

 Length 

 Interest 

 Difficulty 

 Sensitivity 

 Appropriate # of 

Waves 

 Another Survey 

 Extend Survey 

 Effort 
Background

 Current Expenditure Survey (CEQ)

respondents were asked about burden in a post 

survey (end of 5th interview)

 Burden items were developed using Bradburn’s

(1978) definition:

effort

survey length 

frequency of survey requests
Research Question 1

 1) Are responses to the items used in the 

measure internally consistent? 

10
Internal Consistency

 Confirmatory Factor Analysis, Error 

Variance: 1 – r 2

Measures the % of variation in the latent 

construct not explained by the item

 Item Response Theory – Chi Square 

Statistic

Is there a significant proportion of variation in 

the latent construct not explained by this item?

11
Internal Consistency

CFA – Error Variance IRT – Chi Square

12

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Burden

Length

Sensitivity

Another Survey

Waves

Extend

Difficulty

Interest

Effort

Error Variance

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

Burden

Length

Sensitivity

Another Survey

Waves

Extend

Difficulty

Interest

Effort

IRT Chi-Square / df
Item Response Theory -

Chi Square

 Χ 2 also considers 

how difficult an item is to agree w/

– Some items maybe carry more weight...

Item response patterns vs total score

local independence of the item 

– Is it as correlated as we would expect w/ the other 

items?

– Is it more correlated w/ the other items than we 

would expect after controlling for the latent variable 

burden?

13
Research Question 2

 2) If item responses are not internally 

consistent, why might they vary? 

A) Are item relationships higher or lower than 

we would expect? 

B) Are there interaction effects aka differential 

item functioning?

14
Local Independence

Dimensionality

 LD χ2 ≤ -10

 not as correlated as we would expect w/ another 

item

 LD χ2 ≥ 10 

 more correlated than we would expect w/ another 

item after controlling for the latent burden

 measures more than just latent burden 15
IRT LD χ2 Statistics 

by Burden Items

Item Burden Another Extend Length Interest Sensitivity Wave Difficulty

Burden

Another -7.7

Extend -10.1 -15.0

Length -8.6 7.6 5.0

Interest -17.5 -5.4 -7.0 -0.9

Sensitivity 14.9 7.8 -9.5 -4.2 -17.2

Wave -3.4 -0.7 -0.0 0.2 0.1 -8.6

Difficulty 26.6 12.3 -11.9 1.4 -13.8 18.3 -8.8

Effort 20.0 7.6 -12.9 1.9 -28.7 8.1 -0.1 13.1
Research Question 3

 3) How are the subscales of my measure 

performing? 

A) Do the different response options 

distinguish between varying levels of latent 

burden? 

17
Tracelines

Good Item Fit Example

18

How burdensome was this survey to you? 

1 Not at all burdensome

2 A little burdensome

3 Somewhat burdensome

4 Very burdensome

Probability of Response Type given Theta level

Theta = Standardized Scale of the Latent Variable Burden
Tracelines

Good Item Fit Example

19

How burdensome was this survey to you? 

1 Not at all burdensome

2 A little burdensome

3 Somewhat burdensome

4 Very burdensome

Probability of Response Type given Theta level

Theta = Standardized Scale of the Latent Variable Burden
Tracelines

Poor Item Fit Example

Theta = Standardized Scale of the Latent Variable Burden 20

If we had to extend this survey for another 15 minutes, how 

willing would you have been to continue this interview?

(bext = extend survey)

1 Very willing 

2 Somewhat willing 

3 Somewhat unwilling 

is not associated w/unique levels of burden

4 Very unwilling 

Probability of Response Type given Theta level
Tracelines

Poor Item Fit Example

Theta = Standardized Scale of the Latent Variable Burden 21

If we had to extend this survey for another 15 minutes, how 

willing would you have been to continue this interview?

(bext = extend survey)

1 Very willing 

2 Somewhat willing 

3 Somewhat unwilling 

is not associated w/unique levels of burden

4 Very unwilling 

Probability of Response Type given Theta level
Research Question 3

 3) How are the subscales of my measure 

performing? 

A) Do the different response options 

distinguish between varying levels of latent 

burden? 

B) Are the ordinal response options linearly 

related to each other and latent burden as 

whole? 22
IRT - Linearity

 Linearity (Mean Square Error: t )

 Like CFA and the IRT Chi-Square statistic, the IRT t

statistic also looks at the consistency of item responses, 

but is more sensitive to violations of linearity

 If items responses violate the assumptions of linearity 

(they do not have a linear relationship w/ the latent 

variable), they will have significant misfit

23
t Statistics by 

Burden Items

24

0

2

4

6

8

10

12

14

16

18

t Statistic (Absolute Value)

Burden Items
t Statistics by 

Burden Items

25

0

2

4

6

8

10

12

14

16

18

t Statistic (Absolute Value)

Burden Items

Significant Misfit
Item Response Curves 

by Reported Burden

Good Fitting Item

26
Item Response Curves 

by Reported Burden

Poor Fitting Item

27
Conclusions

 While psychometric assessment tools provide 

insight about how our items perform - There is no 

single cohesive tool – it takes multiple methods

 All methods provide different details (even w/in 

IRT) – item may perform well in one aspect, but 

not another

 The various methods can be used to better 

understand and potentially improve our items

28
Our Next Steps

 Exploring how to best improve item subscales to improve utility and linearity across 

all response options

 Examining our definition of burden and 

whether we should include more items to 

measure a multidimensional construct or 

reduce items to assess a single dimensional 

construct

29
Contact Information

Contact Information

Morgan Earp

Earp.Morgan@bls.gov