RTI International is a registered trademark and a trade name of Research Triangle Institute. www.rti.org

Financial record checking in surveys:

Do suggestive prompts 

improve data quality? 

Joe Murphy

RTI International

jmurphy@rti.org

Presented at BLS

July 12, 2016

1

Presentation based on:

Murphy, J. J., Rosen, J., Richards, A. K., Riley, S., Peytchev, A., & Lindblad, M. 

(2016). Financial record checking in surveys: Do suggestive prompts improve 

data quality? Field Methods, 28(3), 247–265.
 Self-reports of financial information in surveys, 

such as income and assets, are particularly 

prone to inaccuracy

– difficulty reporting income because they cannot recall 

the information being requested and use

– faulty estimation strategies (Moore et al., 1999)

 Retrieval mechanism, and ultimately data 

quality, may be improved by respondents 

accessing financial records in reporting income 

and assets

The problem with self-reports

2
 Ask respondents to have copies of their bank 

statements on hand for the interview for easy 

reference and retrieval of the correct 

information

– error mitigated by bypassing memory retrieval through 

the use of records

– better quality data can be collected with little risk to 

respondent cooperation (Moon and Laurie, 2010)

Prompting the respondent to access records

3
 Couper et al. 2013 found encouraging 

respondents to consult records in a web survey 

increased record checking significantly (from 

46% to 55%)

– but not sufficient to change estimate precision

– group asked to check records had a slightly lower 

response rate than those not asked to check records 

(77% vs. 80%)

Past research on prompting for record checking

4
 Annual (since 2003) longitudinal survey of low- to 

moderate-income (LMI) homeowners and renters

 Captures the experiences of households and 

assesses the pros and cons of homeownership for 

LMI households in the United States

 Not a general population survey but CAPS 

respondents reflect the LMI population with respect 

to income and race/ethnicity (Riley et al. 2009)

 Primary goal to measure wealth differences 

between homeowners and renters

CAPI / CATI Experiment: Community Advantage Panel Survey

5
 Wealth-and-assets data noisy because values 

change frequently and reporting requires 

recalling a number of different sources

 Multimode (CAPI / CATI) experiment in 2012 

asking random half to check financial records

 Prompt to consult financial records during 

survey (recent bank statements, mortgage 

statements, school and car loan statements, 

retirement accounts, and insurance policies)

6

CAPI / CATI Experiment: Community Advantage Panel Survey
1. Those asked to prepare records for the interview

will not respond at the same rate as those not 

asked to prepare records.

2. Among those who respond to the survey, those 

encouraged to check records will do so at a 

significantly higher rate than those not 

encouraged.

3. Those asked to check records will display fewer 

behaviors that might indicate suboptimal data 

quality (i.e., rounding).

Hypotheses

7
4. Asking respondents to check records will result in 

some significantly different survey estimates

compared with those not asked to check records, 

suggesting a potential improvement in accuracy 

by prompting respondents to check records.

5. Due to interviewer presence, CAPI respondents 

will be more likely than CATI respondents to 

check records when prompted and more likely to 

provide more accurate responses to financial 

questions.

Hypotheses

8
 Prompt group received lead letter asking them 

to prepare records in advance and suggestive 

prompts during the interview

Experimental Design

9

Mode Record

Checking 

Prompt

No 

Prompt

Total

CAPI 950 949 1,899

CATI 693 692 1,385

Total 1,643 1,641 3,284
 Compared response rates (AAPOR RR1) and rate 

of record checking by experimental group/mode

Unit Response and Compliance

10
Response Rate (%) % Checking Records

Unit Response and Compliance

11
 Is the act of checking records associated with 

less rounding?

 Wilcoxon-Mann-Whitney (WMW) test

– orders reported values by descending frequency, plots the 

cumulative distribution

– compares the area under each curve, and produces a Z 

score indicating whether the samples differ significantly 

(Hand 1997)

 Evidence that the act of checking records itself 

was associated with lower levels of rounding on 

several measures.

Precision of Reported Values

12
 More heaping of responses around $50K, 

$100K, etc. when records not checked

 But did not find similar results for comparison of 

control and experimental groups

Precision of Reported Values

13
 Is checking records associated with higher 

accuracy, regardless of experimental condition?

 Assume differences in mean values between 

those checking and not checking indicates 

improved accuracy with checking

 Since checking records was not experimentally 

assigned, it would be inappropriate to draw 

conclusions from these results, but...

Accuracy of Mean Reported Values

14
 Very little evidence of differences between 

those who were and were not prompted to 

check records.

Accuracy of Mean Reported Values

15
Conclusions

16
Those asked to prepare records for the interview will 

not respond at the same rate as those not asked to 

prepare records.

NO – Almost no difference in response rate by 

experimental treatment.

Hypothesis 1

17
Among those who respond to the survey, those 

encouraged to check records will do so at a 

significantly higher rate than those not 

encouraged.

YES – Overall, record checking was infrequent, but 

was more prevalent when the respondent was 

prompted, especially in CATI.

Hypothesis 2

18
Those asked to check records will display fewer 

behaviors that might indicate suboptimal data 

quality (i.e., rounding).

NO – Some evidence of less rounding when records 

actually checked but not enough between 

experimental groups to see a difference.

Hypothesis 3

19
Asking respondents to check records will result in 

some significantly different survey estimates

compared with those not asked to check records, 

suggesting a potential improvement in accuracy by 

prompting respondents to check records.

NO – Little evidence of differences in survey 

estimates between those asked to check and not.

Hypothesis 4

20
Due to interviewer presence, CAPI respondents will 

be more likely than CATI respondents to check 

records when prompted and more likely to provide 

more accurate responses to financial questions.

NO – The presence of the interviewer did not seem 

to impact likelihood to check records, BUT this 

assumes CATI respondents were always truthful 

about reporting use of records.

Hypothesis 5

21
 For CAPI and CATI surveys:

– Asking respondents to prepare and reference financial 

records during the interview will not reduce 

participation

– but it may only result in a modest increase in the rate 

of records checking

 General conclusion mirrors that of Couper et al. 

(2013) who examined this issue in a web survey 

setting

Takeaways

22
 Even when respondents check records, it is not 

clear that the data provided are more precise or 

accurate.

 Without a more directive intervention than the 

one we employed, suggestive prompts to check 

financial records will do no harm but may also 

do little good.

Takeaways

23
 Consider designs where survey estimates can 

be independently confirmed

 Research into the specific types of financial 

items where records could improve retrieval for 

improved precision and accuracy

 Cognitive or usability laboratory testing 

– ease of the task and confidence in the reported 

amounts

– record observations of respondents’ success and 

difficulties in retrieving and reporting the right 

information

Future directions

24
 Couper, M. P., M. B. Ofstedal, and S. Lee. 2013. Encouraging record use 

for financial asset questions in a web survey. Journal of Survey Statistics 

and Methodology 1:171–82.

 Hand, D. J. 1997. Construction and assessment of classification rules. 

New York: John Wiley & Sons.

 Moon, N., and H. Laurie. 2010. Is it worth the effort? Respondent record 

use on the British household panel study. Presented at the Annual Meeting 

of the American Association for Public Opinion Research, Chicago, IL, 

June 1, 2014.

 Moore, J. C., L. L. Stinson, and E. J. Welniak. 1999. Income reporting in 

surveys: Cognitive issues and measurement error. In Cognition and Survey 

Research, eds. M. G. Sirken, D. J. Herrmann, S. Schechter, N. Schwarz, J. 

M. Tanur, and R. Tourangeau, 155–74. New York: John Wiley & Sons.

 Riley, S., H. Ru, and R. Quercia. 2009. The community advantage program 

database: Overview and comparison with the current population survey. 

Cityscape: A Journal of Policy Development and Research 11:247–56.

References

25