Talking Points: Michael Schober, October 16, 2012

Workshop Discussion of NRC Report.

SUPPORTED JOURNAL: a new mode. Not implemented in the same way in the three 

designs, and we as a panel were not able to fully hash out exactly what each design 

means by this. So let me give my take, and others can chime in, dispute, etc.

NOT simply a diary, and not simply self administered. Not quite an interview either. 

Promoting: Use of records. Done at the R's convenience!!! Potentially asynchronous--

reduce time pressure so that we aren't sending the signal that estimation is desirable. 

Extra motivation, staff support.

All the motivational and cognitive issues raised this morning are serious, and need to be 

addressed. We believe they can't be addressed through minor revisions of current 

modes; it's not clear that we know that errors of reporting in current modes aren't 

systematic in a pernicious pattern. 

The notion of a supported journal seems to us a reasonable compromise between the 

various legitimate needs at a price point that is plausible: large enough sample, more 

intensive data collection that supports record checking and responding in the ways and 

when convenient to the respondent but that doesn't require a too-expensive stable of 

interviewers or sample, potential for real-time data entry and monitoring and subsequent 

intervention, etc.

Premise: in order to get the higher quality record checking interviews we are looking 

for, there will need to be sufficient incentive for Rs to really be willing, to take the time, 

and feel this was worth it. The aim is to create a quite different experience than 

respondents now have.

I propose getting this right will require the insights of experience designers (not just HCI, 

human factors, or app programmers): what makes people want to, be willing, feel safe 

and non-suspicious, to provide their data. 

A point of clarification on terminology:

The issue of recall is complex and we are not all using the term to mean the same thing. 

Need to make further distinctions. Supported journal allows for lots of different options:

Different dimensions/options for interviewing that can be deployed/implemented 

independently, and that can be mixed and matched in a supported journal:

• Retrospective vs concurrent reporting: this mode can be used for both.

• Use of records vs. memory. (Different from estimation vs. precise recall)

• Time pressure: synchronous vs. at R's pace. 

• Self-administered vs. interviewer-administered -- we are blurring this distinction here.

Interviewer support could be face to face, via voice, via text, via automated help...
Different combinations of these options will have different cost structures, and 

presumably different effects on the respondent's experience, and ultimately the data 

quality.

We do not have good evidence on this for tablets. Initial evidence supports the 

plausibility that asynchronous reporting of numerical values on a mobile device via text 

(Schober, Conrad et al. 2012) can lead to less rounding/estimation and greater 

disclosure of sensitive behavioral information than synchronous reporting via voice, 

whether to an interviewer or automated interviewing system. But obviously this is not 

evidence on tablets with consumer expenditure questions in a CE sample.... 

Responsive monitoring: to set this up right will require thinking from a respondentcentered perspective! 

--We need not simply a finger-wagging compliance center, but a truly supportive staff 

on every front--both responding to what R needs AND "pushing"--contacting R, 

motivating R, coming in person, etc. Think of Apple tech support--easily scheduling a 

call for help.

Designing this right will require serious rethinking of how we are conceiving of interface 

or app design, to include the entire process and experience, facilitating getting records, 

making the respondent experience seamless and intuitive, based on how respondents 

(rather than researchers) are thinking of their expenditures (natural units). This also 

allows customization and alternate pathways for different respondents. 

Staffing: Needs a new kind of field representative/responsive monitoring staff, whose 

job is to train Rs, collaboratively, on what we are asking of them, and on what kind of 

accuracy we need.