U.S. Consumer Expenditure Survey (CE) 

CE Online Diary Usability Testing – Task Order #26 

Final Report

Authors 

Douglas Williams

Hanyu Sun

July 17, 2017 FINAL 

Prepared for:

U.S. Department of Labor

Bureau of Labor Statistics

Washington, DC 20212-0022

Prepared by:

Westat

An Employee-Owned Research Corporation®

1600 Research Boulevard

Rockville, Maryland 20850-3129

(301) 251-1500
This page is left intentionally blank.
CE Online Diary Usability Testing – Task Order 

#26 

iii 

 

Table of Contents 

Chapter Page

1 Executive Summary............................................................................................ 1-1

2 Introduction ........................................................................................................ 2-1

3 Methods ............................................................................................................... 3-1

4 Findings................................................................................................................ 4-1

5 Recommendations.............................................................................................. 5-1

Appendixes

A Usability Protocol (Placement)......................................................................... A-1

B User Guide........................................................................................................... B-1

C Usability Protocol (Pick-up).............................................................................. C-1

Tables

3-1 Distribution of demographic compositions for test participants 

(main diarist)........................................................................................................ 3-2

3-2 Distribution of household compositions........................................................ 3-2

4-1 Distribution of device type owned and whether an online shortcut was added (main diarist).............................................................................. 4-1

4-2 Distribution of device type owned and whether an online shortcut was added (other household members).................................................... 4-2

4-3 Reasons for other household member non-participation by 

relationship to main diarist................................................................................ 4-4

4-4 Distribution of difference between expense date and entry date 

for all expenses.................................................................................................... 4-5

4-5 Respondent distribution of failed login attempts to the online 

diary by password type and across all attempts.............................................. 4-6
CE Online Diary Usability Testing – Task Order 

#26 

iv 

 

Contents (continued) 

Tables Page

4-6 Device characteristic detected when accessing the online CE 

Diary ..................................................................................................................... 4-10

4-7 Device types used to enter expenses by respondent type............................. 4-10

4-8 Reasons provided for not entering expenses while out 

(smartphone users only)..................................................................................... 4-11

4-9 Diary session duration by logout type (respondent initiated, or 

system timeout)................................................................................................... 4-12

4-10 Diary session duration by device type (respondent initiated 

logouts)................................................................................................................. 4-12

4-11 Distribution of expense description issues as a percentage of all 

expenses............................................................................................................... 4-15

4-12 Percentage of expenses edited within each category as a 

proportion of number enetered for each category type................................ 4-16

4-13 Distribution of reported difficulties by type................................................... 4-21

4-14 Ease of use rating for desktop and mobile devices....................................... 4-22

4-15 List of reasons given for not using a mobile device to access the 

online CE Diary .................................................................................................. 4-24

4-16 List of reasons given for not using informational help links ....................... 4-25

4-17 List of reasons given for how summary list was helpful or used ................ 4-26

4-18 Distribution of online diary navigation ratings .............................................. 4-27
CE Online Diary Usability Testing – Task Order 

#26 

v 

 

Contents (continued) 

Figures Page

2-1 Screen capture – desktop platform.................................................................. 2-3

2-2 Screen capture – mobile platform.................................................................... 2-5

2-3 Screen capture – active help (mobile).............................................................. 2-6

3-1 Screen capture – passive consent (adult)......................................................... 3-3

4-1 Distribution of initial diary access by respondent type and 

reporting day........................................................................................................ 4-3

4-2 Distribution of initial expense report by respondent type and 

reporting day........................................................................................................ 4-5

4-3 Mean number of expenses entered by reporting day for main 

diarist .................................................................................................................... 4-8

4-4 Mean number of expenses entered by reporting day for other 

household members........................................................................................... 4-8

4-5 Main factors online instruments must take into consideration for 

function and design ............................................................................................ 4-9

4-6 Distribution of expenses by category type (across all device types)............ 4-13

4-7 Distribution of expenses by category within each device type 

(desktop or mobile) ............................................................................................ 4-14

4-8 Screen capture – pop-up no expenses............................................................. 4-18

5-1 Screen capture – example of free-to-search barcode scan for 

automotive part................................................................................................... 5-8

5-2 Screen capture – example of free-to-search barcode scan for bigbox store bulk chocolate bars........................................................................... 5-9
CE Online Diary Usability Testing – Task Order 

#26 

1-1 

 

Westat worked with BLS to develop a one-week diary that is accessible by individual household 

members. The diary was designed to adapt to various screen sizes based on desktop computers, 

laptop computers, and various mobile devices (e.g., smartphones and tablets). The desktop design 

developed as a part of Task Order 24 was the basic framework for the online diary. The online diary 

platforms were finalized in March of 2017 with approval from BLS.

The goals of this task were to complete 60 in-person usability tests, which included a placement 

interview, seven-day respondent reporting period, and a pick-up interview. Interviews began on 

March 21st, 2017, and concluded May 25th, 2017. A total of 62 placement interviews and 61 pick-up 

interviews were completed. For recruited households, all household members age 15 and older were 

asked to complete a one-week diary. For each household, a main diarist was identified who was the 

main contact for placement and pick-up interviews. The main diarist was responsible for 

encouraging the participation of other household members.

Findings in this report are based on two general sources. This includes respondent behaviors as 

determined through a review of data collected by the online diary and qualitative data collected 

during a respondent debriefing conducted during the pick-up interview.

Overall, compliance with the diary task was high, but more so for the main diarist than for other 

household members. About one-third of other household members did not log into the diary or 

enter any expenses. Accessing the diary through logging-in and creating a password was highly 

predictive of using the diary and entering expenses. Failed login attempts were experienced by nearly

one-half (47%) of online diary respondents, with a small proportion (17%) experiencing seven or 

more failed attempts. Failed logins largely occurred during initial diary access when users created 

their password.

Interviewer reminders were important for getting respondents to enter expenses. Reminder contacts 

were made by telephone to households where no expenses were entered by the fourth reporting day. 

The diary data show increased expense entries on the day of reminder contacts and the days leading 

up to the pick-up interview.

Executive Summary 1 
CE Online Diary Usability Testing – Task Order 

#26 

1-2 

 

Respondents used a variety of online accessible devices to access the online diary. Desktop devices 

were used more frequently, but mobile devices demonstrated more timely entry of expenses with 

more expenses entered the day the expense occurred. The type of expense entered varied by device 

type. Food for home consumption was more often entered using desktop devices, while food away 

from home and transportation expenses were proportionally higher for mobile devices. This 

suggests expense (receipt) complexity may influence which device is used. Session duration when 

using the diary was longer for desktop devices than mobile devices. This supports the hypothesis 

that more complex entry sessions are completed with desktop devices.

Respondents overwhelmingly reported positive experiences using the diary and nearly all 

respondents rated the diary as easy to use. Nonetheless, respondents still reported a number of 

difficulties. The most prevalent difficulty was with the username and passwords – difficulty setting 

the password due to requirements and difficulty remembering the password. Difficulty with recalling 

login credentials may have affected the use of mobile devices. Respondents who did not use mobile 

devices for entering expenses reported this was due to perceptions that the mobile version would be 

difficult to use (e.g., due to screen size). Other difficulties reported were generally associated with 

the mobile presentation. Respondents were critical of the lack of feedback on the mobile device as it 

was not obvious that expenses were saved. Respondents also expressed a preference for more detail 

in the expense summary and the addition of future technological functions, such as barcode 

scanning or receipt scanning.

Overall, the online diary performed well and was highly rated by respondents. Westat makes several 

recommendations for improving the online diary.

1. Investigate new approaches to obtaining participation from other household members. 

Some suggestions include: task commitment from the main diarist; collecting contact 

information for other household members from the main diarist for interviewer contact; and 

incentivizing main diarist recruitment efforts.

2. Incorporate an interactive demo with mobile devices during the placement interview to 

address perceptions the mobile version may be too difficult.

3. Simplify login procedures and password requirements.

4. Add content to the expense summary and functionality to produce individual level expense 

statistics.
CE Online Diary Usability Testing – Task Order 

#26 

1-2 

 

5. Add feedback for actions taken in the mobile version to confirm expenses were entered.

6. Research the use of future technologies, such as barcode or receipt scanning.
CE Online Diary Usability Testing – Task Order 

#26 

2-1 

 

The Consumer Expenditure Survey (CE) is sponsored by the Bureau of Labor Statistics (BLS) and 

consists of two distinct data collection components: the Consumer Expenditure Quarterly Interview 

(CEQ) Survey and the Consumer Expenditure Diary (CED) Survey. The CEQ is administered by an 

interviewer and captures major or recurring expenditures, while the CED is self-administered using a 

household respondent and captures minor items and those that are more frequent. For the CED, a 

household respondent (main diary keeper) is asked to complete two one-week diaries to record 

expenditures for household members. Under the current design, the CED uses paper-and-pencil 

diary instruments (PAPI). There are a number of drawbacks to this approach. The use of PAPI 

requires main diary keepers to carry the diary with them. Respondents who do not do this may 

forget to record expenses and would need to keep and organize receipts. The use of PAPI with one 

household respondent may also lead to underreporting of expenditures from other members of the 

household.

The dramatic increase in online technologies and ownership of internet accessible devices, including 

mobile devices make online data collection an attractive alternative to PAPI data collection. Online 

data collection using desktop computers, laptop computers, or mobile devices such as smartphones 

and tablets has the potential to enhance data collection within the CED. A move to online data 

collection is also consistent with the goals of the CE’s Gemini Project redesign that introduces 

individual, online diaries. One goal of this new design is to leverage data collection technologies and 

present sampled households with more timely data entry options, such as online options accessible 

across a variety of devices with optimization for smartphones and tablets.

Online diaries have the potential to improve unit and item response rates as well as measurement 

error. Additionally, an online component and a mobile-optimized option have potential cost savings 

over PAPI reducing hard-copy materials, scanning, and data entry costs. In addition, online diaries 

can incorporate individual diaries for all members of a household, which may reduce the need for 

proxy reporting in the CE.

Introduction 2 
CE Online Diary Usability Testing – Task Order 

#26 

2-2 

 

BLS has conducted a number of tests exploring both online and mobile diaries. One of the first 

occurred in 2014, the Individual Diaries Feasibility Test (IDFT), which assigned respondents to 

either a desktop or mobile diary. The findings from the IDFT served as input for design decisions

for a larger CE Redesign Proof of Concept Test (POC). The POC was the first implementation 

where the online diary supported both desktop and mobile devices. 

More recent work, conducted by Westat, focused on further developing online desktop diary 

instruments that could take advantage of the larger screen size compared to mobile devices. This 

work was completed as Task Order 24, where Westat collaborated with BLS to develop three online 

desktop prototype instruments. The prototype instruments followed three different principles in 

their design.

 Prototype 1 focused on presenting all required actions and feedback on one screen with a 

clear, simplified navigational path.

 Prototype 2 focused on decomposing the primary data elements of the CE Diary into 

separate steps with greater use of visual cues.

 Prototype 3 focused on providing information about different expense categories to improve 

data quality and respondent understanding.

Prototype 1 and Prototype 2 were selected by BLS for laboratory testing. The results of these tests

revealed that the visual cues of Prototype 2 were visually appealing, but the decomposed design took 

longer for respondents to learn how to navigate the online diary. Prototype 1 was preferred for its 

simplicity and found to perform better based on an independent analysis of instrument timings. 

Prototype 1 was also a design that would allow BLS to move forward quickly in future tests.

For the current study, Westat worked with BLS to develop a one-week online diary that is accessible

by individual household members. The diary was designed to adapt to various screen sizes based on 

desktop computers, laptop computers, and mobile devices (smartphones and tablets). Prototype 1 

from Task Order 24 served as the basis of the online diary design. This was a design that could be 

adapted quickly and had a demonstrated preference with respondents in laboratory testing. The 

desktop version developed for the current study is shown below in Figure 2-1.
CE Online Diary Usability Testing – Task Order 

#26 

2-3 

 

Figure 2-1. Screen capture of CE Diary on the desktop platform

The screen presentation shown in Figure 2-1 is what respondents would see for desktop screens and 

laptops. The design displayed all task elements on one screen and organized entry tasks and a 

summary of entered expenses into two distinct fields. The left-hand field is where expenses are 

entered and could be edited. This field was made to be distinct and placed where respondents are 

expected to look first. The right-hand field included a summary of all expenses entered organized by 

date. In order to manage the volume of information displayed as expense entries accumulated, only 

the active date was expanded displaying expenses entered. Other dates could be expanded by 

selecting the desired date. The active date was defined as the date of the last expense entered. 

Expenses entered were organized by date and order of entry, with the most recent entered expense 

at the top of the list.

There are a number of design features implemented for the purpose of improving usability and 

providing respondents with feedback.

1. Preserved expense date and category once selected. Once respondents selected the expense 

date and the expense category these selections were preserved for the next entry. This

reduced redundancy when respondents had a long list of expenses to enter. The date and 
CE Online Diary Usability Testing – Task Order 

#26 

2-4 

 

category could be changed at any time if needed. The date and category selections were not 

preserved if respondents ended their online session.

2. Activation upon completion. Action buttons for entering, clearing, or updating expenses 

were inactive unless certain requirements were met. For entering an expense, the save button 

was de-emphasized (or greyed out) until all fields were completed for the expense. This 

ensured complete data and provided respondents with feedback on the next action to take 

and where it is implemented.

3. Visual cueing. Visual cues, such as temporary highlighting was used to provide respondents 

with feedback. Once an expense was entered the expense was temporarily highlighted in a 

distinct color in the summary list of expenses. This was done to draw the respondent’s 

attention and provide feedback that the expense was successfully entered.

A mobile optimized version of the CE Diary was developed for the current study. The mobile 

version considered the limited screen size available with mobile devices and the touch-based input. 

It was important for the mobile version to maintain the same visual characteristics, organization, and 

navigational cues. This ensured that respondents familiar with one version would be immediately 

familiar with the other. One distinct difference between the mobile and desktop versions was that 

the expense entry field and summary expense field were displayed on separate screens for the 

former, but on one screen for the latter. This is largely a function of the dramatic difference in 

screen size between the two presentations. The mobile optimized version is shown in Figure 2-2
CE Online Diary Usability Testing – Task Order 

#26 

2-5 

 

Figure 2-2. Screen capture of CE Diary on smartphone

After logging into the mobile version of the CE Diary, respondents were first presented with the 

summary expense screen. Adding expenses was done by touching the ‘add’ button in the lower 

right-hand corner which then displayed the expense entry screen. Unlike the desktop version, all 

action functions were placed along the bottom of the screen. This was done as it follows design 

conventions used in mobile applications, thereby leveraging knowledge and experience respondents 

have from interaction with other mobile sites or mobile apps.

The same design features from the desktop version were present in the mobile version with the 

exception of one. Since the expense summary was a separate screen, it was not possible to provide 

active feedback of the expense entered in the same manner by highlighting the new expense.

The online CE Diary featured both passive and active help to guide and inform respondents. Passive 

help was in the form of pop-up informational boxes that were initially displayed to respondents at 

key points of the online instrument. These did not require any respondent action to display. The 
CE Online Diary Usability Testing – Task Order 

#26 

2-6 

 

boxes would automatically disappear after six seconds or if dismissed by the respondent. Passive 

help provided respondents with how to initially enter an expense, how to access additional 

information, and reminded the respondent that the date and category would remain the same for the 

next expense. Figure 2-3 shows an example of active help. Active help was available through links 

identified by ‘i’ that provided definitions for different categories, or clarification for entry fields. 

These required respondent action to access.

Figure 2-3. Screen capture of active help

The online diary developed for this study included an expanded set of expense categories. The CE 

Diary currently uses:

 Food and Drinks Away from Home

 Food and Drinks for Home Consumption

 Clothing, Shoes, Jewelry, and Accessories

 All Other Products and Services
CE Online Diary Usability Testing – Task Order 

#26 

2-7 

 

Previous work conducted by Westat (Task Order 24) testing different desktop online diary designs 

included an expanded set of nine categories. The expanded set of categories is useful for reminding 

respondents of expenses that may be easily forgotten, or where receipts are not commonly kept by 

the respondent. An expanded set of categories is feasible for an online mode as it requires minimal 

additional space. In a PAPI mode expanding the set of expense categories can make the diary 

confusing to navigate and require many additional pages. The final set of expense categories 

included a total of ten based on the results of Task Order 24. The final wording for each category is 

based on wording provided by or reviewed by BLS during the online diary development. The 

expense categories included and wording are listed below.

 Meals, Snacks, and Drinks Away From Home

 Food and Drinks for Home Consumption

 Clothing, Shoes, Jewelry, and Accessories

 Medical Expenses and Supplies

 Entertainment and Recreation

 Home Furnishings and Decorative Items

 Education Expenses and Supplies

 Transportation Expenses

 Personal Care or Hygiene Items

 All Other Products, Services, and Expenses

For this study, Westat worked with BLS to prepare separate usability testing protocols for the 

placement and pick-up interviews and user guide for respondents. Testing was conducted from 

March 20th, 2017, through May 25th, 2017. This report provides the results of the in-person usability 

test. Chapter 3 discusses the methods of the testing, including respondent recruitment, and 

interviewer training and procedures. Chapter 4 provides the results of the placement interview first 

looking at data collected by the diary, then reviewing respondent feedback. Chapter 5 presents 

Westat’s recommendations and reviews future enhancements as options to incorporate into future 

versions of an online-based CE Diary.
CE Online Diary Usability Testing – Task Order 

#26 

3-1 

 

Westat conducted usability testing of the online CE Diary from March through May of 2017. 

Usability testing consisted of three components: 1) a placement interview; 2) a seven-day respondent 

reporting period; and 3) a pick-up interview. A total of 60 placement and pick-up interviews were 

targeted for usability testing. For the placement interview, 62 interviews were completed, and 61

pick-up interviews were completed. Both the placement and pick-up interviews were completed at 

the respondent’s residence. All interviews were completed in the Washington DC metro area 

including the outer suburbs of Maryland and northern Virginia.

Recruitment

Participants were recruited through weekly advertisements placed on the Internet website Craigslist

and through Westat’s database of participants. Those contacted through Westat’s participant 

database included only those who had not participated in any Westat study in the past six months. 

The vast majority of participants were recruited through Internet advertising.

All participants were screened to determine eligibility for the usability test. Participants were required 

to own a computer (desktop, laptop, Chromebook), own a smartphone, have access to the Internet 

in their home, and report some shopping or purchase activity. Diary use was at the individual and 

not the household level. Therefore, recruitment also targeted households with children age 15-17

and households with multiple adults, but this was not a requirement to be included. The recruited 

participant would be the main diarist and was the point of contact for the interviewers. Table 3-1 

shows the demographic composition for recruited participants and Table 3-2 shows household level 

characteristics.

Methods 3 
CE Online Diary Usability Testing – Task Order 

#26 

3-2 

 

Table 3-1. Distribution of demographic compositions for test participants (main diarist)

Characteristic Count Placement Count Pick-up 

Age 

18-24 2 1

25-30 13 13

31-35 9 9

36-40 8 8

41-50 14 14

51-64 13 13

65+ 3 3

Sex 

Male 12 12

Female 50 49

Race 

White 27 27

Black 24 23

Asian 10 10

Other 1 1

Education 

High school or GED 13 12

Some college 12 12

College degree (4-year) 23 23

Graduate School 14 14

Table 3-2. Distribution of household compositions

Characteristic Count Placement Count Pick-up 

Number of Adults in Household 

One 18 18

Two 28 28

Three 13 13

Four or more 3 2

Youth 15-17 in Household 

Yes 19 19

No 43 42

Participants, as the main diarist, were told they would receive $40 for taking part in the usability test 

($20 for completing the placement interview and $20 for completing the pick-up interview). All 

other household members age 15 and older would receive $10 at the pick-up interview for their 

participation. Other household members had to access the online diary and enter at least one 

expense (if any) to be considered a participant.
CE Online Diary Usability Testing – Task Order 

#26 

3-3 

 

Placement Interview Procedure

The placement interview was the first of the three participation components. All placement 

interviews took place in the respondent’s home. Only the main diarist needed to be present, but 

when the visit was first scheduled other age-eligible household members were encouraged to be 

present.

The placement interview began with the interviewer providing background on the study and 

obtaining consent. The consent form administered covered the placement and the pick-up visit. For 

households with youth age 15-17, the parent or guardian was administered the parental consent 

form giving permission to speak with the youth. Other adult household members completed 

consent passively the first time they accessed the online diary. The same process was used for youth 

assent. An example of this is shown in Figure 3-1 below.

Figure 3-1. Screen capture of passive consent (adult)

After covering the study background and consent, the interviewer then covered the content 

components of the placement interview. The protocol for the placement interview is shown in 

Appendix A. The interviewer explained the role of the respondent as the “main diarist.” This 

included entering all purchases the respondent made during the reporting period; reminding other 

household members to enter their own purchases; and assisting other household members in 

accessing the diary and creating links on their computer or other devices. The interviewer defined 

the dates for the assigned seven-day reporting period, then explained how as the main diarist he or 
CE Online Diary Usability Testing – Task Order 

#26 

3-4 

 

she would be able to monitor other household members’ usage of the online diary. The online diary 

included a link for main diarists where they could see whether other household members had 

accessed the diary and how many expenses they had entered. They would not be able to see any 

detail related to the purchases other household members entered.

Once the respondent’s role was explained, the interviewer verified the composition of the household 

and offered to set up links to the online diary on any of the respondent’s computers or mobile 

devices. The links were shortcuts to the online diary and did not store any data on any of the 

respondent’s computers or mobile devices. The interviewer started by collecting the type of internet 

access for the household. The interviewer then recorded each type of device owned or available to 

each household member and whether a link to the online diary was set by the interviewer. 

A user guide was provided to the main diarist of each household. The complete user guide is shown 

in Appendix B. The interviewer reviewed the user guide with the respondent. The user guide

provided detail and visual examples of what information should be entered into the online diary.

This covered examples mapping information from prototypical receipts to diary fields, how to 

navigate the online diary, frequently asked questions, and instructions for creating links to the online 

diary on different devices (computers, Apple devices, and Android based devices). The interviewer 

reviewed the online diary detail fields (description, amounts, etc.) and reviewed each expense 

category to explain differences between various categories. The interviewer then asked if the 

respondent had any questions on what was covered.

The placement interview concluded with the interviewer collecting respondents’ level of comfort 

using computers, mobile devices, and whether they had done any online shopping recently. The 

interviewer provided the respondent with a card that listed the online diary usernames and 

temporary passwords for all eligible household members. The card also listed the assigned dates for 

the seven-day reporting period. The interviewer then set an appointment for the pick-up interview 

8-10 days following the placement interview.

During the seven-day reporting period, project staff would review the online diaries to check for 

login activity and whether any expenses had been entered. If no expenses were entered within four 

days of the placement interview, the interviewer was instructed to call and remind main diarists to 

enter any purchases they had made, or check if they were having any difficulty accessing or using the 

diary.
CE Online Diary Usability Testing – Task Order 

#26 

3-5 

 

Pick-up Interview Procedure

The pick-up interview was the last component of participation for respondents and occurred after 

the assigned seven-day reporting period. The interviewer contacted the respondent the day before 

the scheduled pick-up interview to confirm the appointment and reschedule if necessary. 

Interviewers were provided with a summary of expenses entered by the household, which included 

household member identifier, expense category, and expense description. This was provided so that 

interviewers could review any questionable entries with the respondent. Questionable entries were 

those that included vague descriptions, appeared to include multiple items, or appeared to be a 

mismatch between the category selected and the item description.

Once at the respondent’s residence, the interviewer administered the protocol for the pick-up 

interview. The protocol is shown in Appendix C. For these usability interviews, the primary purpose 

of the pick-up interview was to debrief respondents about their experience using the online CE 

Diary. This focused on overall impressions of the online diary, difficulty with how to use or navigate 

the online diary, and device (desktop or mobile) specific questions relating to using the online diary. 

Measures relating to usage or function of the online diary were collected by asking respondents to 

report their level of satisfaction on a five-point scale. Interviewers reviewed questionable entries 

using the expense summary previously described with respondents and probed to determine if there 

were any expenses that were missed or forgotten by respondents.

Attempts were made to collect feedback from all household participants, but only the main diarist 

was required to be present. For other household members who were not present, the main diarist 

was given an online link to provide to other household members where they would complete a short 

questionnaire about their experience online. A total of 6 online questionnaires were completed by 

other household members not present during the pick-up interview.
CE Online Diary Usability Testing – Task Order 

#26 

4-1 

 

The findings presented cover a number of different topics related to the usage of the online CE 

Diary. Findings are divided into two main sections: findings based on respondent behavior and use 

as determined through a review of the diary data collected; and findings based on respondent 

feedback during the debriefing (pick-up) interview. 

4.1 Diary Use and Data 

Diary Acceptance and Use

One of the first tasks to get the respondent acquainted with the online diary was for the interviewer 

to set up short-cut links on any online accessible devices. Overall, 48 of the 62 (77%) main diarists

allowed the interviewer to create a link on any device. Table 4-1 shows a breakdown of device type 

ownership and whether the interviewer created a link to the online diary on the device. Interviewers 

created links on the devices suggested by the respondent, but asked to create links for all devices 

they would use.

Table 4-1. Distribution of device type owned and whether an online short-cut was added (main 

diarist)

Device Type Owned Link Added Percent Link 

Primary Desktop or Laptop 54 29 54%

Secondary Desktop or Laptop 9 0 --

Smartphone 61 35 57%

Other Device (tablets) 21 6 29%

For other household members, the number of online short-cut links created was much lower with 

only 18 of 72 (25%) having a link created on a device they used. Table 4-2 shows a breakdown of 

device type ownership and online link creation for other household members. The low number of 

links created for other household members is because it was rare for other household members to 

be present during the placement interview. For multiple person households, at least one other age 

eligible household member was present 24% of the time. Interviews were generally conducted 

during the day when other household members may be at work or school.

Findings 4 
CE Online Diary Usability Testing – Task Order 

#26 

4-2 

 

Table 4-2. Distribution of device type owned and whether an online short-cut was added (other 

household members)

Device Type Owned Link Added Percent Link 

Primary Desktop or Laptop 42 12 29%

Secondary Desktop or Laptop 9 0 --

Smartphone 67 5 7%

Other Device (tablets) 10 2 20%

The devices listed in Tables 4-1 and 4-2 are those accessible to the respondent and are often shared 

among household members. Desktop and laptop devices are often shared among household 

members, while smartphones are primarily person-level devices. The higher number of links set on 

desktop or laptop devices for other household members compared to smartphones reflects device 

sharing.

Setting up links to the online diary was beneficial as a way to ensure the respondent had access to 

the diary on at least one device. However, this does not guarantee respondents will follow through 

and further access the diary. Respondents were required to log in and change their password the day 

of their placement visit, but did not do this while the interviewer was present.

Of the 62 placement interviews completed, 61 main diarists accessed the diary by logging on and 

creating a password. Looking at this by reporting day, 75% changed their password by the third 

reporting day. The numbers are much lower for other household members where of the 72 other 

household members, only 50 or just over two-thirds, accessed the diary and changed their password. 

Figure 4-1 shows the distribution of when initial diary access occurred by respondent type and 

reporting day.
CE Online Diary Usability Testing – Task Order 

#26 

4-3 

 

Figure 4-1. Distribution of initial diary access by respondent type and reporting day

Figure 4-1 shows that initial diary access occurred throughout the reporting period with notable 

differences between the main diarist and other household members. About two-thirds of main 

diarists initially accessed the diary on the first two reporting days. For other household members, 

initial access occurred more throughout the week with the lowest points occurring during the middle 

of the reporting period. A small proportion of respondents did not access the diary until the end of 

the reporting period (day seven or later) with this proportion higher for other household members 

(8% for main diarist, 20% for other household members).

During the pick-up visit, interviewers probed about the participation of other household members.

Reasons for non-participation by other adults were largely lack of interest or reports they were too 

busy to track expenses. There were a few (4) cases where the main diarist did not ask other 

household members because they felt they were too busy, or would not be interested. One other 

adult household member felt participation was unnecessary since the main diarist was entering

expenses (although this did not capture the other adult’s expenses). Finally, one youth did not 

participate as the main diarist reported the age eligible youth had no means for incurring expenses as 

the youth’s parents (one was the main diarist) paid for everything.

0%

10%

20%

30%

40%

50%

60%

1 2 3 4 5 6 7 8+

Main Diarist Other Household Members
CE Online Diary Usability Testing – Task Order 

#26 

4-4 

 

Table 4-3. Reasons for other household member non-participation by relationship to main 

diarist

Device Type Related Adults Non-related Adults Youth 

Not interested / too busy 10 3 1

Main diarist did not ask others 4

Not present during reporting period 2

Felt unnecessary due to main 

diarist participation

1

No means / no expenses 1

Interviewer probing revealed that there was some undercoverage of household members reported in 

recruitment and some proxy participation by other household members.

 One main diarist did not report two other adults who lived in the household. The 

respondent did not consider these adults as part of the household (adult parents who live 

with the respondent), but a separate household.

 Two main diarists entered expenses for other household members using their individual 

diaries. The main diarist collected receipts from other household members to track their 

expenses.

Accessing the online diary was highly predictive of using the online diary defined as entering at least 

one expense. All 61 main diarists who accessed the diary entered at least one expense, while 49 of 

the 50 other household members who accessed the diary entered at least one expense. Figure 4-2 

shows the distribution of when the first expense was reported by respondent type and reporting day.
CE Online Diary Usability Testing – Task Order 

#26 

4-5 

 

Figure 4-2. Distribution of initial expense report by respondent type and reporting day

The main diarist was instructed to access and create a new password following the placement visit 

(the same day) and to instruct other household members to do the same. This does not mean there 

would be any expenses to report the same day. The distribution shown in Figure 4-1 may reflect 

respondents who waited until there was an expense to report. Figure 4-2 has a similar distribution as 

expected, but nearly half of respondents (main diarist and others) who accessed the diary on the first 

reporting day did not enter an expense that day. It is not possible from Figure 4-2 to determine the 

timeliness of expense entry. Instead, we examine the difference between expense date and entry 

date. Table 4-4 below includes all expenses entered into the online diary and shows the distributional 

difference between expense date and when the expense was entered.

Table 4-4. Distribution of difference between expense date and entry date for all expenses

Difference 0 1 2 3 4 5 6+ 

Percent 27.0 34.0 10.7 10.7 5.7 3.6 8.3

In Table 4-4, the column with a heading of zero represents expenses that were entered the day they 

occurred. Just over one-quarter of expenses were entered the day they occurred with 61% having 

been entered within one day. Overall, a large majority of expenses were entered one day or more 

after the expense occurred.

0%

10%

20%

30%

40%

50%

60%

1 2 3 4 5 6 7 8+

Main Diarist Other Household Members
CE Online Diary Usability Testing – Task Order 

#26 

4-6 

 

Failed Login Attempts

Failed login attempts to access the online diary are a potential indicator of difficulty with the diary. 

For a paper diary the only barrier to access is recalling where the diary may have been placed. The 

online diary by comparison requires a username and a password. The online CE Diary tracked failed 

attempts to access the diary at the password level. Meaning that respondents needed to at least 

correctly recall (correctly enter) their username in order to track failed login attempts. Failed 

attempts at the username level were not captured, as there is no way to link the attempt to a 

respondent without an identifier (e.g., username). Due to this, the total number of failed attempts 

may be underrepresented. Table 4-5 below shows the distribution by total number of respondent 

failed login attempts.

Since login activity and failed login attempts were tracked by respondent, a successful login was 

necessary in order to create a record. This means that login attempts that were completely 

unsuccessful (no successful login) could not be captured. There were no reports by respondents of 

complete inability to access the online diary.

Table 4-5. Respondent distribution of failed login attempts to the online diary

Total number of 

failed attempts 0 1 2 3 4 5 6 7+ 

Total 46.9 12.6 8.1 3.6 3.6 3.6 4.5 17.1

Less than half of all respondents (47%) did not have any failed attempts to access the online diary. 

Another 36% experienced 1 to 6 failed attempts, while 17% experienced 7 or more failed attempts 

to access the diary. A review of the characteristics of respondents with the most failed attempts (7 or 

more) did not reveal any patterns in terms of age, or device usage1. The maximum number of failed 

attempts was 52, followed by 41 failed attempts. The respondent with 52 failed attempts was an 

older respondent (over age 50). The respondent called the help desk reporting difficulty entering her 

temporary username and password (43 of the 52 attempts occurred when setting her password for 

the first time).

 

1 The average age for those with 7 or more failed login attempts is 39 years and lower than the average age of those with 

no failed attempts (42 years). Of these 19 respondents, 11 attempted to login using a desktop device, 7 used a mobile 

device, and 1 respondent attempted with both device types.
CE Online Diary Usability Testing – Task Order 

#26 

4-7 

 

Exactly one-half of these failed attempts to access the online diary occurred during initial access in 

the process of setting up a password. The temporary password assigned consisted of lower-case 

letter and number only, while the respondent was required to create a password that was 8 

characters long, and included a number, upper-case and lower-case letter and one special character 

(!@#$%^&*).

Usage Characteristics

How respondents used the online diary is examined by reviewing distributions of when expenses 

were entered, usage of different devices types (desktop/laptop and mobile devices), and the types of 

expenses entered by respondents.

Figure 4-3 shows the mean number of expenses entered by reporting day for the main diarist. Figure 

4-4 shows the same information for other household members. During the reporting period,

household diary use was monitored by project staff. Any household that had not entered any 

expenses by the fourth reporting day received a reminder call to enter any expenses and check 

whether respondents had any difficulty accessing or using the online diary.

The fourth reporting day was used as it was the midpoint of the reporting period. It was believed 

that respondents would be able to recall any expenses not entered and the contact would remain 

salient for the last three reporting days. Only the main diarists received the reminder contact and 

other household members were not contacted. Individual contact information was not collected 

during the placement visit for other household members. In many cases there may be a household 

phone, but due to the increasing proportion of cellular telephone only households this is decreasing. 

The main diarists was asked to remind other household respondents to enter their expenses.
CE Online Diary Usability Testing – Task Order 

#26 

4-8 

 

Figure 4-3. Mean number of expenses entered by reporting day for main diarist

Figure 4-4. Mean number of expenses entered by reporting day for other household 

members

4.3

6.4

6.9

8.3

6.9

5.5

9.9

11.3

0.0

2.0

4.0

6.0

8.0

10.0

12.0

1 2 3 4 5 6 7 8+

4.0

2.4

4.2

8.1

2.8

5.3

6.0 5.9

0.0

2.0

4.0

6.0

8.0

10.0

12.0

1 2 3 4 5 6 7 8+
CE Online Diary Usability Testing – Task Order 

#26 

4-9 

 

Both Figures 4-3 and 4-4 show peaks in the mean number of expenses entered in the middle and 

end of the reporting period. The peak in the middle reflects reminder contacts initiated by 

interviewers demonstrating their effectiveness which extended to other household members. This is 

encouraging as it indicates the main diarist communicated the reminder to other household 

members. The peak at the end is the result of expenses entered by respondents just prior to the 

pick-up interview. 

Device Types Used

In terms of device usability, the online CE Diary was required to be accessible across a variety of 

devices, such as desktop computers, laptop computers, and various mobile devices (generally 

smartphones). This creates challenges in relation to online instrument design. Unlike computerized 

modes such as CAPI or CATI where the instrument must function on one device, an online 

instrument must function across multiple devices, computer operating systems, and internet 

browsers. Figure 4-5 illustrates this challenge.

Figure 4-5. Main factors online instruments must take into consideration for function and 

design

Device Type  Desktop  Laptop  Tablets  SmartphonesOperating System

 Windows  Mac OS  Android  iOS

Internet Browser

 Chrome  IE  Edge  Safari  Firefox

X X

What is missing in Figure 4-5 are the various versions of operating systems and web browsers that 

may affect online instrument functionality. For the online CE Diary, information on the device type, 

operating system, and browser was captured as paradata. Table 4-6 shows the various types detected. 

It is important to note that devices such as desktops and laptops cannot be separately detected.
CE Online Diary Usability Testing – Task Order 

#26 

4-10 

 

Table 4-6. Device characteristic detected when accessing the online CE Diary

Characteristic Number 

Device 

Desktop/Laptop 97

Mobile (smartphone) 47

Mobile (tablet) 4

Operating System 

Windows 74

Mac OS 18

Android 22

iOS 28

Windows Mobile 1

Unknown 5

Browser 

Chrome 86

Safari 32

Microsoft 15

Firefox 8

Unknown 7

The device characteristics in Table 4-6 represent a total of 22 unique combinations detected by the 

online diary. The number of unique devices takes into account the operating system version. No 

respondent reported an inability to access or use the online CE Diary because it would not work 

with their device.

Respondents were not instructed on what device type to use to access the online diary and could use 

whichever they preferred. A number of respondents were “mobile only” meaning their only device 

was a smartphone. Seven main diarists were mobile only and 20 other household members (of those 

entering expenses) were mobile only. Table 4-7 shows the number of respondents using either a 

desktop, laptop, or a mobile device. The row totals are greater than the number of respondents for 

each group as some respondents used both device types (13 main diarist and 7 other household 

members used both device types).

Table 4-7. Device types used to enter expenses by respondent type

Device Type Desktop / Laptop Mobile (smartphone / tablet) 

Main Diarist 43 31

Other Household Members 37 20

Desktop devices were used more than mobile devices and had a higher average number of expenses 

entered. Desktop devices had a mean of 8.3 expenses entered compared to mobile devices with a 
CE Online Diary Usability Testing – Task Order 

#26 

4-11 

 

mean of 5.3. This may reflect how the devices were used as respondents could use their mobile 

device anywhere they have online access, such as while out shopping. A quarter of main diarists (15) 

reported entering at least one expense while away from home. Examining differences between 

expense date and entry date, 24% of all expenses entered with a desktop device were entered the day 

of the expense compared to 33% of all expenses entered with a mobile device. 

Respondents who used their mobile device, but only while at home were asked why they did not 

enter any expenses while out. Fourteen respondents provided a reason with the most common 

reason being some form of preference to enter expenses at home. The reasons provided by 

respondents are shown in Table 4-8 below.

Table 4-8. Reasons provided for not entering expenses while out (smartphone users only)

Device Type Count 

Preferred home / never thought about using while out 5

Too busy / felt would be too hard to do 3

Concern about making mistakes 2

Forgot password 2

Poor data connection 1

Home gives more time to concentrate on task 1

Session Level Timings (Duration)

Activity timings were captured as paradata in the online diary. This permits a review of how much 

time respondents spent each time the diary was accessed – defined as a session. A session duration is 

calculated by taking the difference between the session start time (when the respondent successfully 

logged into the diary) and logout time (session end time). Session start times do not vary, but logout 

times can vary based on how the respondent exits the diary, or whether the online diary forcibly logs 

the respondent out after a period of inactivity. Respondents initiated logouts by clicking the logout 

link in the top-right of the online diary.

Respondents were forcibly logged out of the online diary after 15 minutes of inactivity, but session 

timings for these logout types average more than six hours. This is due to the expiration setting 

created for authentication tokens which are created each time the respondent logs into the online 

diary. The authentication token was set to expire after six hours to allow respondents to remain 

within the diary without having to log in again every 15 minutes. For usability testing these were set 
CE Online Diary Usability Testing – Task Order 

#26 

4-12 

 

for six hours, but can be made much shorter. Table 4-9 shows the mean duration (in minutes) of 

online diary sessions by logout type and Table 4-10 shows the mean duration (in minutes) by device 

type for respondent initiated logouts.

Table 4-9. Diary session duration by logout type (respondent initiated, or system timeout)

Logout type Session Count Mean Duration (minutes) 

Respondent logout 326 6.9

System timeout 472 335.4

Table 4-10. Diary session duration by device type (respondent initiated logouts)

Device Type Session Count Mean Duration (minutes) 

Desktop 240 7.9

Mobile 86 4.0

Respondent sessions averaged just under 7 minutes. When examined by device type, desktop 

sessions averaged much longer than mobile sessions. Desktop sessions were nearly double the 

length of mobile sessions with an average of nearly 8 minutes, compared to 4 minutes. 

Expense Categories

The expense categories available to respondents included 10 different categories. This is more than 

the four currently in use for the paper CE diary. Respondents could click or select links that would 

provide brief definitions of what should be included, but otherwise the determination of what 

category to select was up to the respondent. How these categories were used by respondents and if 

any differences by type of device used when entering expenses are reviewed.

Figure 4-6 shows the distribution of all expenses by category type. Food type expenses were a large 

majority of all expenses entered. Food for home consumption was 43% of all expenses with food 

away from home an additional 20%. Education expenses and home furnishings and decorative items 

made up the smallest proportion of expenses with less than 2% combined. The low proportion for 

education expenses may reflect the seasonality of this expense type. The six new categories (items 4 

through 9 in Figure 4-6) consisted of 19.3% of all expenses entered, with transportation expenses 

making up the largest proportion of these.
CE Online Diary Usability Testing – Task Order 

#26 

4-13 

 

Figure 4-6. Distribution of expenses by category type (across all device types)

Figure 4-7 shows the same distribution, but breaks out the device type used to enter expenses. The 

most striking observation is that the proportions for meals away from home and entertainment 

expenses more than double for mobile devices compared to desktop device. A hypothesis for this 

observation is that these are expenses that are likely to have low complexity (single receipt item, such 

as restaurant meal or movie tickets), or occur during extended periods away from home. The 

proportions shown for food for home consumption support this hypothesis where the proportion 

for desktop devices is more than double mobile devices. Expenses for food for home consumption 

may involve more complex receipts where there are multiple items on one receipt (e.g., a grocery 

trip) or perishable items where the respondent will be returning home.

The previous section noted that only a few respondents reported using mobile devices while away 

from home. However, many respondents used mobile devices to enter expenses while at home and 

27 respondents were mobile only. The likely hypothesis for the distributions shown in Figure 4-7 is 

that respondents are more willing to use mobile devices for less complex expenses (receipts).

0 5 10 15 20 25 30 35 40 45 50

1 - Meals, Snacks, and Drinks Away From Home

2 - Food and Drinks For Home Consumption

3 - Clothing, Shoes, Jewelry, and Accessories

4 - Medical Expenses and Supplies

5 - Entertainment and Recreation

6 - Home Furnishings and Decorative Items

7 - Education Expenses and Supplies

8 - Transportation Expenses

9 - Personal Care, or Hygiene Items

10 - All Other Products, Services, and Expenses

proportion of all expenses (n = 2,201)
CE Online Diary Usability Testing – Task Order 

#26 

4-14 

 

Figure 4-7. Distribution of expenses by category within each device type (desktop or 

mobile)

Data Quality

A review of data quality focuses on the expense description field of the online diary. This is an open 

text field respondents are required to complete for each expense. Project staff reviewed each 

expense and coded for the following qualities:

 Erroneous multiple items – cases where more than one expense was included in a single 

entry.

 Vague description – the expense cannot be determined from the description.

0 10 20 30 40 50 60

1 - Meals, Snacks, and Drinks Away From Home

2 - Food and Drinks For Home Consumption

3 - Clothing, Shoes, Jewelry, and Accessories

4 - Medical Expenses and Supplies

5 - Entertainment and Recreation

6 - Home Furnishings and Decorative Items

7 - Education Expenses and Supplies

8 - Transportation Expenses

9 - Personal Care, or Hygiene Items

10 - All Other Products, Services, and Expenses

proportion of all expenses within device

(mobile n = 679; desktop n = 1,522)

Mobile Desktop
CE Online Diary Usability Testing – Task Order 

#26 

4-15 

 

 Establishment name – cases where the establishment name (e.g., restaurant name) was given 

in the description and no other information provided.

 Incorrect category – where the category selected does not seem appropriate based on the 

item description.

 Unidentifiable – cases where the description is unintelligible and the expense cannot be 

determined.

Across all expenses 9% (202) were determined to have at least one of the five quality issues

described above. In terms of respondents 55% (60) had at least one expense determined to have one 

of the five quality issues. Table 4-11 shows the coding distribution as a percentage of all expenses.

Table 4-11. Distribution of expense description issues as a percentage of all expenses

Description Issue Type Count Percent 

Erroneous multiple items 52 2.3%

Vague description 37 1.6%

Establishment name 59 2.6%

Incorrect category 41 1.8%

Unidentifiable 13 0.6%

There was no single issue type that disproportionately accounted for those identified by coders. The 

use of establishment name was marginally the most common and exclusively occurred for meals 

away from home. In these cases the restaurant name was provided for the description. This may 

reflect confusion on the part of respondents with how to describe meals with multiple items or for 

multiple persons. For example, “dinner with family” versus “hamburger, fries, coke, etc...”

There is some clustering of description quality issues within respondents as 14 respondents account 

for 55% of coded issues. These respondents tended to be older with an average age of 45 (the 

average age across all respondents is 38). The youngest respondent in this group was 16 followed by 

a 30 year old, while the oldest was 66. Overall, data quality for the description field is high with a 

small number of respondents responsible for nearly half of the issues observed.

Editing and Deleting Expenses

Expenses could be edited or deleted from the online diary. Deleting expenses was rarely used with 

only 35 expenses deleted – roughly less than 2% of all expenses entered. Editing expenses was more 
CE Online Diary Usability Testing – Task Order 

#26 

4-16 

 

frequently used with 186 expenses edited (8% of entered expenses). Table 4-12 shows the 

percentage of expenses for each category edited as a proportion of expenses entered within each 

category. This accounts for the uneven distribution of expense types.

Table 4-12. Percentage of expenses edited within each category as a proportion of number 

entered for each category type

Device Type Percent 

Meals, Snacks, and Drinks Away From Home 9.9%

Food and Drinks For Home Consumption 8.8%

Clothing, Shoes, Jewelry, and Accessories 5.1%

Medical Expenses and Supplies 15.6%

Entertainment and Recreation 8.5%

Home Furnishings and Decorative Items 9.5%

Education Expenses and Supplies 9.1%

Transportation Expenses 4.8%

Personal Care, or Hygiene Items 10.0%

All Other Products, Services, and Expenses 6.7%

The category for medical expenses and supplies had proportionally more expenses edited compared 

to all other categories. The online diary did not capture what was edited for each expense, for 

example, whether the date, category type, or amount was changed. The categories listed in Table 4-

12 are the final categories and may reflect changes made by respondents from another category. 

However, the higher proportion for this category type suggests respondents may encounter more 

confusion for expenses in this category compared to others.

Forgotten Expenses

For any days where an expense was not reported, respondents were asked if there were any expenses 

they missed or forgot to report. This is challenging as respondents are being asked to recall expenses 

they may have difficulty recalling. Interviewers primed respondents by asking the respondent to 

think about automatic charges, subscriptions that may have renewed, online or digital purchases, or 

purchases where they may not have gotten a receipt.

Fourteen respondents reported missed or forgotten expenses. The online diary data was not 

corrected with this information, but we present expenses types that were reported as forgotten or 

missed.
CE Online Diary Usability Testing – Task Order 

#26 

4-17 

 

 Six respondents forgot to include automatic withdrawals or online subscriptions, including 

mortgage, cell phone payment, rent, pet wellness plan payment, highway tolls paid with EZPass, subscription to Amazon Digital, and subscription to YouTube Red.

 Four respondents forgot to report expenses with no receipts, including vegetable purchases

at a farmer’s market, purchase of lip gloss, a church donation, and parking.

 Two respondents forgot to include payments for gasoline.

 One respondent spent 25 cents on fish food and decided this was not significant enough to 

report.

 One respondent saved a receipt, but could not recall what was purchased since the receipt 

was vague – the purchase was labeled as “miscellaneous.”

Expenses that were unreported were largely due to the expense occurring outside the respondents 

immediate awareness (e.g., an automatic subscription, or renewal), or were because a receipt was not 

provided or saved by the respondent.

No Purchases

Respondents were instructed to indicate if they did not have any expenses for the reporting period. 

This was done by logging into the online diary on the seventh reporting day (or later) where a popup would show if no expenses had been entered. The pop-up box included a question asking if the 

respondent did not have any expenses for the reporting period.

Overall, there were seven respondents who indicated they did not have any purchases, but later 

entered expenses. Only one respondent did not have any expenses, but this person did not access 

the diary on the reporting day the pop-up question was visible.

It is possible that the conflicting reports are due to expenses that occurred after the respondent 

reported there were no expenses. However, a review of the online diary question for no expenses 

shows the question wording was potentially confusing and negatively worded. The wording used 

was: “It looks like you have not reported any expenses. Did you not have any expenses to report 

during your reporting period? (yes/no).” This likely confused respondents who were late expense 

reporters who misreported no expenses. Figure 4-8 shows the text from the pop-up question.
CE Online Diary Usability Testing – Task Order 

#26 

4-18 

 

Figure 4-8. Screen capture of pop-up question asking if no expenses were incurred for the 

reporting period

Summary: Diary Use and Data

This section focuses on how the online diary was used by respondents by reviewing data entered by 

respondents supported by interviewer notes and paradata. Overall, the diary functioned across a 

number of devices and software systems. The limited review of data quality suggests that data quality

is high, reflecting how well respondents followed and understood the expense entry task. True 

values for the expenses entered are unknown and may reveal larger discrepancies. Nonetheless, there 

are a number of conclusions that can be drawn.

1. Respondents were willing to allow interviewers to create links to the diary on their personal 

devices. However, this only extends to the main diarist as it was rare for other household 

members to be present during the placement interview. Creating a link is not predictive of 

using the diary since all but one main diarist entered expenses (the one non-participant 

allowed a link to be created).

2. Use of the online diary by other household members was much lower (68%) relative to the 

main diarist. This suggests the main diarist is not well equipped (and in some cases not 

motivated) to gain cooperation from other household members for the diary task.

3. Interviewer reminders are important and increase compliance with the online diary task. 

Reminders were only provided to households where no expenses had been entered by the 

fourth reporting day.

4. Mobile devices (smartphones) were frequently used by respondents and increased the 

number of expenses that were entered the day the expense occurred. This decreases the 

chance that expenses may be forgotten. Mobile device use may be linked to receipt 
CE Online Diary Usability Testing – Task Order 

#26 

4-19 

 

complexity rather than location. Only a quarter of respondents (main diarist) reported 

entering an expense while away from home.

5. Session duration was much longer for desktop devices (nearly double) compared to mobile 

devices. This supports the conjecture that respondents use mobile devices for less complex 

online diary entries. This also supports the use of mobile devices for more immediate entries 

(as expenses occur), supported by the increase in same-day entries for mobile devices.

4.2 Placement Interview 

Very little data was collected during the placement interview as the purpose of the placement 

interview was to orient the respondent for the reporting task. Interviewers were asked about the 

overall procedures to see if there was any perception that respondents were impatient with the 

placement interview, received any comments regarding the incentive, or if any other issues were 

encountered.

Interviewers reported that the placement procedures worked well. Interviewers generally did not feel 

rushed and felt respondents were attentive. Only one interviewer reported that one respondent gave 

impressions that the placement interview was taking too long. A few interviewers noted that the 

instructions were viewed as overly detailed by some respondents, while other respondents appeared 

to find this helpful. There were very few comments from respondents about the incentive offered. 

One respondent (another household member, not the main diarist), reported that the expense 

reporting task was too tedious to do for only $10. Many interviewers commented that they feel a 

larger incentive for other household members would be helpful to get them to participate in the 

diary reporting task.

4.3 Respondent Feedback 

The previous section reported how well the diary functioned looking at data entered and respondent 

behavior. Respondent behavior included when data was entered, who in the household entered 

expense data, and the device used to enter expense data. This section focuses on respondent 

reactions to the online diary task, ratings of various functions within the online diary, and any 

recommendations or changes to the online diary.
CE Online Diary Usability Testing – Task Order 

#26 

4-20 

 

Respondent feedback was collected from 74 respondents; 60 main diarists and 14 other household 

members present during the pick-up interview. Only 6 respondents (other household members) 

completed the online questionnaire. Results are presented for all respondents.

General Reactions

The debriefing started with asking respondents to provide their general reactions to interacting with 

the online diary. This offers a summary reaction from respondents of their experience and reveals 

salient problems or difficulties without direct probing.

Overwhelmingly, respondents had a positive experience using the online diary. Of 74 respondents, 

68 provided a positive description of the diary. Respondents most commonly used words such as 

“easy” or “straightforward” when describing the diary.

The five respondents reporting negative reactions reported multiple difficulties. There were two 

forms of difficulties, those associated with the expense entry task, and those associated with 

accessing or using the online instrument. The difficulties reported are summarized below.

Difficulties with the expense entry task:

 Two respondents found selecting the expense category difficult, unsure where to place some 

expenses.

 One respondent found entering expenses “cumbersome and time consuming” due to the 

need to enter items separately.

Difficulties with using the online diary

 Two respondents had trouble accessing the diary, having trouble with their username or 

password.

 One respondent found navigation “clunky.” The respondent liked that it was possible to tab 

through each field, but felt it took too long to tab to “save.”

 One respondent reported entering the total cost was confusing as she was not sure if the 

dollar sign ($), or zeroes after the decimal should be entered.
CE Online Diary Usability Testing – Task Order 

#26 

4-21 

 

 One respondent thought her expenses had not been saved (mobile version) as she did not 

realize it was necessary to click on the date to expand the expense summary.

While a large number of respondents had positive impressions of the online diary, a number 

reported frustration with specific tasks, or difficulties with using the online diary. Table 4-13 lists the 

issues reported, again classified as problems with the expense entry task or using the online diary.

Table 4-13. Distribution of reported difficulties by type (counts do not reflect unique 

respondents as multiple reasons provided by some respondents)

Difficulties with expense task Count 

Entering individual items tedious 8

Difficulty with determining category 3

Difficulty excluding tax (not separated on receipt) 3

Difficulty determining other component (e.g., shipping) 3

Difficulties with online diary 

Trouble logging in, setting, or remembering password 11

Confusion with how to save/expenses lost (mobile) 3

Did not know what to do with no purchases/did not see ‘no 

expenses’ pop-up

3

Did not like selecting date twice (mobile) 2

Slow performance on laptop 1

Slow performance on mobile 2

Instrument froze after entering one expense (tablet) 2

Instrument froze after entering one expense (laptop) 1

Problems remembering or setting the password for the diary were the most prevalent. One 

respondent reported taking a picture of his username and password with his phone so he could 

remember it. 

The most troubling problem was for the three respondents who reported their expenses were not 

saved. These were all associated with mobile use of the diary. This occurred when respondents did 

not use the ‘save’ button and instead used the browser back button to go back to select the date and 

enter a new expense. One additional respondent commented that it would be helpful to have a 

confirmation for the mobile version that the expense was saved.

For the six respondents who reported slow performance of the online diary, or the online diary 

froze after one expense was entered, they switched to another device which solved the difficulty.
CE Online Diary Usability Testing – Task Order 

#26 

4-22 

 

Rating Diary Use by Device

Respondents were asked to provide a relative rating of how difficult or easy it was to use the online 

diary to enter expenses. Ratings were asked separately for desktop devices and mobile devices. 

Ratings were only collected for devices used by the respondent. Use was defined as entering at least 

one expense with the device type to be rated. Table 4-14 shows the ease of use rating distributions 

for each device type.

Table 4-14. Ease of use rating for desktop and mobile devices

Ease of Use Rating 

Desktop/Laptop 

(n = 51) 

Mobile (smartphone/tablet) 

(n = 32) 

Very Easy 62.8% 65.6%

Easy 35.3% 28.1%

Neither -- --

Difficult -- 6.3%

Very Difficult 2.0% --

Nearly all respondents rated both device types as easy or very easy to use. One respondent 

negatively rated the desktop version while two respondents negatively rated the mobile version. 

Reasons for the negative ratings are summarized below.

Desktop/Laptop

 One respondent attempted to use a laptop, but system froze after entering expenses. All 

expenses were instead entered using an iPad with no similar issues.

Mobile Device

 One respondent reported using a smartphone was difficult due to the smaller screen relative 

to their desktop device. Respondent also preferred viewing entry and summary fields on one 

screen versus the two screens for the smartphone.

 Respondent attempted to use an iPad mini, but the instrument froze after entering expenses 

and did not display expenses entered.

When providing support for the rating given, respondents generally used words or phrases that 

referred to the layout for the desktop version. For the mobile version, respondents used words or 

phrases that more often referred to the convenience or mobility of the device.
CE Online Diary Usability Testing – Task Order 

#26 

4-23 

 

Previously it was noted that three respondents reported they found it difficult to select a category for 

their expense. However, five respondents reported finding the categories helpful or liked the 

categories available to select. How helpful the expense categories are viewed is expected to be 

associated with how typical they are of the category description. One example is from a respondent 

who noted that the help text classified gasoline under “All Other Products, Services, and Expenses,” 

but entered this under transportation expenses. Nearly all entries of “gas” or “gasoline” were 

entered in the category for “Transportation Expenses.” This mismatch may have contributed to 

reported confusion.

Device Preference

Respondents who entered expenses using both desktop and mobile devices were asked which device 

they preferred when accessing the online diary. Most respondents only used one type of device, but 

a small number used both. Preferences were collected for 11 respondents2and were fairly evenly 

split between desktop and mobile devices. Four respondents reported preferring desktop; six 

preferred mobile, and one expressed no preference.

Reasons provided by respondents for preferring the desktop version were due to the larger screen 

size (all screens/fields can be viewed at once) and availability of a physical keyboard. Reasons for 

preference of mobile (for these respondents’ smartphones) were often based on lifestyle behaviors 

associated with smartphones. These respondents reported they already do “everything” on their 

smartphone, or already had their smartphone with them so it was easier to use for entering expenses.

When asked which device these respondents found easier to use, this often matched the stated 

preference. Only two respondents differed. One expressed that both were equally easy to use, while 

one respondent (preferring the smartphone) said the desktop would be easier due to the larger 

screen.

 

2 Of the 11 debriefing participants, 10 were main diarists, one was another household member. A review of web diary 

data shows that 13 main diarists and 7 other household members used both device types.
CE Online Diary Usability Testing – Task Order 

#26 

4-24 

 

Reasons Mobile Not Used

All respondents recruited (main diarist) were required to own a smartphone to participate in the 

usability test. During the placement interview, respondents were informed they could use multiple 

devices to access the online diary. A number of respondents never used a smartphone to enter any 

expenses. Table 4-15 shows a list of reasons provided by respondents for not using a smartphone to 

enter any expenses.

Table 4-15. List of reasons given for not using a mobile device to access the online CE Diary 

(counts exceed respondent count due to multiple reasons provided)

Reason for not using mobile device 

Count 

(n = 44) 

Larger screen / keyboard or mouse availability 19

Already set-up on laptop / did not want hassle of setting up 

on smartphone

5

Too difficult with password / couldn’t remember password 4

Doesn’t like using phone for detail tasks (difficult to type) 4

Doesn’t like apps (thought link was an application) 2

Did not want to use data 2

More comfortable with laptop 2

Saved receipts to use with laptop 2

Already using laptop for other tasks 2

Instrument froze during use 1

Few expenses / no chance to use 1

Wanted to stick with one device 1

Didn’t think to use smartphone 1

Online access unavailable on device 1

Phone memory full so would not load 1

Thought would be too difficult (screen size) 1

Thought would be too difficult (typing) 1

The most prevalent reason for not using a smartphone to enter expenses is a general preference for 

the larger screen provided by desktop devices, or the availability of a physical keyboard or mouse. 

Two additional respondents felt that a lack of these features makes using the smartphone too 

difficult. The next most frequent reason provided reflected that once respondents started on one 

device they stuck with it. Two of the five respondents in this group allowed the interviewer to create 

a link on their smartphone, but still chose not to use it. These reasons are followed by difficulties 

recalling the username or password, or a preference for completing the task on larger devices. Two 

respondents noted that they used their phone to make a note of expenses while they were out, that 

were later entered using a desktop device.
CE Online Diary Usability Testing – Task Order 

#26 

4-25 

 

Overall, the majority of the reasons provided by respondents suggest perceptions that the online 

diary task may be difficult to complete on a mobile device. This was not a barrier to participation as 

these respondents completed the task on desktop devices. It does suggest that not enough may have 

been done during the placement interview to show how the mobile version of the online diary 

operates.

Feedback on Specific Diary Functions

Respondents were asked about specific functions of the online diary. These included the use of help, 

reviewing a summary of expenses, editing expenses, and any difficulty associated with navigating the 

diary. This section reviews the feedback collected from respondents.

Informational help links

Help or additional information about a category or response field was available through hyperlinks 

indicated by ‘i’. Whether or not these were used was not available in paradata and use is based on 

respondent reports. A total of 73 respondents we’re asked about the use of help links. Less than half 

(43%) reported ever using these. Nearly all (accessing help links) felt the information within the help 

links was helpful, with one dissenting respondent. The dissenting respondent reported the help link 

was selected out of curiosity and not needed.

Of those who did not report ever using the help links, Table 4-16 gives a distribution of the reasons 

the links were not used.

Table 4-16. List of reasons given for not using informational help links

Reason for not using information links 

Count 

(n = 41) 

No need for help or additional information 24

Did not notice the informational links 11

Did not know that the ‘i’ was a link to more information 6

Most respondents reported they felt they did not need any additional help, while many respondents 

were not aware of the informational links. No respondent reported unintentionally selecting the help 

links, or report that the links interfered with the expense reporting task. When the informational 
CE Online Diary Usability Testing – Task Order 

#26 

4-26 

 

links were used, most respondents reported doing so to help determine which category to place an 

expense.

Summary of expenses

A summary of expenses was available, organized by expense date. Respondents were asked if this 

was helpful to have. A large majority (90%) of respondents found this to be helpful, only seven 

respondents stated it was not helpful to them, five of the seven reporting they never reviewed the 

expense summary.

Table 4-17 provides a list of responses given for how the summary list of entered expenses was 

helpful or used by respondents (includes seven respondents who did not find the summary helpful).

Ten respondents who did not provide comments on how the summary list was helpful are excluded 

from Table 4-17.

Table 4-17. List of responses given for how summary list was helpful or used (total exceeds 

respondent count as some respondents offered more than one)

How summary list was helpful or used 

Count 

(n = 66) 

Used to confirm expenses were entered 22

Used as a reminder of what expenses have not been 

entered

12

Increased awareness of spending 11

Used to verify expenses were correct 9

Used to confirm expense date is correct 2

Not used 10

The summary list of expenses was most commonly used as a way to confirm that an expense was 

entered, but was also frequently used to determine what days or expenses had not been completed. 

Eleven respondents reported that the summary table increased their awareness of how much they 

spend, which was viewed positively. Four respondents volunteered that they would like to see a total 

for each day summarizing how much their spending was for that day.

Editing expenses

Feedback on the process for editing expenses was collected from 50 respondents who reported ever 

editing any expense. All respondents reported the process for editing expenses was easy. Many 
CE Online Diary Usability Testing – Task Order 

#26 

4-27 

 

respondents noted that the edit icon is recognizable by anyone familiar with computers. There were 

a few specific comments indicating confusion or frustration with the process.

 One respondent did not realize the icon had to be clicked/selected to edit an expense. The 

respondent expected to be able to click on the expense line in the summary screen.

 One respondent stated it was not possible to change the category while editing, so the 

expense was instead deleted and a new expense entered. The respondent specifically stated 

the save button would not work. Several other respondents reported successfully changing 

the expense category when editing. When the category is changed new fields may show 

depending on the category that will require completion. The save button will not be 

functional until all fields are completed. It is likely this was not recognized by the 

respondent.

 One respondent had to change the date for a series of expenses that were entered. The 

respondent reported it was tedious to have to change each one separately and would have 

preferred a way to edit multiple expenses at the same time.

Navigating the online diary

Respondents were asked how difficult or easy it was to navigate the online diary. Navigation was 

defined as knowing where you are within the online diary, or the ability to easily determine the next 

step or action to be taken. Table 4-18 shows the distribution of ratings provided by 74 debriefing 

respondents.

Table 4-18. Distribution of online diary navigation ratings

Ease of navigation rating 

Count 

(n = 74) 

Very Easy 63.5

Easy 31.1

Neither 4.1

Difficult 1.4

Very Difficult --

Overwhelmingly, a large majority of respondents found navigating the online diary easy to do (95%). 

The four respondents giving lower ratings offered the following reasons.
CE Online Diary Usability Testing – Task Order 

#26 

4-28 

 

 A respondent wanted the date selected in the summary field (desktop, left side) to become 

the date selected in the entry field (desktop, right side).

 A respondent said it was difficult for her kids (two age 15-17) to report they did not have 

any expenses for the week. Suggesting indicating “no expenses” daily would be helpful.

 A respondent stated entering expenses was “busy work” with the need to itemize each 

expense.

 A respondent reported logging in was difficult and entering the first expense was confusing,

but the process became easier with more use.

Respondent Suggested Changes

At the conclusion of the debriefing, respondents were offered an opportunity to suggest changes 

they would like to see in the online diary. These were specified by version (desktop or mobile), but 

often related to the instrument generally. Respondents were not prompted to think about any aspect 

or function of the diary. A total of 36 of the 74 debriefing respondents offered statements that they 

would not make any changes to the online diary.

The list of suggested changes is provided below, by online diary presentation (desktop or mobile) 

specified by the respondent. Counts of respondents suggesting the change listed are provided in 

parentheses, but will not reflect the true number of respondents favoring the suggestion.

Desktop version – suggestions to function, design, or usability

 (3) Allow users to create their own username

 (2) Make password easier (one comment that password requirements are more secure than 

respondent’s bank password)

 (2) Add a field to indicate no purchases were made by day/date

 (2) Add a summary dollar total for each day to the expense summary

 (2) Add confirmation that expense entry was saved

 (1) Improve search function to function more like “Google” – that is allow for close 

matches if unable to recall exact description entered

 (1) Clear the date field after saving an expense so that it does not default to the previous date 

used

 (1) Move the entry field for cost of alcohol closer to the total expense box
CE Online Diary Usability Testing – Task Order 

#26 

4-29 

 

 (1) Add function to edit multiple expenses at the same time (e.g., to change date)

 (1) Make the font for the expense categories larger or bold

Desktop version – suggestions related to new or advanced functions

 (3) Add functionality to scan barcodes

 (2) Add functionality to scan receipts

 (2) Add a daily reminder to enter expenses (e.g., text or email)

 (1) Add a reminder to itemize expenses

Desktop version – suggestions related to content / task requirements

 (2) Add a field to indicate item quantity when multiples of the same item were purchased

 (1) Would like to be able to include tax for purchases

 (1) Would like more detail (in help) for “Other” category

 (1) Would like to be able to enter a total receipt rather than itemize

 (1) Add more category choices

 (1) Add category for pet expenses

Changes suggested for mobile version – bolding indicates changes suggested to both device 

presentations.

Mobile version – suggestions to function, design, or usability

 (3) Allow users to create their own username

 (2) Make password easier

 (2) Increase font size for mobile

 (2) Add confirmation that expense entry was saved

 (1) Make help icon more noticeable

 (1) Add clarification to indicate the function of help icons

 (1) Add a way to show what other icons, such as search and filter do

 (1) Change function so date does not need to be entered twice (e.g., on screen listing dates, 

then again in expense entry fields)

Mobile version – suggestions related to new or advanced functions

 (1) Add functionality to scan barcodes

 (1) Add functionality to scan receipts
CE Online Diary Usability Testing – Task Order 

#26 

4-30 

 

 (1) Add functionality to take picture of receipts

 (1) Would like to see an app option for online diary on a smartphone

Mobile version – suggestions related to content / task requirements

 (1) Would like to be able to enter a total receipt rather than itemize

Summary

This section focused on respondent reactions to using the online diary. Overall, respondents had a 

positive view of the online diary and offered comments indicating the online diary was easy to use. 

The layout of the desktop version appeared to be easier to use and resulted in less confusion in 

determining that an expense had been entered. The most notable difficulty reported by respondents 

was due to difficulties remembering or entering their password. Respondents had a number of 

suggestions for changes to the diary, but very few resulted in a consensus relating to the function of 

the diary. Most suggestions for changes were for improving the process for accessing the diary or 

adding technological enhancements. The main conclusions drawn from this section are listed below.

1. The online diary was generally well liked by respondents and the process for entering 

expenses was generally intuitive to respondents. Despite this, there were a number of 

difficulties reported suggesting minor changes to the diary. Aside from logging into the diary,

most difficulties related to the need for feedback that an action occurred (e.g., confirmation 

an expense was saved).

2. The device respondents chose to use was favored for different reasons. Desktop type device 

users tended to favor the layout and physical interface, while mobile users tended to favor 

the convenience and mobility of these devices. Respondents who chose not to use a mobile 

device did so due to a preference for physical features, such as screen size, or perceptions 

that a mobile interface would be too difficult.

3. Logging into the online diary was reported the most difficult part of the diary. Problems 

were generally associated with setting up the password, but extended to ability to remember 

login credentials. This may have prevented many respondents from using a mobile device to 

access the online diary.

4. The use of help links was low, but awareness of help links and what they do was reasonable. 

However, more should be done to increase the saliency of informational help links.
CE Online Diary Usability Testing – Task Order 

#26 

4-31 

 

5. The expense summary table was viewed as very helpful and informative to respondents. Any 

changes suggested centered around providing additional information, such as displaying daily 

and total summary costs.

6. Respondents offered a number of suggestions for changes to the online diary. The most 

common was simplifying the login process or password requirements. Another notable 

suggestion and reported difficulty is the need for feedback that an expense was entered in 

the mobile version. Additional suggestions focused on expanding functionality through other 

technologies, such as bar code scanning.
CE Online Diary Usability Testing – Task Order 

#26 

5-1 

 

 

In this section we present recommendations based on respondent behaviors observed as a result of 

reviewing the diary data, or direct feedback from respondents. We discuss changes to procedures 

and the online instrument. We first discuss direct or immediate changes that should be made, then 

follow with new technologies that could be integrated which will require additional testing and 

research. 

5.1 Recommended Changes 

Placement interview procedures

Relying on the main diarist or household respondent to recruit and get other household members to 

participate excluded about one-third of other household members. There is no clear change to easily 

overcome reliance on the main diarist. BLS should conduct testing into different approaches for 

reaching and gaining cooperation from other household members. Some thoughts on this are 

offered below.

1. Obtain commitment from the main diarist. Ask the main diarists if they are willing to get 

other household members started on the online diary. If they are willing to do so, explain the 

expectations, role, and responsibilities of the main diarist. Review literature on negative value 

wording of cooperation request (e.g., see Tourangeau and Ye, 20093). An example of 

implementing this would be informing the main diarist that the information they provide will 

be less valuable without completed diaries from other household members.

2. Collect contact information for other household members. If the main diarist is unwilling to 

get other household member participation, collect contact information (email, respondent 

names) and best time to contact. This information can be used so that interviewers can 

contact other household members and recruit them directly. 

 

3 Tourangeau, Roger, and Cong Ye. 2009. The framing of the survey request and panel attrition. Public Opinion 

Quarterly, 73(2), 338-348.

Recommendations 5 
CE Online Diary Usability Testing – Task Order 

#26 

5-2 

 

 

3. Incentivize main diarist recruitment efforts. The incentives provided in the usability test 

were provided as a token of appreciation for participation. Allocate incentives to reward 

successful recruitment by the main diarist. If the main diarist can get other household 

members to access the diary, data showed this was predictive of entering expenses. This 

should be tested to determine of the incentive improperly motivates respondents to use the 

login credentials of other household members to obtain the incentive.

Many respondents avoided using mobile devices due to perceptions the mobile version would be 

difficult to use. These were communicated directly, or veiled as preferences for larger screens or 

other physical attributes. During the placement visit, procedures should be developed for the 

interviewer to provide and engage the main diarist in a demonstration. The interviewer should be 

equipped with a mobile device and demonstrate the interface and ask the main diarist to enter an 

example purchase.

The purpose of this is not to drive respondents to use mobile devices, but to remove any barriers to 

mobile device usage. If mobile devices improve data through timeliness, potentially reducing 

forgetting expenses (especially those without receipts), their use should be encouraged.

Design, layout, and function of the online diary

Login or diary access procedures should be simplified. This was the most common barrier to 

accessing the diary and prevented the use of mobile devices for some respondents. To the extent 

possible, BLS should simplify the username and password requirement for accessing the diary. At 

the least, investigate allowing respondents to create their own username and password with limited 

restrictions (e.g., exclude the need for special characters). Alternatively, consider allowing 

respondents to save usernames and passwords to devices that are not shared, or using simplified 

single logins (username only, or unique PIN).

The layout for desktop devices worked very well for respondents and there are no recommended 

changes to the desktop layout. Respondents offered a number of suggestions, but many are 

idiosyncratic and offer no discernable improvement or are not possible (for example, moving the 

cost of the alcohol field, or allowing tax to be included). We recommend two general changes that 

focus on adding content to the summary of expenses and improves feedback for mobile users.
CE Online Diary Usability Testing – Task Order 

#26 

5-3 

 

 

Many respondents reported they liked that the summary list of expenses offered a tracking of their 

expenses, but would have liked to see a summary of spending for each day and overall. It would 

require only a minor addition to add a summary total of spending for each reporting day. Currently 

each day selected provides a summary count of entries. A dollar total could be added to this.

However, if reporting features are favored by respondents this can be taken a step further. Include 

functionality to provide summary statistics that shows or charts dollar amounts by day or by 

category (expense type). This may improve reporting as respondents may be more willing to access 

the diary to view statistics, or summary statistics may quickly show missed expenses. For example, if 

no entries are shown for food away from home, but the respondent recalls eating out.

For the mobile version, active feedback should be built into the instrument. When an expense is 

saved, the entry fields clear and default to the last date and category used. For many respondents it 

was not clear that the expense was saved. This could cause duplication of entries. In the desktop 

version respondents see confirmation the expense was saved. In the expense summary, the expense 

date is expanded (if not already), the expense added, and the expense temporarily highlighted. A 

pop-up message is recommended that confirms the expense was saved for the mobile version. A 

settings function is also recommended where respondents can deactivate pop-ups once they become 

familiar with the expense entry process.

Another recommendation to reduce confusion with saving in the mobile version is to provide 

feedback when respondents deviate from the designed flow of the instrument. For example, when 

respondents use the “back” button of the browser, rather than navigation buttons within the 

instrument (back, clear, and save). The action can be detected and a pop-up with warning text that 

data will be lost and the correct action(s) to take.

A final recommendation is for the wording and implementation for collecting when respondents

have no expenses to report. This is collected in the current online diary on the last reporting day or 

later. There were some recommendations by respondents to provide a way to indicate no expenses

on a more frequent basis. This should be tested, as there is concern that respondents may use this as 

an easy way to complete the reporting task without entering any expenses. There is benefit to 

collecting this information every day if it encourages respondents to access the diary on a daily basis. 

The wording for collecting whether there were any expenses was potentially confusing due to the 

negative wording: “It looks like you have not reported any expenses. Did you not have any expenses 
CE Online Diary Usability Testing – Task Order 

#26 

5-4 

 

 

to report during your reporting period? (yes/no).” Positive wording will be less confusing and 

instead focus on whether the respondent has any expenses to report – discouraging false reporting.

 You have not reported any expenses. Have you had any expenses since <1st reporting day>?

If the response is “no – have not had any expenses,” this should be followed with a confirmation 

message.

 You have recorded that you do not have any expenses to report for this period. If later, you 

recall any expenses, you may enter them at any time.

Note that the first message can be modified to be a daily message, but replacing the display text in 

the angle brackets with the last day expenses were reported, or with “today.” The confirmation 

message would equally need to be modified to reflect no expenses were reported for a specific 

reporting day.

Future technologies

A number of respondents stated they would like the ability to scan barcodes for products or take 

pictures/scan receipts. Respondents were not specifically asked to what extent these functions 

would be desirable. There may be a preference for these greater than indicated by the few 

respondents reporting this. BLS should investigate additional technological functions available 

through smartphones that would improve data quality or reduce respondent burden. A review of 

technological functions that are currently available is provided in the next section.

5.2 Future Enhancements to an Online CE Diary 

Overview

The online diary reporting system for the Consumer Expenditure Diary Survey administered in this 

usability test utilizes the latest web programming technologies and best practices that are currently in 

widespread use. This allows the diary to be accessible across multiple platforms (operating systems) 
CE Online Diary Usability Testing – Task Order 

#26 

5-5 

 

 

and devices. Examples of this approach include application of a responsive web design and use of 

modern style elements in the user interface. A responsive website optimizes the page layout of the 

elements based on the screen size of the device used by a respondent. While the elements, 

navigation, and flow of the instrument are essentially the same, the smartphone and desktop users 

see different layouts for a better user experience. The diary employs style elements and user 

interactions that reflect modern design practices and programming tools as well as what respondents 

see and interact with on other consumer web sites. This essentially leverages knowledge gained from 

interacting with other web sites where motivation to learn how to navigate is higher.

The online diary used a proven open-source software development tools that facilitate continued 

modernization and integration of future technology advancements. The Bureau of Labor Statistics 

(BLS) has a strong interest in future enhancements that can reduce respondent burden or improve 

data quality. Ideally, any future enhancements would target both reductions in burden and 

improvements in data quality.

We list a number of future enhancements that merit further consideration in future development 

efforts and testing. The enhancements we list all focus on features associated with mobile devices 

(e.g., tablets and smartphones such as iOS and Android based devices). The reasoning for this is that 

mobile devices are easier to manipulate, have some unique capabilities, such as texting, and are 

mobile. 

BLS has determined that any future enhancements need to be accessible with the online diary and 

cannot require a native “app” that would be downloaded to the respondent’s device (i.e., 

smartphone or tablet). There are two reasons for this: 1) there is a belief that respondents are 

unwilling to download an app (that is perceived as undesirable) on their device; and 2) an app will 

not meet BLS security requirements for storing and transmitting respondent data.

We believe that future enhancements that access features, other than web access, would perform 

best within the structure offered by a device app. Any web instruction or request to use, for 

example, the device’s camera would interfere with security restrictions within the device and require 

additional user interaction to grant permission. A native app only needs to be granted access once, 

when installed, has more control (e.g., over image size and quality determination) and can be 

integrated with other services.
CE Online Diary Usability Testing – Task Order 

#26 

5-6 

 

 

The reasons against using an app stated by BLS are anecdotal and surmountable. There is likely a 

proportion of respondents who are unwilling to download an app to their smartphone. Possible 

reasons for this are privacy concerns, suspicion of the government, or simple undesirability. 

However, there is a dearth of empirical evidence supporting that this is a majority of respondents 

with access to a mobile device, or even what proportion of respondents would refuse an app. Apps 

are commonly used and accepted by mobile device owners. Examples are retail applications such as 

Amazon, Walmart, and Target; or product access, such as Netflix, HBO GO, or health and fitness 

apps that communicate with other products (e.g., Fitbit, ShockBox, Nike+). Westat believes an app 

would be widely accepted by respondents. In a project conducted by Westat to track food purchases, 

National Food Study (FoodApps), 62% of respondents completed their food log using an app. For 

the FoodApps project, respondents could use their own device or one provided by Westat. Data is 

not available for the proportion of respondents that chose to use their own device, but the high 

percentage using the food log app is encouraging.

The second reason provided by BLS relates to data security. The FoodApps project had similar 

concerns. Additionally, FoodApps wanted to avoid the cost and effort of having to support multiple 

implementations of the food logs (food diary) – one for PCs, as well as versions for a variety of 

mobile device types. Security concerns and uniform web design were addressed by integrating a web 

browser into the native app so that data is not stored by the respondent’s device or any third parties. 

The design uses an app with an integrated web browser. The app accesses the camera for scanning 

receipts or barcodes. The web browser connects to the online food log where all data is stored and 

the respondent enters supplemental information not provided by the receipt or barcode.

We next describe potential functionality (enhancements) that can reduce respondent burden or 

improve data quality.

Technologies

1. Features using the smartphone camera

There are several functions that can be implemented with a camera on a mobile device. These 

include photos of a product, service, or receipt, barcode scanning, and item recognition using 

machine learning.
CE Online Diary Usability Testing – Task Order 

#26 

5-7 

 

 

Photos of a product, service, or receipt / using photos as an expense reminder. There are a 

number of methods for implementing this. One is through the online CE Diary instrument. 

Allowing a website to access a device’s camera is a fairly recent development and will not work on 

older versions of many device’s operating systems. Making sure the device’s operating system is upto-date will be important. Programming can be added to the CE website that will request access to 

the device’s camera. This allows the respondent to create a simple reminder of expenses that need to 

be entered at a later time.

There are some limitations with this approach. The first is that this may be viewed as a security risk 

by the device and a message will be displayed to the respondent. The respondent will need to grant 

permission to allow the website access to the camera. This may be avoided with some devices using 

a secure certificate connection (https). Permission would still be required, but not asked as often. 

The second limitation is that web access to the camera offers less control over the image file size 

and may result in large files that use data or fail to load to the website with weak connections.

Accessing the camera through a native app addresses these limitations when accessing the camera 

online. Permission is granted to use the camera when the app is installed or upon first use. The app 

can also reduce the file size of the image or store the image locally, reducing the need to upload the 

image.

Barcode scanning. Barcode scanning requires the same camera access and has the same web access 

limitations as taking photos to serve as a reminder. However, barcode scanning requires additional 

code to read the image (detect the barcode). It is possible to implement this functionality through an

online instrument without native app support. Westat has tested this and found it to be functionally 

possible, but unreliable. The issue with reliability is that the act of acquiring an image, then 

detecting/reading a barcode and submitting the result to a database are separate serial processes. 

The barcode is read after the image is taken, but may not have sufficient quality to allow 

interpretation. The low reliability can result in respondent frustration and abandonment of barcode 

reading.

Barcode scanning can be implemented through a native app where the process and reliability are 

improved as a seamless one-step process. The app controls the image and the barcode detection. 

That result is then sent to a database which provides the detail of the barcode. This information is 
CE Online Diary Usability Testing – Task Order 

#26 

5-8 

 

 

then populated in the web application. This approach is currently in use by Westat in the FoodApps 

project. Information from the product barcode can populate the purchase description (a potentially 

difficult field for smartphone device respondents since text entry is required). The barcode may be 

able to be mapped to BLS expense categories, although this may take some initial investment on the 

part of BLS in developing the mapping process or database. This has the potential to greatly reduce 

the amount of information the respondent is required to enter and improve purchase classification.

Barcode scanning requires access to a database of item or UPC barcodes (see examples below). 

There are a number of free-to-search providers; however, BLS should investigate how accessible and 

complete these are. Barcode databases appear to be most complete for grocery or food items, but 

some (accessible through a fee) are more complete. It should also be noted that for some retail 

chains, store branded products may have a barcode unique to that store and not registered in any 

UPC database. Some database providers allow adding products to the database improving it over 

time by collecting this information from respondents.

Figure 5-1. Example of free-to-search barcode scan for automotive part
CE Online Diary Usability Testing – Task Order 

#26 

5-9 

 

 

Figure 5-2. Example of free-to-search barcode scan for big-box store bulk chocolate bars

Item detection / machine learning. This involves using the camera to recognize and identify 

items. This is done by applying machine learning though large volumes of images to “learn” how to 

identify certain items. This would be most useful applied to grocery or food items that lack a 

barcode (e.g., fruits, pastries, vegetables, etc.). Westat is currently part of a proposed grant 

developing this functionality for identifying food items with encouraging results. This would require 

a native app, but could tie in with a web browser embedded within the app.

The more images or items the process is exposed to the better it would become. When initially 

deployed the process would have pre-learned a number of common items, but as more and more 

respondents used the item detection, the more it would learn and extend to a wider range of 

purchases.

A similar process could be extended to purchase receipts. This would be most successful for receipts 

from large retail chains. While receipts are not standard between stores, machine learning could be 

applied to extract the purchase date, identify the number of items expected from a receipt and 

extract the cost for that item and possibly account for appropriate discounts. Since receipts often 

lack clear detail of the item description, or detail that is easily interpreted, the respondent would be 

required to select the category and enter a description. The main benefit from this approach is that it 
CE Online Diary Usability Testing – Task Order 

#26 

5-10 

 

 

would prevent respondents from entering an entire receipt (of multiple items) as one purchase and 

ensure all items are entered by walking them through the receipt image.

Image texting. This approach is best suited to a web only instrument without any app support. 

However, respondents would be required to register their (smart)phone number. This would work 

by allowing the respondent to take a picture and text it to a number connected to the CE Diary 

instrument. By registering their phone number the web instrument can store this image in the 

respondent’s diary. The next time the respondent accesses the web diary he or she will be presented

with the reminder images to enter expenses. Westat has developed this capability, but it has not been 

fielded for any project.

2. Features using the smartphone microphone

Functions utilizing the microphone in a smartphone are more limited compared to those using the 

camera. The most obvious function is using the microphone for speech to text entry.

In the CE Diary, respondents are required to provide an item description and item cost which 

requires text or numeric entry. These fields can be difficult for smartphone users as the keyboard is 

much smaller and prone to error (or more colloquially, the fat finger problem). The microphone on 

the smartphone can be used to translate the respondent’s speech to text entry.

Web programming can be utilized to access the device’s microphone; however, this will encounter 

the same security issues as with the camera. Respondents will see a security alert or a request to 

access the microphone, even if they initiate the microphone request. For example, a potential design 

is to place a microphone icon next to open fields so respondents can choose to use the keyboard or 

speech to text.

Speech-to-text functions do not solely rely on the device’s microphone. Additional software or 

service is required to convert the speech (usually referred to as “utterance”) to text. This would have 

to be added to the online CE Diary instrument.

Speech-to-text functionality could be implemented natively through an app, where a web browser is 

embedded within it and native access to the microphone is allowed. This keeps all data within the 

online instrument while avoiding multiple requests to access the microphone. Using an app would 
CE Online Diary Usability Testing – Task Order 

#26 

5-11 

 

 

avoid the need for additional software to support speech to text in some devices. For example, 

Apple iPhone devices have this functionality built into the device. However, this may not comply 

with BLS security or confidentiality requirements. Native speech-to-text functions on Apple iPhone 

devices are processed through Apple servers.

Westat has not used speech to text for web instruments, but has tested its use for the Bureau of 

Justice Statistics as part of the National Crime Victimization Survey. This was tested as part of a 

telephone Interactive Voice Response (IVR) system. Westat found this process to be possible, but 

there were some limitations. The most significant limitation was in recognition accuracy. Speech to 

text operates best when it has a sampling of the respondent’s voice, which is often not possible in 

survey type settings. Westat observed that in many cases for binary yes/no questions an utterance of 

“no” was recognized as “yes” with high confidence. When the dictionary of words is greatly 

expanded, as it would be for the CE Diary, this can result in wider errors. Westat also noted that 

ambient noise, such as wind, crowds/other people, or subway background noise, can make speech 

recognition difficult. For the CE Diary, if respondents are more likely to use this while outside the 

home, or immediately after a purchase, this could be common and problematic. Another issue is 

social perception, where respondents may be reluctant to dictate purchases into their phone in 

public surroundings.

Speech recognition is currently seeing growth in home devices termed “home assistants.” The most 

popular examples are the Amazon Echo and Google Home. It is likely that the technology 

surrounding speech recognition is likely to see rapid innovation and respondents will become more 

accustomed to interacting with these. Westat does not view speech recognition or speech to text as a 

technology that BLS should pursue at this time, but as technology improves will be worth 

monitoring and pursuing at a future time.

3. Features using the smartphone GPS, or geolocation

A respondent’s location can be determined through an online instrument such as the CE Diary. 

Location is great for creating a reference or marking where an event took place, but it is difficult to 

see how this would benefit the CE Diary. Location information could be valuable paradata to see 

when and where respondents are accessing and using the diary, but provides little if any information 

about the purchase itself.
CE Online Diary Usability Testing – Task Order 

#26 

5-12 

 

 

Access to location services will be viewed as a security risk and require the respondent’s permission 

every time accessed or requested. This is a common theme throughout this review that while web 

tools are the most flexible solution in terms of instrument access and accessibility, any programming 

that requests use or access to a device’s (smartphone) tools will alert the user and require user action 

(acceptance or refusal). Native apps avoid this issue since the user provides permission when the app 

is installed, or upon first use.

A native app also provides greater functionality and application of GPS for geolocation. A native 

app as recommended by Westat would include an embedded web browser so that the CE Diary 

remains separate and data is not stored on the user’s device. An application of geolocation for the 

CE Diary would use “geofencing.” Geofencing is a process where a response is triggered when a 

user enters or leaves a specified area. For each CE sample, high traffic retail areas can be mapped so 

that if a respondent enters these the app triggers a reminder for the respondent to enter any 

expenses. A more general application would be any time respondents have left their home and have

not accessed the diary for that day, triggering a reminder.

Westat has widely used GPS for field interviewers as a method of validating interviews. For example,

ensure the location recorded by the interviewer’s device matches the sample address. 

The benefit to BLS of geolocation is as a method to improve reporting through tailored reminders. 

These reminders are based on knowledge that the respondent is not at home, or based on 

knowledge that the respondent is in an area where expenditures are likely to occur. This will require 

determining two things. The first is the level of effort required to map retail areas in a sampled 

geographic area. The second is how receptive respondents are to having their location tracked.

Summary

The web-based online instrument developed for the CE Diary is a great solution for BLS as it 

provides an instrument accessible across multiple devices and platforms. BLS is interested in 

expanding the functionality of the online diary by taking advantage of new technologies or features 

generally accessible with today’s mobile devices – generally smartphones.
CE Online Diary Usability Testing – Task Order 

#26 

5-13 

 

 

Westat has summarized how three common features, the camera, microphone, and GPS 

(geolocation) can be applied to the online CE Diary. The primary interest of BLS is in how these 

could be applied though a web instrument and not a native app. Native apps are of concern to BLS 

due to potential reluctance by respondents to download an app, and lack of security compliance with 

BLS standards. Westat believes there is a lack of empirical evidence to support respondent 

reluctance, and that issues addressing security can be overcome with hybrid app and web browser 

solutions.

While web programming has advanced greatly to the degree that all the features described in this 

document can be implemented through a web tool, there are limitations that will affect the 

respondent’s experience. Web tools that access a respondent’s device require additional permission. 

This can appear intrusive and burdensome when requests become repetitive. Westat has developed 

and implemented native app solutions that use a hybrid design with an embedded web browser. This 

allows access to native device functions, improving quality in the process and the respondent 

experience. This also preserves data security by keeping all supplementary data requests (e.g., access 

to a barcode database) and respondent data on the agency’s servers. 

Finally, while mobile devices offer many features, the most promising would appear to be those 

associated with the device’s camera. These have the potential to collect the most data relevant to the 

CE Diary and utilize a function that respondents are comfortable and familiar with.
CE Online Diary Usability Testing – Task Order 

#26 

A-1 

 

Appendix A 

Placement Interview Protocol 

Introduction: Placement Visit 

Thank you for agreeing to take part in this study. Let me tell you a little more about Westat and what 

you will be doing as a participant.

Westat is a company that conducts research on many topics, for different organizations. This 

particular project is for the Bureau of Labor Statistics. The Bureau of Labor Statistics conducts a 

study every year that asks people to record all of the purchases that all members of their household 

make for an entire week. This information goes into calculating the Consumer Price Index, which is 

the key measure of inflation in the U.S. Westat has been asked to test some different procedures to 

make it easier for people to report what they purchase.

As you may recall from the letter we sent and earlier discussions with our staff, we will be asking you 

to record all the purchases you make over the next seven (7) days. We also want other household 

members 15 or older in this household to record their purchases. Today I’ll go over the following:

 Set up access to the online purchase diary. If it is ok with you I can set up shortcut links on 

your desktop or laptop computer and on your smartphone. I’ll also leave instructions on 

how to do this in case the shortcut links are deleted.

 Provide you with a user guide. This is a reference for how to use and access the online 

purchase diary.

 Provide you with usernames and temporary passwords for you and other participants in this 

household. Each participant will be prompted to change their password the first time they 

log in. They will also be asked to answer a security question in case they forget their 

password.

 Your role and the role of other participants in your household in completing the online 

purchase diary over the next seven (7) days.

This is a new process and we are trying to figure out what works and what doesn’t. Your 

participation today will help us design a better system for people to enter their purchases in this 

study. By recording the purchases you make each day during the seven day reporting period, you’ll 

be helping us design a better system. We’ll also set up a time for a second visit after the seven day

reporting period. In that visit I’ll have some additional questions about your experience and how you 

used the system. I may also ask you about some of the purchases you reported. Do you have any 

questions?
CE Online Diary Usability Testing – Task Order 

#26 

A-2 

 

Before we get started, I want to let you know that everything we cover in our discussions and 

everything you record in the online diary will be kept private - only the few people actually working 

on the project will have access to the information you share with us. Participation is also voluntary, 

you may quit at any time. At the end of today’s visit, I’ll give you $20 as a token of our appreciation 

just for starting the study. At the end of the second visit, you’ll receive another $20 as a token of our 

appreciation. Each other participant in your household that participates will also receive $10 for 

entering their purchases for the seven day period.

CONSENT FORM: Here is a form I must ask you to look over and sign – it basically covers the 

points I’ve just gone over with you and indicates you have agreed to take part (ONE COPY FOR 

RESPONDENT, ONE COPY FOR US)

IF YOUTH 15-17 PRESENT IN HOUSEHOLD – IDENTIFIED IN THE SCREENER –

COMPLETE PARENTAL CONSENT.

BEGIN INTERVIEW:
CE Online Diary Usability Testing – Task Order 

#26 

A-3 

 

Respondent’s Role: Placement Visit 

For the seven day reporting period you will be the primary respondent. This means:

 You’ll enter any purchases you make in the seven day reporting period.

 You’ll remind other household members to enter their own purchases.

 We’ll ask you to help other household members who are not present set up their links to the 

online diary.

As mentioned we want you to record all the items you purchase for seven days. The reporting 

period for you and your household is:

Day 1: MONTH DAY, YEAR

Day 2: MONTH DAY, YEAR

Day 3: MONTH DAY, YEAR

Day 4: MONTH DAY, YEAR

Day 5: MONTH DAY, YEAR

Day 6: MONTH DAY, YEAR

Day 7: MONTH DAY, YEAR

As the primary respondent, you will be able to see if other participating household members have 

entered any expenses for each reporting day, but not what they entered. While you may know what 

these expenses are, the information is not shown to you (or others) in the online diary to protect 

their privacy. If you see that no purchases were entered for a given day, please remind them to enter 

any purchases they have made. However, it is normal to have a day where no purchases were made.
CE Online Diary Usability Testing – Task Order 

#26 

A-4 

 

Set-up: Placement Visit 

HOUSEHOLD COMPOSITION

Next I would like to set up a link to the diary website on your computer or mobile device, but I 

would first like to record all the household members 15 or older who live here. 

When we set up this appointment you reported that there are |____| participants including you in 

this household. Please tell me the first name, age and gender for each including you.

FIRST NAME AGE GENDER

RESPONDENT |___________________| |__|__| MALE FEMALE

PARTICIPANT 2 |___________________| |__|__| MALE FEMALE

PARTICIPANT 3 |___________________| |__|__| MALE FEMALE

PARTICIPANT 4 |___________________| |__|__| MALE FEMALE

What is the primary type of internet access for participants in this household:

 DSL

 CABLE (e.g., Comcast/Xfinity)

 FIOS (e.g, Verizon)

 Mobile (e.g., through a cellular carrier)

 Other

 Don’t know
CE Online Diary Usability Testing – Task Order 

#26 

A-5 

 

DIARY ACCESS

The link I would like to add to your computer or devices is just a shortcut link. It can be deleted at

any time and does not record any information. If more than one person uses the computer or device 

they can all use the same link. Each person will still be required to enter their assigned username and 

the password they created.

INTERVIEWER: RECORD EACH DEVICE THE RESPONDENT HAS AND IF A 

SHORTCUT LINK WAS SUCCESSFULLY ADDED. NOTE IF RESPONDENT REFUSED 

OR YOU WERE OTHERWISE UNABLE TO CREATE A LINK ON A GIVEN DEVICE.

PRIMARY RESPONDENT HAS DEVICE LINK SET COMMENTS

1. DESKTOP1/LAPTOP1 YES NO YES NO ______________________

2. DESKTOP2/LAPTOP2 YES NO YES NO ______________________

3. SMARTPHONE YES NO YES NO ______________________

4. OTHER: ___________ YES NO YES NO ______________________

PARTICIPANT 2 HAS DEVICE LINK SET COMMENTS

1. DESKTOP1/LAPTOP1 YES NO YES NO ______________________

2. DESKTOP2/LAPTOP2 YES NO YES NO ______________________

3. SMARTPHONE YES NO YES NO ______________________

4. OTHER: ___________ YES NO YES NO ______________________

PARTICIPANT 3 HAS DEVICE LINK SET COMMENTS

1. DESKTOP1/LAPTOP1 YES NO YES NO ______________________

2. DESKTOP2/LAPTOP2 YES NO YES NO ______________________

3. SMARTPHONE YES NO YES NO ______________________

4. OTHER: ___________ YES NO YES NO ______________________

PARTICIPANT 4 HAS DEVICE LINK SET COMMENTS

1. DESKTOP1/LAPTOP1 YES NO YES NO ______________________

2. DESKTOP2/LAPTOP2 YES NO YES NO ______________________

3. SMARTPHONE YES NO YES NO ______________________

4. OTHER: ___________ YES NO YES NO ______________________
CE Online Diary Usability Testing – Task Order 

#26 

A-6 

 

PROVIDE ASSIGNED USERNAME AND TEMPORARY PASSWORDS TO RESPONDENT.

 
CE Online Diary Usability Testing – Task Order 

#26 

A-7 

 

User Guide: Placement Visit 

REVIEW USER GUIDE WITH RESPONDENT.

As a reminder we want you and other participants to enter all the items you purchase each day. Also 

note that we are interested in the individual items you purchase or expenses you have and not a total 

of everything purchased at a specific location or store. For each purchase you will do the following:

 Enter individual items or expenses.

For meals, snacks and drinks away from home, such as lunch, you would enter the lunch 

meal as one expense and for the expense amount you would include any tax and tip.

For all other expenses, such as items from a grocery store, you would enter each item and 

the purchase price after any discounts of that item excluding tax or any tip. For example, 

bananas, bread, cookies, etc...

 Select the date.

If later during the reporting period you remember a purchase you made, but forgot to enter, 

you can still add it by selecting the appropriate date.

 Select the category that fits the item.

Select from the categories that best describe the item or expense.

 Describe the item or expense.

Provide a brief description of the item or expense. For some purchases or expenses you may 

not have a receipt. For example, a taxi ride, or snacks from a vending machine. It is 

important that you enter these. Please provide a descriptive label. Avoid vague terms, such as 

groceries, food, meal, toy, stuff, etc...

 Provide details for the expense.

This will vary for some expenses. For meals, snacks and drinks away from home we are 

interested in what type of meal it was (for example, breakfast, lunch, dinner, or snack) and 

whether any alcohol was included. For alcohol, report the type (beer, wine, other) and 

separately report the cost of the alcohol (however, remember to include the tax and tip in 

the total).

For food and drinks for home consumption we are interested in how the food was 

packaged.
CE Online Diary Usability Testing – Task Order 

#26 

A-8 

 

For clothing, shoes, jewelry, and accessories we are interested in how old the person is 

the item was purchased for.

For some of these and other expense types we are also interested in whether the purchase or 

expense was for someone outside of your household.

Finally, report the cost of the item. This is the cost after any discounts. For meals, snacks 

and drinks away from home this should also include tax and tip – note all other category 

types exclude this.

You can enter any purchases or expenses at any time during the reporting period. As noted when we 

set up your access you can enter these on any device. You do not have to be at home to enter this 

information, but internet access is required.

ASK RESPONDENT IF ANY QUESTIONS – REVIEW AND ADDRESS
CE Online Diary Usability Testing – Task Order 

#26 

A-9 

 

Online Experience and Closing: Placement Visit 

I have just a few questions before we finish.

1. How comfortable are you with using computers (such as a desktop or laptop) to access and 

use the internet? Would you say...

 Very comfortable

 Somewhat comfortable

 A little comfortable

 Not at all comfortable

2. How comfortable are you with using mobile devices (such as a smartphone or tablet) to 

access and use the internet? Would you say...

 Very comfortable

 Somewhat comfortable

 A little comfortable

 Not at all comfortable

3. Have you made any online purchases in the past 30 days? For example, you bought 

something over the internet and had it delivered to your house or picked it up at a store. 

Some examples are Amazon.com, Target.com, and Wallmart.com.

 Yes END

 No GO TO 4

4. (IF NO) Have you ever made any online purchases?

 Yes

 No

THANK RESPONDENT AND END INTERVIEW.

PROVIDE $20 AND HAVE RESPONDENT SIGN RECEIPT.

SET-UP APPOINTMENT FOR NEXT PICK-UP VISIT 8 TO 12 DAYS FROM TODAY’S 

DATE. ASK FOR OTHER PARTICIPANTS TO BE PRESENT AT THAT TIME IF 

POSSIBLE.

DATE: ____/____/_________

TIME: ____ : ____ AM PM
CE Online Diary Usability Testing – Task Order 

#26 

B-1 

 

Appendix B 

User Guide 
CE Online Diary Usability Testing – Task Order 

#26 

B-2 

 
CE Online Diary Usability Testing – Task Order 

#26 

B-3 

 
CE Online Diary Usability Testing – Task Order 

#26 

B-4 

 
CE Online Diary Usability Testing – Task Order 

#26 

B-5 

 
CE Online Diary Usability Testing – Task Order 

#26 

B-6 

 
CE Online Diary Usability Testing – Task Order 

#26 

B-7 

 
CE Online Diary Usability Testing – Task Order 

#26 

B-8 

 
CE Online Diary Usability Testing – Task Order 

#26 

B-9 

 
CE Online Diary Usability Testing – Task Order 

#26 

B-10 

 
CE Online Diary Usability Testing – Task Order 

#26 

B-11 

 
CE Online Diary Usability Testing – Task Order 

#26 

B-12 

 
CE Online Diary Usability Testing – Task Order 

#26 

C-1 

 

Appendix C 

Pick-up Interview Protocol 

Introduction: Pick-up Visit 

Thank you again for taking part in this study. I want to talk to you now about your experience 

completing the online diary.

Today I’ll review some of the things you entered and about your experience with the online system. 

I’ll also ask about things that worked well and things that did not work well for you.

As I noted in our first visit, this is a new process and we are trying to figure out what works and 

what doesn’t. Your feedback will help us design a better system for people to use in the future.

I also want to remind you that everything we cover in our discussions and everything that you have 

recorded in the online diary will be kept private - only the few people actually working on the project 

will have access to the information you share with us. Participation is also voluntary, you may quit at 

any time. At the end of today’s visit, I’ll give you another $20 as a token of our appreciation just for 

completing this interview.

[IF ANY OTHER HOUSEHOLD MEMBERS PARTICIPATED] I also have $10 for the other 

household members who tracked their purchases and expenses in the online system.

If it’s OK with you, I would like to audio record our conversation, just so that I can review it later 

when we’re analyzing all the interview data.

[CONSENT OBTAINED IN FIRST VISIT – HAVE FORM PRESENT IF NEEDED AND 

REVIEW AS NECESSARY]

BEGIN INTERVIEW:
CE Online Diary Usability Testing – Task Order 

#26 

C-2 

 

Diary Review: Pick-up Visit 

GENERAL FEEDBACK

 What is your overall impression after using the online diary?

[IF NEEDED] I’m interested in anything you liked about it as well as what you did not like 

about it. Also include any problems or things that may have frustrated you.

________________________________________________________________________

________________________________________________________________________

________________________________________________________________________

REVIEW DIARY ENTRIES

PROBE ON ANY OF THE FOLLOWING:

• DAYS WITH NO ENTRIES

o On Day |__| [DATE] you did not report any purchases or expenses, is that correct? 

Think about even small purchases, like candy or soda from a vending machine, or 

purchases where you may not have gotten a receipt, like gas or anything else.

• VAGUE DESCRIPTIONS (E.G., GROCERIES, FOOD, TOY, SHOP, MISC)

o For this purchase [CATEGORY] you described it as [DESCRIPTION], can you tell 

me more about this purchase?

 DETERMINE IF ENTRY INCLUDES MORE THAN ONE ITEM

 ASK RESPONDENT IF THEY USE THE ‘i’ FOR GUIDANCE

• EXCLUDED OR FORGOTTEN EXPENSES

o Are there any purchases or expenses that you did not include? These could be 

purchases or expenses that you did not think belonged or you had trouble entering.

o Can you think of any other expenses that may have been forgotten or overlooked? 

These could be small purchases that you may not have gotten a receipt for, or 

automatic charges. For example, highway tolls, or online subscriptions. It could also 

be purchases as gifts for someone outside your household. For example, candy, wine, 

etc. 
CE Online Diary Usability Testing – Task Order 

#26 

C-3 

 

• DIFFICULT TO CLASSITY

o For any purchases or expenses, were they any that you had a hard time finding the 

right category for?

 DETERMINE WHICH EXPENSES – HOW DID THEY DETERMINE 

WHICH CATEGORY TO USE.

 WERE ANY OF THESE EXPENSES LEFT OUT/NOT REPORTED?

USER GUIDE

o In our initial visit we provided you with a user guide. Did you use it at all?

 Yes

 No

o Why or why not? [IF REFERENCED ASK WHAT WAS HELPFUL ABOUT IT, 

OR IF THERE WAS INFORMATION THAT WOULD HAVE BEEN 

HELPFUL.]

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________
CE Online Diary Usability Testing – Task Order 

#26 

C-4 

 

USING THE ONLINE DIARY

• DESKTOP/LAPTOP

o You entered a total of |___| expenses using a desktop or laptop computer. Overall, 

how easy or difficult was it for you to do this?

 Very difficult

 Difficult

 Neither

 Easy

 Very easy

o What was [EASY/DIFFICULT] about the online diary?

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

• MOBILE DEVICE

o You entered a total of |___| expenses using [LIST DEVICES DETECTED]. 

Overall, how easy or difficult was it for you to do this?

 Very difficult

 Difficult

 Neither

 Easy

 Very easy

o What was [EASY/DIFFICULT] about the online diary when using these devices?

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________
CE Online Diary Usability Testing – Task Order 

#26 

C-5 

 

• IF AT LEAST ONE ENTRY BY MOBILE DEVICE

o Did you ever enter any purchases or expenses while away from home? For example, 

while out shopping, at a restaurant, or for any other reason.

 Yes

 No

o [IF YES] PROBE FOR DETAILS ON HOW WELL IT WORKED AND WHAT 

PURCHASES OR EXPENSES THEY ENTERED. (FOR EXAMPLE 

ENTERING A LUNCH MEAL)

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

o [IF NO] PROBE FOR DETAILS: E.G. DIFFICULTY, FORGOT PASSWORD, 

TOOK TOO LONG TO LOAD, DIDN’T WANT TO USE UP DATA, ETC...]

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________
CE Online Diary Usability Testing – Task Order 

#26 

C-6 

 

• IF USED BOTH

o You entered expenses using a desktop or laptop computer and using a mobile device. 

Which one did you prefer?

 Desktop/laptop

 Mobile device (smartphone/tablet)

 I have no preference

o [IF INDICATED A PREFERENCE] Which did you feel was easier to use when 

entering your expenses?

 Desktop/laptop

 Mobile device (smartphone/tablet)

o Can you tell me more about that? For example, what made one easier, or another 

more difficult?

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

• IF NEVER USED MOBILE DEVICE FOR ENTRY AND HAS DEVICE

o Expense entries could be made using a mobile device, for example your smartphone. 

You did not enter any expenses using your smartphone or other mobile device. Can 

you tell me why?

[PROBE FOR DETAILS: E.G. DIFFICULTY, TOOK TOO LONG TO LOAD, 

DIDN’T WANT TO USE UP DATA, ETC...]

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________
CE Online Diary Usability Testing – Task Order 

#26 

C-7 

 

• FUNCTION

o Within the online diary there were help links indicated with an ‘i’ (show screenshot). 

Did you ever use these?

 Yes [PROBE TO SEE IF R CAN RECALL WHICH WERE USED]

 No

o [IF YES] Did you find these helpful?

 Yes

 No

o Can you tell me more about that? 

PROBE FOR WHY OR WHY NOT – WHAT WOULD MAKE THEM MORE 

HELPFUL OR WHAT WAS MISSING/WERE THEY LOOKING 

FOR/EXPECTING

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

o A summary of the expenses you entered was visible either as the first screen (mobile) 

or while entering expenses (desktop). Was this helpful to see?

 Yes

 No

o Can you tell me more about that? 

PROBE FOR WHY OR WHY NOT – WHAT IF ANYTHING WOULD THEY 

CHANGE ABOUT IT.

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________
CE Online Diary Usability Testing – Task Order 

#26 

C-8 

 

o How easy or difficult was it to edit any expenses you had entered?

 Very difficult

 Difficult

 Neither

 Easy

 Very easy

 I did not edit any expenses

o Why did you say [RESPONSE]? 

PROBE FOR WHAT THEY FOUND EASY OR DIFFICULT – OR WHY 

THEY DID NOT EDIT ANY EXPENSES (E.G. DIDN’T KNOW THEY 

COULD OR DID NOT NEED TO)

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

o Overall how easy or difficult was it to figure out where you were in the diary, or what 

you were supposed to do?

 Very difficult

 Difficult

 Neither

 Easy

 Very easy

o What was [EASY/DIFFICULT] about the navigating the online diary?

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

o Finally, what changes would you like to see in the online diary? [SPECIFY 

WHETHER FOR MOBILE OR DESKTOP VERSION]

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________
CE Online Diary Usability Testing – Task Order 

#26 

C-9 

 

FEEDBACK FROM OTHER HOUSEHOLD PARTICIPANTS COLLECTED ONLINE.

Closing: Plick-up Visit 

THANK RESPONDENT AND END INTERVIEW.

PROVIDE $20 AND HAVE RESPONDENT SIGN RECEIPT. PROVIDE $10 FOR OTHER 

HOUSEHOLD PARTICIPANTS