U.S. Consumer Expenditure Survey (CE) 

Electronic Desktop Diary Design Improvements – Task Order #24 

Final Report 

 Authors 

Aaron Maitland

Douglas Williams

 August 9, 2016 

Prepared for:

U.S. Department of Labor

Bureau of Labor Statistics

Washington, DC 20212-0022

Prepared by:

Westat

An Employee-Owned Research Corporation®

1600 Research Boulevard

Rockville, Maryland 20850-3129

(301) 251-1500
This page is left intentionally blank.
Electronic Desktop Diary Design Improvements – 

Task Order #24 

iii 

 

Table of Content 

Chapter Page

1 Executive Summary............................................................................................ 1-1

2 Introduction ........................................................................................................ 2-1

3 Methods ............................................................................................................... 3-1

4 Findings................................................................................................................ 4-1

5 Recommendations.............................................................................................. 5-1

Appendixes

A Usability Protocol ............................................................................................... A-1

B Screenshots of Final Prototypes....................................................................... B-1

Tables

3-1 Demographic distribution of test participants ............................................... 3-1

4-1 Prototype preference by order of presentation.............................................. 4-1

4-2 Participant task difficulty ratings by prototype .............................................. 4-6

4-3 Independent coder task difficulty ratings by prototype ................................ 4-7

4-4 Task success ratings by independent coders and task timing by 

prototype.............................................................................................................. 4-8

Figures

2-1 Initial prototype 1 ............................................................................................... 2-2

2-2 Initial prototype 2 ............................................................................................... 2-3

2-3 Initial prototype 3 ............................................................................................... 2-4
Electronic Desktop Diary Design Improvements – 

Task Order #24 

iv 

 

Contents (continued) 

Figures Page

2-4 Final prototype 1................................................................................................. 2-5

2-5 Final prototype 2................................................................................................. 2-6

4-1 Participant ratings of how easy or difficult the prototypes were by 

version .................................................................................................................. 4-3

4-2 Participant ratings of how confident felt completing the entries in 

the prototypes by version.................................................................................. 4-4

4-3 Participant ratings of how much training the average person 

would need to get started with the prototypes by version............................ 4-5

5-1 Prototype 1 .......................................................................................................... 5-2

5-2 Resized screen for prototype 1 ......................................................................... 5-3

5-3 Prototype 2 .......................................................................................................... 5-4
Electronic Desktop Diary Design Improvements – 

Task Order #24 

1-1 

 

Westat in collaboration with Consumer Expenditure Survey (CE) staff participated in a development 

process to develop prototypes of the desktop version of the CE diary. Westat developed three initial 

prototypes during the Fall of 2015. After reviewing the three prototypes, the Bureau of Labor 

Statistics (BLS) decided on two versions to continue with further development. 

Westat conducted usability testing of two different versions of the CE desktop diary—Prototype 1 

and Prototype 2— in April and May of 2016. A total of 30 participants completed the testing at

Westat’s usability testing laboratory in Rockville, Maryland. All participants had experience using 

computers, smartphones, or tablets and 28 out of the 30 participants had a desktop or laptop 

computer at home. Participants completed a one-on-one interview. The participants entered 

expenses into both versions of the CE diary while the interviewer observed their behavior.

The analysis in this report is based on information from multiple sources. Both quantitative and 

qualitative debriefing information were analyzed. This includes ratings by the participants and an 

independent coder. The amount of time spent on different tasks in the usability interview was also 

analyzed. Finally, interviewer notes about problems the respondents encountered provided 

diagnostic information about the type of problems observed. 

Overall, participants preferred the more streamlined and simpler Prototype 1 because it included all 

of the necessary information and tasks on a single screen. Some participants thought that Prototype 

2 was more visually appealing; however, it took longer to learn how to use it. Overall, ratings by 

both the participants and independent coders and analyses of timings demonstrated that Prototype 1 

performed better than Prototype 2 during the usability test. Although, Prototype 2 did better when it 

was presented second and participants had become more comfortable with the CE reporting tasks. 

Westat recommends Prototype 1 if the need is to move forward relatively quickly with a 

straightforward design that is similar to the current paper version and other web versions of the CE 

diary. Although Prototype 2 did not perform as well as Prototype 1 in the usability testing, we think 

future research on the CE could focus on a more visually appealing design that would motivate 

participants to understand and complete the CE diary.

Executive Summary 1 
This page is left intentionally blank.
Electronic Desktop Diary Design Improvements – 

Task Order #24 

2-1 

 

The Consumer Expenditure Survey (CE) is sponsored by the Bureau of Labor Statistics (BLS). 

Researchers use CE data to study consumer spending in the United States; the data also are used in 

the computation of the Consumer Price Index (CPI). The study consists of two distinct data 

collection efforts, the Consumer Expenditure Quarterly Interview (CEQ) survey and the Consumer 

Expenditure Diary (CED) survey. For the CED, respondents are asked to complete two, 1-week 

diaries in which they record purchases made by their household during that week. Together, these 

surveys provide current and continuous information on the buying habits of America’s consumers. 

Recent trends for the CE surveys have shown declining response rates and concerns about 

measurement error. One recommendation given for addressing these issues was to provide 

respondents with the option to utilize mobile devices to record their purchases, which presents a 

number of benefits over paper diaries.

Westat produced a Data Capture and Financial Software Report, which identified new technologies 

that might be used to capture expenditure data (Westat, 20111). An Evaluation of Financial 

Application Software team was formed to explore generic expense tracking apps available on current 

mobile devices to evaluate their features. This team concluded that a standard existing app could not 

meet the specific needs of collecting CE data and that a custom design would be required.

The U.S. Census Bureau conducted the Individual Diaries Feasibility Test (IDFT) in 2014 in which a 

desktop diary and a mobile diary were tested. Respondents were asked screening questions that

assigned them to one of the two diary instruments. Using the initial findings from the IDFT, 

another project began as part of the overall CE Gemini Redesign project to further develop an 

electronic diary to be used in the CE Redesign Proof of Concept Test (POC). The updated design 

allows a respondent to access a single electronic diary via a desktop computer or a mobile device to 

record household expenses. The POC test was fielded in 2015. Due to time and resource constraints 

in preparation for the POC test, some ideas that were initiated for a desktop version were not able 

to be explored and implemented. 

 

1 Westat. (2011). Data capture technologies and financial software for collecting consumer expenditure data. Rockville, 

MD: Westat.

Introduction 2 
Electronic Desktop Diary Design Improvements – 

Task Order #24 

2-2 

 

The development effort up until that point had focused most closely on the mobile version of the 

diary. For example, the mobile version has undergone several rounds of usability testing which has 

yielded recommendations for improving the design. Hence, these development efforts have created 

a mobile version that works well for the purposes of the CE. However, the BLS would like to have 

seen more development of the desktop version of the diary that takes advantage of the larger screen 

size and different input options involving the mouse and keyboard. 

Westat, in collaboration with CE staff, participated in a process to develop prototypes of the 

desktop version of the CE diary. Westat developed three initial prototypes during the Fall of 2015. 

Prototype 1 included a significant amount of white space and guided the respondent through the 

steps to enter purchases on the left-hand side of the screen while showing a summary of reported 

purchases on the right. Prototype 1 is shown below in Figure 2-1. The categories are modified 

slightly from the typical CE categories by separating out the “other” category into home furnishings 

and decorative items, entertainment and recreation, transportation expenses, medical expenses and 

supplies, and education expenses and supplies. We maintained the categories of food and drinks 

away from home; food and drinks for home consumption; and clothing, shoes, jewelry, and 

accessories. We also include a category for all other products, services, and expenses.

Figure 2-1. Initial Prototype 1

The principles behind Prototype 1 focused on presenting all respondent actions and feedback on 

one screen. All entry fields followed a clear linear (top to bottom) navigational path to keep the 

response task simple and to minimize any skip errors (errors of omission). Entry fields were 
Electronic Desktop Diary Design Improvements – 

Task Order #24 

2-3 

 

centered in the left-hand side of the activity frame. Results of these entries or response actions were 

visible in the right-hand side of the activity frame. This presented the respondent with their most 

recent entries allowing an opportunity to keep track of what was entered without requiring 

navigation to another screen.

Prototype 2, shown in Figure 2-2, walked the respondent through the data entry process in steps and 

provided more visual cues for categories of expenses. Initial Prototype 2 begins with the respondent 

selecting the day on which an expense occurred. The participant then selects a category for the 

expense by clicking on an icon. Hence, this prototype includes a visual cue for each of the expense 

categories as opposed to Initial Prototype 1 where text is selected from a dropdown menu. The 

expense categories are the same as Initial Prototype 1. The respondent enters the details about the 

item on the third screen. The fourth screen gives the participant options for entering another of the 

same type of expense for the same day, a different type of expense for the same day, an expense for 

a different day, or logging out of the diary. This gives the respondent the opportunity to enter in 

similar expenses without having to select the category or day again.

Figure 2-2. Initial Prototype 2

The principles behind Prototype 2 focused on decomposing each component of entering purchases. 

This allows efficiency gains in the response task, potentially reducing perceived burden or 
Electronic Desktop Diary Design Improvements – 

Task Order #24 

2-4 

 

redundancy, by not requiring the respondent to select the purchase date or category for each item. 

Prototype 2 also incorporates more visual cues to help communicate assigned days for purchase 

entry, and meaning behind category labels by showing prototypical category items. Visual cues are 

also used to help with navigation with the use of icons (or small picture) to help determine the next 

action to take after purchase(s) have been entered.

Initial Prototype 3, shown in Figure 2-3, provided more contextual information to the respondent 

about what types of expenses should be reported. For example, the first screen provides examples of 

the types of products that would be included under each category. The respondent selects the date 

from a calendar on the second screen. The details for the item are entered on the third screen. 

Finally, on the fourth screen, the respondent can select whether they want to enter another expense, 

see a summary of expenses entered, or logout. 

Figure 2-3. Initial Prototype 3

The principle behind Prototype 3 focused on improving respondent understanding by providing 

information, such as, category examples to help the respondent select the appropriate category. 

Prototype 3 also followed the structure of many commercial websites where respondents are likely 

to shop to capitalize on navigational cues respondent may be familiar with. Like Prototype 2, this 
Electronic Desktop Diary Design Improvements – 

Task Order #24 

2-5 

 

prototype decomposed the task of entering expenses, but also incorporated more examples to help 

improve data quality.

Westat provided the three prototypes to BLS in December 2015. After reviewing the three 

prototypes, BLS selected two for further development. BLS chose Prototype 1 since it was the most 

similar to the current CE design, it appeared less burdensome, and the response task was relatively

clear. BLS also chose to continue with Prototype 2 since it was a visibly appealing alternative. 

Prototype 3 development stopped because of concerns that the screen may appear too busy and 

imply a higher level of burden. In addition to selecting the prototypes, BLS made several suggestions 

to improving Prototype 1 and Prototype 2. 

Westat continued development of the two prototypes through February of 2016. The two final 

prototypes were presented to BLS on February 9, 2016. Prototype 1 and Prototype 2 are shown 

below in Figure 2-4 and Figure 2-5. Additional screenshots are shown in Appendix B.

Figure 2-4. Final prototype 1
Electronic Desktop Diary Design Improvements – 

Task Order #24 

2-6 

 

Figure 2-5. Final prototype 2

Westat prepared a protocol for usability testing of these two prototypes with CE staff and 

conducted the testing in April and May of 2016. This report provides the details about the testing of 

the two prototypes. Chapter 3 provides details about the methods that were used to conduct the 

testing. Chapter 4 presents the findings of the usability test. Finally, Chapter 5 presents Westat’s 

recommendations for a desktop diary design for the CE.
Electronic Desktop Diary Design Improvements – 

Task Order #24 

3-1 

 

Westat conducted usability testing of two prototypes of the CE desktop diary in April and May of 

2016. A total of 30 participants completed the testing at Westat’s usability testing laboratory in 

Rockville, Maryland.

 Recruiting 

Participants were recruited by running an advertisement on the Internet web site Craigslist. Table 3-

1 shows the distribution of age, sex, race, and education for the 30 participants. 

Table 3-1. Demographic distribution of test participants

Characteristic Count 

Age 

30 and under 6

31-49 5

50-59 12

60+ 7

Sex 

Male 12

Female 18

Race 

White 13

Black 14

Other 3

Education 

High school or GED 9

Some college 10

College degree 11

All of the participants had experience using computers, smartphones or tablets and 28 out of the 30 

participants had a desktop or laptop computer at home. One of the participants had a smartphone, 

but no desktop computer at home. Another participant had used computers at work, but did not 

have one at home. We asked participants during screening how often they use a desktop or laptop 

computer and 24 reported that they use the computer “Always” or “Very often,” four reported that 

they use the computer “Sometimes” or “Rarely,” and two reported that they “Never” use a desktop 

or laptop computer. 

Methods 3 
Electronic Desktop Diary Design Improvements – 

Task Order #24 

3-2 

 

 Procedures 

Participants visited the usability lab at Westat for a one-on-one interview. The complete protocol is 

shown in Appendix A. The interviewer gave the participant some background on the study and then 

administered the consent form. Once consent was obtained the interviewer demonstrated how to 

enter expenses into the first prototype that the participant encountered. The demonstration helped 

to introduce the respondent to the diary and the data entry procedures since it was not possible to 

give a full explanation of the task as would be done in the actual diary placement due to time 

constraints in the lab. The interviewer entered one food and drinks away from home and one food 

and drinks for home consumption example. We demonstrated these two categories because of the 

differing data entry procedures between the categories. Food and drinks away from home requires 

the participant to enter all of the expenses on the receipt on one line and enter total cost including 

tax and tip. Food and drinks for home consumption (like all other categories) requires the 

participant to itemize the receipt and enter the cost of each item without tax. 

Once the demonstration was complete the participant was asked to complete six tasks in one of the 

prototypes of the diary. One half of the participants were asked to use Prototype 1 for the first six 

tasks and the other half were asked to use Prototype 2 to complete the first six tasks. These tasks 

consisted of fictional scenarios and included receipts that were created by the study team for the 

participant to enter. At the completion of each individual task, the interviewer asked the participant 

“How easy or difficult was it to complete this task?” The interviewer recorded the participants’

answers and noted their comments. The interviewer may have also probed on any issues that were 

observed during the completion of the task. At the end of all six tasks for the first prototype 

encountered, the interviewer administered probes about the participants’ impressions of this version 

of the diary, how easy or difficult the diary was to complete, how confident they were in completing 

the diary, and how much training they felt the average person would need to use the diary. 

After completing the tasks using the first prototype, the participant used the opposite prototype of 

the diary for the next six tasks. The interviewer demonstrated how to use the other prototype using 

the same examples that were used to explain the first prototype. The participant then completed six 

additional tasks. The interviewer followed the same probing procedures on both prototypes. 

Participants were asked to bring in a receipt of their own and to enter some of the items from that 

receipt on the second prototype that they encountered. 

The interviewer administered some final probes at the end of the interview once the participants had 

completed all of the tasks for both prototypes of the diary. These probes asked participants to 
Electronic Desktop Diary Design Improvements – 

Task Order #24 

3-3 

 

indicate which prototype that they prefer, the most difficult part of completing the diary, the easiest 

part of completing the diary, and provide any comments about how to make the diary easier to 

complete.

Interviewers entered their findings from the debriefing questions and qualitative descriptions of any 

issues they observed during the interview into a unified notes document that was used for analysis. 

 Analysis 

The analysis in this report is based on three primary sources: quantitative and qualitative debriefings, 

ratings and timings, and interviewers’ notes.

Regarding the information provided through the debriefings, some of it is presented in quantitative 

tallies of the participants’ ratings or preferences. For example, participants rated the difficulty of 

each task after it was completed. In addition, participants provided overall ratings of task difficulty, 

confidence, and the amount of training needed after completion of all six tasks for each prototype. 

Participants were also asked which prototype that they prefer after completing the tasks for both. 

Qualitative debriefing information is also summarized in this report. For example, participants were 

asked about their thoughts on each prototype. They were also asked to describe the easiest and 

hardest parts of completing the diary and also to comment on how to make the diary easier to 

complete. 

Timings and independent ratings of task success were also used to evaluate the two prototypes. To 

obtain this information, two research assistants reviewed the video recordings to record the amount 

of time the participants spent on the task and also rated whether the person was not successful, 

partially successful, or fully successful at completing the task. Not successful means that the 

participant could not complete the task without help from the interviewer. Partially successfully 

means that the participant completed most of the task, but made a mistake on a minor element of 

the task or had to make multiple attempts. A fully successful rating means that the participant

completed the task without any errors. The timings were recorded from the time that the participant 

started the task to the time that they completed the task and the interviewer asked them to rate the 

difficulty of the task. The ratings include some time where the participant might be talking to the 

interviewer and not actually spending time on the task. 
Electronic Desktop Diary Design Improvements – 

Task Order #24 

3-4 

 

Another source of information includes the interviewers’ notes regarding the problems that 

participants encountered during each task. We use these notes to provide a rich explanation and 

diagnosis of the most serious problems. Many of the quantitative ratings and timings can be 

indicators of problems; however, these notes from the interviews provided the best diagnostic 

information about the problems that were encountered.
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-1 

 

The findings are presented in three different sections. We first present the participants prototype 

preference based on their own self report. Next, we compare participants’ ratings for each 

prototype. These ratings include overall ratings for each prototype and specific ratings on the tasks 

that were performed with each prototype. This section also includes the amount of time that 

participants spent on each task and independent ratings of success with each task. Finally, we 

present task-by-task descriptions of the issues that were encountered with each prototype. 

 Prototype Preference 

Table 4-1 shows which prototype the participants preferred after completing all of the tasks with 

both prototypes in the usability testing session. Overall, 19 out of the 30 participants preferred 

Prototype 1. The table also shows that the preferences were affected by the order in which the 

participants used the prototypes. Each prototype was more likely to be preferred when it was the 

second prototype encountered by the participant in the usability test. Nearly all of the participants 

who used Prototype 1 last preferred that prototype. Preference was more evenly divided when 

Prototype 2 was used last. 

Table 4-1. Prototype preference by order of presentation

Prototype 

Order of presentation 

Overall 

Prototype 1 first, 

Prototype 2 second 

Prototype 2 first, 

Prototype 1 second 

Prototype 1 6 13 19

Prototype 2 7 2 9

Not sure 2 0 2

Total 15 15 30

The participants provided reasons why they preferred one or the other prototype. Prototype 1 was 

viewed as a simpler layout than Prototype 2 for most participants. The participants who preferred 

Prototype 1 mentioned that they liked having everything right in front of them including the data 

entry fields and the summary of everything already entered. Many of the participants who preferred 

Prototype 1 mentioned that they like having everything on one screen rather than having to navigate 

through multiple screens. In addition, the summary on the right-hand side with the easy-to-find 

Findings 4 
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-2 

 

delete and edit buttons was also a clear advantage with Prototype 1. One negative aspect to 

Prototype 1 was that entering the date became repetitive for some participants.

Prototype 2 had some advantages that those who preferred that version mentioned. Several 

participants mentioned that they liked the first screen that lists the actual dates in the study. This was 

clearer for them than using the calendar. They also liked that they did not have to enter the date for 

every expense that was entered. Also, having the icons made it somewhat easier to select the 

categories more quickly. One participant mentioned that the icons made it easier because he has 

glasses and could see the categories much better. One drawback to Prototype 2 was that it was 

difficult to navigate to a screen where an entry could be edited or deleted. The expense review page 

did not include an obvious navigation icon for deleting or editing. Some participants also got 

confused about how to navigate between screens particularly during the early tasks. 
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-3 

 

 Rating and Timing Data 

Next, we show the ratings the participants provided after they completed each version of the tasks 

during the usability test. Figure 4-1 shows participant ratings of how easy or difficulty it was for the 

participants to use each prototype on the first six tasks and the last six tasks. Prototype 1 was rated 

as much easier compared to Prototype 2 on the first six tasks. The ratings of the prototypes are quite 

similar on the last six tasks.

Figure 4-1. Participant ratings of how easy or difficult the prototypes were on the first six tasks 

and the last six tasks (scale: 1=Extremely difficulty, 2=Very difficult, 3=Somewhat 

difficult, 4=Neither easy nor difficult, 5=Somewhat easy, 6=Very easy, 7=Extremely 

easy).

2

 

2 Note. The numbers on the bars in Figure 4-1, Figure 4-2 and Figure 4-3 represents the number of participants in each 

category.

6

0

4

5

4

6

8 5

4

6

3

1

0

1

0

1

1 1

2

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

First 6 Tasks,

Prototype 1,

Mean=5.9

First 6 Tasks,

Prototype 2,

Mean=5.2

Last 6 Tasks,

Prototype 1,

Mean=6.1

Last 6 Tasks,

Prototype 2,

Mean=5.7

Somewhat difficult

Neither

Somewhat easy

Very easy

Extremely easy
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-4 

 

Figure 4-2 shows participant ratings of how confident they were filling out the entries for each 

prototype on the first six tasks and the last six tasks. The number of participants reporting extremely 

or very confident ratings is quite similar across all tasks for Prototype 1; however, there is higher 

confidence for Prototype 2 on the last six tasks compared to the first six tasks. 

Figure 4-2. Participant ratings of how confident felt completing the entries in the prototypes on 

the first six tasks and the last six tasks (scale: 1=Not at all confident, 2=A little 

confident, 3=Somewhat confident, 4=Very confident, 5=Extremely confident)

6

2 2

7

8

6

10

4

1

6

3 3

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

First 6 Tasks,

Prototype 1,

Mean=4.3

First 6 Tasks,

Prototype 2,

Mean=3.7

Last 6 Tasks,

Prototype 1,

Mean=3.9

Last 6 Tasks,

Prototype 2,

Mean=4.3

Somewhat confident

Very confident

Extremely confident
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-5 

 

Figure 4-3 shows participant ratings of how much training participants think the average person 

needs to get started using each prototype depending on whether they completed the prototype for 

the first six tasks or the last six tasks. The majority of participants thought that the average person 

would need “none” or “a little” training across all prototypes regardless of when they encountered 

the prototypes. There is very little variation in the ratings across the groups.

Figure 4-3. Participant ratings of how much training the average person would need to get 

started with the prototypes on the first six tasks and the last six tasks (scale: 

3=None, 2=A little, 1=A moderate amount, 0=A lot)

2

1

4

2

8

8

8

7

4 4

2

5

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

First 6 Tasks,

Prototype 1,

Mean=0.9

First 6 Tasks,

Prototype 2,

Mean=0.8

Last 6 Tasks,

Prototype 1,

Mean=1.1

Last 6 Tasks,

Prototype 2,

Mean=0.8

Moderate

A little

None
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-6 

 

Table 4-2 shows the participants ratings of task difficulty by prototype. The mean rating across 

participants for each prototype is shown in the last column. Overall, Prototype 1 has a higher mean 

rating indicating that it is rated as the easier of the two diaries by participants. Task 2 (lunch), task 3 

(groceries), task 5 (Return book), task 8 (Dinner), task 10 (Shopping trip 2), and task 11 (Return 

shirt) were tasks that seem to have been viewed as easier with Prototype 1 compared to Prototype 2.

Table 4-2 does not show any tasks where Prototype 2 was clearly viewed as easier than Prototype 1 

by the participants. 

Table 4-2. Participant task difficulty ratings by prototype

Task Prototype 

Rating 

Mean 

5 

Very easy 

4 

Somewhat 

easy 

3 

Neither 

2 

Somewhat 

difficult 

1 

Very 

difficult 

1 (Log in 1) 1 13 2 0 0 0 4.9

2 15 0 0 0 0 5.0

2 (Lunch) 1 10 5 0 0 0 4.7

2 7 6 1 1 0 4.3

3 (Groceries) 1 7 4 2 1 1 4.0

2 1 10 1 2 0 3.7

4 (Shopping trip 1) 1 8 4 3 0 0 4.3

2 5 9 0 0 0 4.4

5 (Return book) 1 14 1 0 0 0 4.9

2 5 3 0 2 3 3.4

6 (Medicine) 1 9 5 1 0 0 4.5

2 8 4 0 1 0 4.5

7 (Log in 2) 1 15 0 0 0 0 5.0

2 15 0 0 0 0 5.0

8 (Dinner) 1 9 5 0 0 0 4.6

2 4 6 1 1 0 4.1

9 (Movie) 1 11 2 0 0 0 4.8

2 11 2 0 0 0 4.8

10 (Shopping trip 2) 1 9 3 1 0 0 4.6

2 6 3 2 2 2 4.0

11 (Return shirt) 1 9 0 0 3 0 4.3

2 6 2 2 3 0 3.8

12 (Own expense) 1 10 1 0 0 0 4.9

2 8 1 0 0 0 4.9

Overall 1 124 32 7 4 1 4.6

2 91 46 7 12 3 4.3
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-7 

 

Table 4-3 shows ratings of task difficulty from independent coders by prototype. The mean rating 

across coders for each prototype is shown in the last column. Similar to the participant ratings, 

Prototype 1 has a higher overall mean rating indicating that it is rated as the easier of the two diaries 

by the coders. Task 2 (lunch), task 3 (groceries), task 4 (Shopping trip 1), task 5 (Return book), task 

6 (medicine), task 10 (Shopping trip 2), and task 11 (Return shirt) were tasks that seem to have been 

viewed as easier with Prototype 1 compared to Prototype 2. Table 4-3 does shows that task 9 may 

have been slightly easier with Prototype 2 compared with Prototype 1. 

Table 4-3. Independent coder task difficulty ratings by prototype

Task Prototype 

Rating 

Mean 

5 

Very easy 

4 

Somewhat 

easy 

3 

Neither 

2 

Somewhat 

difficult 

1 

Very 

difficult 

1 (Log in 1) 1 10 3 0 0 0 4.8

2 13 1 0 0 0 4.9

2 (Lunch) 1 7 8 0 0 0 4.5

2 4 5 2 1 2 3.6

3 (Groceries) 1 4 5 2 4 0 3.6

2 2 4 1 3 3 2.9

4 (Shopping trip 1) 1 7 5 3 0 0 4.3

2 5 3 1 2 2 3.5

5 (Return book) 1 14 0 0 1 0 4.8

2 2 2 0 2 8 2.1

6 (Medicine) 1 12 2 1 0 0 4.7

2 9 1 0 1 1 4.3

7 (Log in 2) 1 13 1 0 0 0 4.9

2 15 0 0 0 0 5.0

8 (Dinner) 1 4 8 1 1 1 3.9

2 4 7 3 1 0 3.9

9 (Movie) 1 9 3 0 1 0 4.5

2 10 3 0 0 0 4.8

10 (Shopping trip 2) 1 9 1 2 1 0 4.4

2 4 5 2 3 0 3.7

11 (Return shirt) 1 8 3 1 1 0 4.4

2 5 5 0 1 2 3.8

12 (Own expense) 1 10 2 0 0 0 4.8

2 10 1 1 0 0 4.8

Overall 1 107 41 10 9 1 4.5

2 83 37 10 14 18 3.9
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-8 

 

Table 4-4 shows ratings of task success from independent coders and timings on each task by 

Prototype. Overall, Prototype 1 had a higher success rate and took less time to complete than 

Prototype 2. Participants appear to be more successful with Prototype 1 compared to Prototype 2 

on task 4 (Shopping trip 1), task 5 (Return book), and task 6 (Medicine). Table 4-4 does not show

any tasks where Prototype 2 had a clear advantage to Prototype 1 in terms of task success. The 

timing column of the tables illustrates that Prototype 1 took less time on the first six tasks. The 

differences in timing are much smaller for the last six tasks with the exception of task 11 (return 

shirt). 

Table 4-4. Task success ratings by independent coders and task timing by prototype

Task Prototype 

Success rating Timing 

Fully 

successful 

Partially 

successful Not successful Mean (MM:SS) 

1 (Log in 1) 1 14 1 0 00:21

2 14 0 0 00:28

2 (Lunch) 1 8 6 0 01:27

2 8 4 2 02:47

3 (Groceries) 1 2 12 0 05:17

2 4 7 2 06:34

4 (Shopping trip 1) 1 8 6 0 04:00

2 3 9 1 05:33

5 (Return book) 1 14 1 0 00:10

2 4 2 8 01:36

6 (Medicine) 1 12 2 0 02:32

2 8 3 1 02:38

7 (Log in 2) 1 14 0 0 00:15

2 15 0 0 00:11

8 (Dinner) 1 3 10 2 02:17

2 2 12 0 02:19

9 (Movie) 1 8 4 0 01:06

2 9 3 0 00:50

10 (Shopping trip 2) 1 6 7 0 02:54

2 9 4 0 03:27

11 (Return shirt) 1 11 2 0 00:23

2 11 0 2 00:54

12 (Own expense) 1 10 2 0 01:51

2 10 2 0 01:42

Overall 1 110 53 2 01:54

2 97 46 16 02:22
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-9 

 

 Task-by-Task Issues 

In this section, we summarize problems that were encountered with each prototype. 

Task 1: Login 1

No one experienced any issues with the log in screen on either prototype. One participant initially 

had some issues with the keyboard on the laptop computer, but had no problems with the actual 

task of entering the user name and password.

Task 2: Lunch

Prototype 1

 Two participants attempted to type in the date rather than selecting the date with the 

calendar. Their entered date disappeared when they tried to save the entry since the date 

was not in the correct format. The two participants did not notice the calendar at first. 

 One participant itemized the receipt by meal and entered the price without tax for each 

meal.

 Two participants skipped fields such as the type of meal or whether alcohol was 

included.

Enter your last name as the user name and enter “password1” for the password to log in to 

version 1.

On April 11 you go out to eat lunch with your child and order one meal for yourself and one 

meal for your child. Enter the expense into the diary. 
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-10 

 

Prototype 2

 Three participants entered incomplete descriptions by not entering the drinks. 

 One participant itemized the meals on the receipt incorrectly and entered the price of 

each meal without tax.

 One participant missed the cost field.

 Two participants were confused about whether to click finish or add item after entering 

the information from the receipt on the expense detail screen.

Task 3: Groceries

Prototype 1

 Eight participants did not itemize until the interviewer reminded them that they needed 

to itemize these types of events. Once the participants were reminded to itemize they 

were able to enter the items correctly.

 Six participants entered the Kleenex and bath tissue as Food and Drinks for Home 

Consumption. Some thought that everything on the receipt should be entered into the 

same category and at least one participant was just focusing on the “home 

consumption” part of the category. 

On the way home you stop at the grocery store and purchase some food. Enter these expenses 

in the diary.
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-11 

 

 One participant tried to type in the date, but entered the wrong format and the entered 

date was not accepted.

 One participant noted that it was tedious to select the date with the calendar for every 

item.

Prototype 2

 Four participants did not itemize until the interviewer reminded them that they needed 

to itemize these types of events. Once again, when the participants were reminded to 

itemize they were able to enter the items correctly.

 Three participants categorized the Kleenex and bath tissue as Food and Drinks for 

Home Consumption.

 One participant initially entered the wrong date. The interviewer made them aware of 

this and the participant had a difficult time finding the correct navigation icons to 

change to the correct date. 

 One participant missed the cost field, which was noticed when he tried to click “Add 

expense.”

 Two participants initially clicked on “Enter another of this type of expense...,” which 

was incorrect. One of these participants clicked “go back” and correctly selected “Enter 

a different type of expense for...”

Task 4: Shopping Trip 1

On April 13 you go shopping at a local discount store and purchase a chair, a video game for 

your child, and a book to read. You also stop to purchase gas and get your car washed on the 

way home. Enter these expenses in the diary.
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-12 

 

Prototype 1

 One participant noted that it was tedious to select the date for every item. Another 

thought that it was tedious to use the calendar and that it should be easier to just type in 

the date. The date field only accepted one specific format. Another participant thought 

that the calendar button is not obvious enough.

 Five participants included the tax for the car wash. One participant corrected this by 

editing once the interviewer asked him about this. 

 One participant missed the cost field and left it blank.

Prototype 2

 Five participants expressed confusion or had problems using the navigation button. A 

few participants were confused about whether to click on “Enter another of this type of 

expense...” or “Enter a different type of expense...”, but eventually chose the correct 

icon sometimes with the help of the interviewer. Another participant was confused 

because the previous task had many items on the expense review page and the 

navigation icons were not visible on the screen. This participant tried using the back and 

forward buttons on the browser button until the interviewer instructed her to scroll 

down. She then selected the correct navigation icon. 

 Three participants incorrectly entered all of the items from the discount store as home 

furnishings. This seemed to be due to these participants thinking that everything on one 

receipt belongs in the same category, but could have also been due to difficulties with 

navigating to other categories.

 One participant entered the price of the car wash including tax. Another participant 

noted that the instructions on whether or not to include tax were not very noticeable.

Task 5: Return Book

Prototype 1

 There were no problems deleting the book from the list on Prototype 1.

Prototype 2

 Only six participants were able to successfully complete this task. Four used the 

dropdown menu that listed expenses for different days. Two were able to navigate back 

to the expense date screen where all of the items were listed. Once participants got to 

the correct screen they were able to easily delete the book.

You decide to return the book that you bought. Please go back and delete that item.
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-13 

 

 The remaining participants could not successfully navigate to a screen that would allow 

them to delete the correct item. A few tried removing the last item from the expense 

review screen which was incorrect since it was typically the car wash item. Several tried 

clicking on the bread crumbs at the top of the screen to navigate to a different location. 

Some also tried clicking the back and forward buttons in the browser. As one 

participant said, “It doesn’t tell you how to go back so you’re just lost you don’t know 

what to do...I don’t know how to get back to the book to delete it.” Another was 

looking for a home screen button, “all of these are asking to enter expense” referring to 

icons at the bottom of the screen “if there’s a home button or something... I could go 

back to the main screen and maybe something is there.”

Task 6: Medicine

Prototype 1

 One participant grouped mouthwash, toothpaste, and shampoo and recorded the sum 

of their prices in a single line because they were similar products. This might have been 

an artifact of the task since there was a space on the receipt between the medicine and 

these other three items.

 We did not observe any other problems. It was a little ambiguous which category the 

hygiene products should go into. Some participants categorized the hygiene products as 

medicine. Other categorized these products as other type of expense. Some mentioned 

that it would be nice to have a personal care category.

Prototype 2

 Similar to Prototype 1, one participant entered the hygiene products on one line and 

participants also noted the ambiguity with the hygiene products in terms of which 

category they should be placed into.

On April 15 you stop at the pharmacy to purchase some medicine to treat your sore throat. 

You also pick up some personal care products. Enter these expenses in the diary.
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-14 

 

 One participant needed to be instructed to click on “add expense” in order to record 

the first item that was entered. This participant tried to click on “finish” before “add 

expense.”

Task 7: Login 2

 No problems were observed with logging in with either prototype.

Task 8: Dinner

Prototype 1

 Six participants did not include the tip in the total cost. This was the first task on which 

the participants saw an extra receipt with the tip and the may have overlooked the credit 

card receipt.

 Two participants recorded the cost of alcohol incorrectly by only reporting the cost of 

either the beer or the wine.

 One participant incorrectly chose the food and drinks for home consumption category.

 One participant typed in the date in the wrong format and received an error message 

asking him to record the date.

 Two participants recorded an incomplete description as simply “steak and shrimp 

dinner” implying only one meal.

Enter your last name as the user name and enter “password1” for the password to log in to 

version 2.

On April 12 you and your spouse go out to eat dinner at a restaurant. Enter the expense into 

the diary.
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-15 

 

Prototype 2

 Two participants itemized the meal incorrectly. 

 Three participants did not include the tip in the total cost.

 Two participants were confused about the “add expense” and “finish” buttons and 

needed to be instructed to click “add expense” before clicking “finish.”

 Two participants recorded an incomplete description as “steak and shrimp dinner”

implying only one meal.

 Two participants incorrectly recorded the price of alcohol including only the cost of the 

either the beer or the wine. Another participant made a mistake with slider buttons 

selecting the alcohol. The participant accidentally selected “Wine” and “other” thinking 

that the appropriate slider buttons appeared after the words.

Task 9: Movie

Prototype 1

 Three participants entered the cost including tax.

Prototype 2

 Two participants ignored the subtotal and incorrectly recorded the cost as $15. One 

participant included the tax in the cost.

 Three participants were confused about the “add expense” and “finish” buttons. There 

was only one item on the receipt so they just wanted to click “finish” once they were 

done entering the movie tickets.

You and your spouse also go out to a movie that night after dinner. Enter this expense in the 

diary.
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-16 

 

 One participant incorrectly selected “Add another of this type of expense...” from the 

expense review screen and had a difficult time navigating to the correct place to “Add a 

different type of expense...”

Task 10: Shopping Trip 2

Prototype 1

 Five participants incorrectly grouped the three purchases on the clothing and jewelry 

store receipt onto a single line because they were in the same category.

 One participant commented that it was repetitive to enter the date for each item.

Prototype 2

 Three participants incorrectly grouped the three purchases on the clothing and jewelry 

store receipt onto a single line because they were in the same category. One of these 

participants entered the store name in the description.

 Four participants encountered issues with the navigation buttons on this task. These 

issues were encountered at slightly different steps. One participant incorrectly selected 

“Add a different expense for the same day...” when it should have been for a different 

day. She entered the information from the clothing and jewelry receipt and again 

selected “Add a different expense for the same day...” incorrectly. After entering the 

bus fare she realized the mistake and edited that entry from the expense review page 

with no problems, but she did not correct the other entries. One of the other 

participants made a similar mistake by clicking on “Add a different expense for the 

same day...” Another participant took an inefficient path to entering the items by 

clicking on “Add expense for a different day” after each expense that was entered. This 

returned her to the expense date page where she would select the same day each time. 

Other than taking more time, this strategy seemed to work. Another participant wanted 

to edit the details for the necklace after it had been entered. He had already advanced to 

the expense type paged and clicked the back button on the browser from there. This 

On April 14 you take the bus downtown to shop for some jewelry, cosmetics, and clothes. 

Enter the expenses in the diary.
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-17 

 

brought him back to the expense review page, but it was now empty. He was not able to 

edit successfully after clicking several different icons. 

 One participant noted that it was tedious to click “add event” and “finish” all the time.

 One participant entered “shirt” for the dress shirt and ignored the pop-up asking him to 

be more specific about the type of shirt.

Task 11: Return Shirt

Prototype 1

 Four participants were not able to do this task because they entered the items from the 

previous task on the same line. Some of these participants actually removed the shirt by 

editing and removing the shirt from the description and then subtracting the cost of the 

shirt from the total. Anyone who itemized correctly in Task 10 was able to easily delete 

the dress shirt from the list.

Prototype 2

 Four participants were unable to find a list where they could delete the dress shirt. 

Seven participants located the dropdown list on the expense review pages and deleted 

the item and three navigated back to the expense date and deleted the item.

Task 12: Own Expense

Prototype 1

 One participant entered multiple items under food and drinks for home consumption 

on one line incorrectly and added the items together to get the cost. 

 One participant entered a breakfast sandwich for food and drinks away from home, but 

did not enter the hash browns on the receipt.

 One participant entered dog treats and food and drinks for home consumption.

You decide to return the shirt that you bought. Please go back and delete that item.

Enter the first three purchases on the receipt that you brought with you to the interview.
Electronic Desktop Diary Design Improvements – 

Task Order #24 

4-18 

 

Prototype 2

 One participant excluded tax from a food away from home item.

 One participant recorded the correct price for a food and drinks for home consumption 

item. The participant ignored the discounts at the bottom the receipt.

 One participant switched categories by always returning to the main calendar rather 

than clicking on “Enter a different type of expense for the same day.”
Electronic Desktop Diary Design Improvements – 

Task Order #24 

5-1 

 

In this section, we present recommendations based on the testing of the two prototypes. We 

discuss the advantages and disadvantages of each prototype in this section. The decision about 

which prototype to move forward with depends largely on the goals and objectives of the CE 

program. 

Before we discuss our recommendations for which prototype to move forward with, we would like 

to make a brief comment on the expenditure categories used in this usability test on both 

prototypes. The prototypes used in this usability test used an extended set of expenditure categories 

than has been used in the CE. The current CE uses the categories of Food and Drinks Away from 

Home; Food and Drinks for Home Consumption; Clothing Shoes, Jewelry, and Accessories; and All 

Other Products, Services, and Expenses. The prototypes in this report used the same first three

categories and decomposed the last category a little further. The prototypes in this report included 

the additional categories of Home Furnishings and Decorative Items; Entertainment and Recreation; 

Transportation Expenses; Medical Expenses and Supplies; and Education Expenses and Supplies. 

Although it is difficult to make direct comparisons to the original categories since we did not include 

original categories in the testing, our experience suggests that the extended set of categories made 

people think about a higher level of detail than would be expected with the original set of 

expenditure categories. This was most evident with Task 6 where mouthwash, toothpaste, and 

shampoo were included on a receipt with medicine. Several participants commented that these 

products did not quite fit within Medical Expenses and Supplies, but thought that was the best fit 

given the existing categories. Some specifically mentioned that they would have like to have seen 

personal care products as an option. The motivation for including a broader set of categories is to 

stimulate respondents to think about items that may have been forgotten otherwise. This test does 

seem to indicate that the participants thought about expenditures at a higher level; however, the a 

better test of the effectiveness of the categories would be in a field test where one can compare the 

prevalence of the items in the new categories when the original categories are used and when the 

new categories are used. We recommend this kind of field test in the future.

Recommendations 5 
Electronic Desktop Diary Design Improvements – 

Task Order #24 

5-2 

 

Looking at the prototype recommendations, Prototype 1 shown in Figure 5-1 is preferred if the 

desire is to move forward relatively quickly with a straightforward design that is similar to the 

current paper version and other web versions of the CE diary.

Figure 5-1. Prototype 1

Prototype 1 performed very well in the usability test and has a few main advantages. First, it has the 

advantage of being easier for respondents to learn and use. The necessary fields that the respondent 

needs to provide accurate answers in the diary are all contained on one screen. The data entry 

elements are on the left-hand side of the screen and the summary on the right-hand side of the 

screen gives the respondent instant feedback when items are entered into the diary. Respondents will 

also find it very easy to edit and delete items when necessary through the visible summary list. 
Electronic Desktop Diary Design Improvements – 

Task Order #24 

5-3 

 

We also feel that Prototype 1 is the more adaptable of the two prototypes to a mobile platform. The 

resizing of the web browser in Figure 5-2 gives an idea of how the prototype would render on a 

smartphone in a responsive design. The user would have access to the data entry fields at the top of 

the screen with the search and summary list stacked underneath.

Figure 5-2. Resized screen for Prototype 1

Some minor enhancements could improve the performance of Prototype 1. First, some participants 

in the usability test preferred to type in the date; however, they did not always enter the date in the 

correct format. It would be helpful for the date field to capture alternative formats for entering dates 

if the participant prefers. Second, participants were often confused about when to include or exclude 

tax. The help text about whether or not to include tax and tips could be made more visible so that 

the user pays more attention to this on the screen. Similar help text about whether a description is 

adequate or not could also be made more visible.

Although Prototype 2 (shown in Figure 5-3) did not perform as well as Prototype 1 in the usability 

testing, we think there are some improvements that could be made to improve the usability of this 

prototype. It is worth considering future research on similar prototypes since participants often saw 

this prototype as more visually appealing. Participants also liked certain features such as not having 

to select the date every time. The action of simply clicking on the day rather than selecting from the 
Electronic Desktop Diary Design Improvements – 

Task Order #24 

5-4 

 

calendar or typing in the date also seemed to work better for participants. Some participants also

thought that the icons helped to visualize and remember the categories that are used for data entry. 

The hardest features of this prototype for participants to adapt to concerned the navigation through 

multiple screens and the editing and removal of items. 

Figure 5-3. Prototype 2

In order to improve Prototype 2 and keep some of its more visually appealing elements, we suggest 

reducing the number of screens and making it easier to view the list of items already entered. One 

possible way to achieve these goals is to remove the expense review page. With this approach the 

user would select a date and select a category. Then the respondent would enter all of the possible 

items under the category like the current design; however, once finished entering, the respondent 

would be taken back to the expense date or home page where they would have to start over for the 

next type of expense. While there would be an extra step for selecting the date, respondents would 

be returned back to the home page every time where they would see the list of already entered 

expenses. This would make the editing and removal buttons much more visible and salient for the 

respondent. A cancel button from the expense date page could also return the respondent to the 

home page. The “Enter another type of expense for...” and “Enter a different type of expense...”

icons would no longer be needed. It would still be harder to adapt Prototype 2 for a mobile device 
Electronic Desktop Diary Design Improvements – 

Task Order #24 

5-5 

 

and this would take some extra effort. Reducing the size the category icons would help to make the 

design more amenable to a mobile device. 

In conclusion, we recommend Prototype 1 if the immediate need is to move forward with a design 

that is more or less ready to field and compatible with previous and current iterations of the CE web 

diary. Future research could be focused on creating a more visually appealing and modern design 

that motivates participants to understand the CE response process. It is possible that a hybrid 

approach between Prototype 1 and Prototype 2 is preferred. For example, the screen could be 

subdivided into three panels for entering or selecting different elements of the response task such as 

the date, categories using a smaller set of icons, and providing the description and other important 

information for the products. The list of entered expenses would be moved down to the bottom of 

the page. This type of design would work best on a desktop computer as was the goal for this task. 

In addition to usability testing, we would also recommend a field test to test the effect of any 

different designs on a broader set of objectives such as response to the survey. 
This page is left intentionally blank.
Electronic Desktop Diary Design Improvements – 

Task Order #24 

A-1 

 

Appendix A 

Usability Protocol 

Thank you for coming in today. Let me tell you a little about Westat and what we’re going to do 

today.

Westat is a company that conducts research on many topics, for different organizations. This 

particular project is for the Bureau of Labor Statistics. The Bureau of Labor Statistics conducts a 

study every year that asks people to record all of the purchases that all members of their household 

make for an entire week. Westat has been asked to test some different procedures to make it easier 

for people to report what they purchase. For this task today we will be asking you to record some 

purchases on the computer. Some of the purchases you will be entering will be on receipts that we 

have created for you and other times you will enter purchases from the receipts that you brought 

with you today. Your participation today will help us design a better system for people to enter their 

purchases in this study.

At this stage, we are still developing the study procedures to figure out how to make it as easy as 

possible for people to record their purchases. So that is what you’ll be helping us with here today. 

Do you have any questions about that?

Before we get started, I want to let you know that everything we cover in the interview will be kept 

private – only the few people actually working on the project will have access to the information you 

share with us. If there are any questions you don’t want to answer that is perfectly fine – We’re really 

more interested in how the questions we ask and tools were are using work.

If it’s OK with you, I would like to video record our conversation, just so that I can review it later 

when we’re analyzing all the interview data. (And finally, some of our colleagues from the Bureau of 

Labor Statistics are here today – behind the mirror – to observe and see for themselves how the 

study is going.)

CONSENT FORM: Here is a form I must ask you to look over and sign–it basically covers the 

points I’ve just gone over with you and indicates you have agreed to take part. (ONE COPY FOR 

RESPONDENT, ONE COPY FOR US)

BEGIN INTERVIEW:
Electronic Desktop Diary Design Improvements – 

Task Order #24 

A-2 

 

We would like people to enter in all kinds of purchases that they make. Here is a sheet of paper that 

shows different types of purchases that we would want people to report and how we want them to 

do that.

[REVIEW “TYPE OF PURCHASES” DOCUMENT. STRESS WHEN THE RESPONDENT 

SHOULD ENTER THE TOTAL RECEIPT (E.G., FOOD AWAY FROM HOME) AND 

WHEN THEY SHOULD ENTER INDIVIDUAL ITEMS (E.G., ALL OTHER EXPENSES).]

I’m going to be showing you two versions of a diary where you can enter your expenses. Here is 

what the first version looks like. 

[LOG IN AND DEMONSTRATE ENTERING EXPENSES FROM THE FOLLOWING 

RECEIPTS.]

Any questions? Let’s begin.

I’m going to give you some tasks that I would like to you do on the computer.

Super Grocery

1234 Montgomery Blvd.

Rockville, MD 20850

04/11/16

GROCERY Price

Wheat Bread $3.50

GROCERY NONEDIBLE

PaperTwls $2.85

Total $6.35

The Sandwich Shop

1234 Main Street

Rockville, MD 20850

04/11/16

QTY ITEM Price

1 Turkey Breast Sandwich $7.15

1 Med. Coke

1 Ham Sandwich $7.25

1 Med. Coke

Subtotal $14.40

Tax $0.86

Total $15.26
Electronic Desktop Diary Design Improvements – 

Task Order #24 

A-3 

 

Interview Protocol 

[NOTE TO INTERVIEWER] After each task ask the following question.

How easy or difficult was it to complete this task?

o Very easy

o Somewhat easy

o Neither easy nor difficult

o Somewhat difficult

o Very difficult

Task Task Name Text Read to Respondents

1. Login 1 First, I’d like you to enter your last name as the user name and enter 

“password1” for the password to log in to version 1.

2. Lunch Next, you will see a screen instructing you to enter an expense below. On 

[DATE] you go out to eat lunch with your child and order one meal for 

yourself and one meal for your child. Enter the expense into the diary. 

3. Groceries On the way home you stop at the grocery store and purchase some food. 

Enter these expenses in the diary.
Electronic Desktop Diary Design Improvements – 

Task Order #24 

A-4 

 

4. Shopping trip 1 On [DATE] you go shopping at a local discount store and purchase a 

chair, a video game for your child, and a book to read. You also stop to 

purchase gas and get your car washed on the way home. Enter these 

expenses in the diary. 

5. Return book You decide to return the book that you bought. Please go back and delete 

that item.

6. Medicine On [DATE] you stop at the pharmacy to purchase some medicine to treat 

your sore throat. You also pick up some personal care products. Enter 

these expenses in the diary.

ADMINISTER PROBES SPECIFIC TO VERSION 1
Electronic Desktop Diary Design Improvements – 

Task Order #24 

A-5 

 

Debriefing Questions for first prototype

Q1. What do you think about this version of the diary?

Q2. Was the computer diary easy or difficult to use?

o Extremely difficult

o Very difficult

o Somewhat difficult

o Neither easy nor difficult

o Somewhat easy

o Very easy

o Extremely easy

Q3. How confident did you feel in filling out the entries in the diary?

o Not at all confident

o A little confident

o Somewhat confident

o Very confident 

o Extremely confident

Q4. How much training do you think the average person would need to get started using the diary?

o None

o A little

o A moderate amount

o A lot
Electronic Desktop Diary Design Improvements – 

Task Order #24 

A-6 

 

Now I would like to show you another version of the diary. 

[DEMONSTRATE BY ENTERING ONE EXPENSE.]

7. Login 2 Now, I’d like you to enter your last name as the user name and enter 

“password1” for the password to log in to version 2.

8. Dinner Next, you will see a screen instructing you to select the day that some 

expenses occurred. On [DATE] you and your spouse go out to eat dinner 

at a restaurant. Enter the expense into the diary.

9. Movie You also go out to a movie that night after dinner. Enter this expense in 

the diary.

10. Shopping trip 2 On [DATE] you take the bus downtown to shop for some jewelry, 

cosmetics, and clothes. Enter the expenses in the diary.
Electronic Desktop Diary Design Improvements – 

Task Order #24 

A-7 

 

11. Return shirt You decide to return shirt that you bought. Please go back and delete that 

item.

12. Own expense Enter the first three purchases on the receipt that you brought with you to 

the interview.

ADMINISTER PROBES SPECIFIC TO VERSION 2

Debriefing Questions second prototype

Q1. What do you think about this version of the diary?

Q2. Was the computer diary easy or difficult to use?

o Extremely difficult

o Very difficult

o Somewhat difficult

o Neither easy nor difficult

o Somewhat easy

o Very easy

o Extremely easy

Q3. How confident did you feel in filling out the entries in the diary?

o Not at all confident

o A little confident

o Somewhat confident

o Very confident 

o Extremely confident

Q4. How much training do you think the average person would need to get started using the diary?

o None

o A little

o A moderate amount

o A lot

[ASK Q5-Q8 AFTER BOTH VERSIONS OF THE DIARY HAVE BEEN COMPLETED]
Electronic Desktop Diary Design Improvements – 

Task Order #24 

A-8 

 

Q5. Which version of the diary do you prefer–the first one or the second one? Why? (remind 

respondent what each one looked like if needed)

Q6. What was the most difficult when entering these expenses?

Q7. What was the easiest part of entering expenses?

Q8. Do you have any thoughts for how to make this easier? If yes, what are your thoughts?

INTERVIEWERS: If there is any time left, check with observers to see if they have any additional 

questions for this participant. If not, thank the participant and dismiss (give them their incentive and 

have them sign the receipt).
Electronic Desktop Diary Design Improvements – 

Task Order #24 

B-1 

 

Appendix B 

Screenshots of Final Prototypes 

PROTOTYPE 1

Sign-in screen.

Main data collection screen
Electronic Desktop Diary Design Improvements – 

Task Order #24 

B-2 

 

Example expense data collection fields (Food & Drinks Away From Home)

Example expense data collection fields (Food & Drinks For Home Consumption)
Electronic Desktop Diary Design Improvements – 

Task Order #24 

B-3 

 

Example expense data collection fields (Clothing, Shoes, Jewelry, & Accessories)

Example expense data collection fields (All Other Categories)
Electronic Desktop Diary Design Improvements – 

Task Order #24 

B-4 

 

Example expense enter and new expense data collection fields

Example edit of entered expense
Electronic Desktop Diary Design Improvements – 

Task Order #24 

B-5 

 

PROTOTYPE 2

Sign in screen

Expense day selection and summary expense screen
Electronic Desktop Diary Design Improvements – 

Task Order #24 

B-6 

 

Expense type selection

Example expense entry (Food and Drinks Away From Home)

Example expense entry (Food and Drinks for Home Consumption)
Electronic Desktop Diary Design Improvements – 

Task Order #24 

B-7 

 

Example expense entry (Clothing, Shoes, Jewelry, & Accessories)

Example expense entry (All Other Categories)

Example additional expense entry
Electronic Desktop Diary Design Improvements – 

Task Order #24 

B-8 

 

Review of entered expenses

Expense date entry with summary of entered expenses shown
Electronic Desktop Diary Design Improvements – 

Task Order #24 

B-9 

 

Example of other expense type question items