Case study: Uncovering problems and opportunities through usability testing
By: Jay Hahn
Tags: Usability, Case Study, User Testing, User Research, UX Research

Objective

To conduct usability tests on the ZigWay mobile app prototype to uncover problems and opportunities in visual and interaction design.

Heuristic Evaluation

First, I conducted a heuristic evaluation of the ZigWay mobile app prototype. This was done by analyzing the prototype in the categories of Learnability, Efficiency, Memorability, Error Management, and Satisfaction. For each, I assigned a score from 1–5 and explained reasons behind each assessment. This evaluation was useful as a jumping off point. From here, I would conduct further usability research to support or refute my initial analysis.

Learnability — 3/5 Efficiency — 4/5 Memorability — 4.5/5 Error Management — 4/5 Satisfaction — 4/5

Web Content Accessibility Guidelines (WCAG) Audit

Next, a WCAG audit was performed to analyze the ZigWay mobile app prototype for its accessibility features. The audit ran through several of the WCAG 2.1 requirements and checked the prototype to see if it met the requirement.

Proto-Persona

After conducting further research into ZigWay and their product, I created a primitive proto-persona to represent my initial assumptions regarding ZigWay’s target users’ demographics and behaviors. This proto-persona would be used as a tool in the early stages of development, and would be supported or refuted by later research.

A proto-persona for ZigWay’s target user

Usability Testing

In order to test the assumptions of the proto-persona and the initial heuristic evaluation, two stages of usability testing were conducted: 5 participants took part in informal, moderated usability tests through Zoom, and 3 participants took part in formal testing through Usertesting.com. The main demographics of the participants were 21–45 years old, annual income under $40,000, and familiar with using mobile applications.

Informal, Moderated Testing

I recruited 5 participants for informal, moderated usability testing over Zoom. Each usability test was allotted 30 minutes, with 5–10 extra minutes each for initial impressions and post-test review.

Within each usability test, the following user tasks were given to each participant:

Register for an account as a SuperUser.

Place an order for a Subscriber.

Collect payment from a Subscriber and send the collected payment to ZigWay.

Return oil containers and collect compensation for them.

Find a Subscriber’s order history.

Find the payment history.

Testing on UserTesting.com

Following the informal tests, I slightly revised my usability testing plan to focus in more on the problem areas identified by the first round participants. I submitted this revised testing plan to UserTesting.com to recruit 3 more participants. These usability tests were unmoderated, but the recordings and responses were later analyzed.

Usability Testing Plan on UserTesting.com

User Journey Map

The next step was to map the user’s journey while navigating the ZigWay mobile app prototype to visualize the user’s actions and experiences. The map shows each action that the user takes, their feelings and needs, and their overall satisfaction in each stage. The last section includes potential opportunities for improvement based on the pain points that the user faces in their journey through the app.

User Journey Map for San San Win (Persona)

User Insights

Initial Impressions

Based on the first impression of the Login and Home pages, all users gave feedback regarding changing the color scheme to reflect a more modern and professional image. With the knowledge that the ZigWay app would handle payments and important, everyday transactions, the users reported they would feel hesitant in using this app purely based on visual design.

Users also mentioned that the bottom navigation bar and many of the icons felt very intuitive because they felt familiar to them.

Usability

Similar to the navigation bar and icons, all users found the Login/Register processes intuitive because they had seen similar processes from many other apps.

One of the most commonly reported problem screens for users was the Order section. Many of the users felt that the Home and Order screens were slightly confusing because they weren’t sure which information or features to focus on first. Another point of confusion was that there seemed to be some level of overlap between main functions, such as the Orders and Subscribers sections. This overlap felt redundant for users, as they sometimes had to go through extra steps to complete the task at hand.

Lastly, some users were confused by the abundance of confirmation pages in the Order sections and the lack of a confirmation page in the Collect Payments page. While attempting to place an order for a Subscriber, many users thought they completed the Order process before actually reaching the final Submit page because they had already seen a Confirm Order button. Meanwhile, on the Collect Payments page, users felt that a confirmation page would be a critical error prevention step to catch any mistakes in payment amount entry.

Recommendations

The following recommendations are based on the synthesis and analysis of usability testing research: