Collaborative design methods
By: Kathiravan Subramanium
Tags: Collaboration, Design, Razorpay Design, Process, Ideation

5. User testing

Stakeholders

Designers, Users

Duration

45–60mins

Conducted for

Understanding the user acceptance

We conducted this exercise after we had created some final prototypes of the product. Before passing them on to the tech team for implementation we wanted to validate them with the users. In our case, we had created two iterations with slightly different approaches. So in order to validate the approach, we had invited 7–10 users as per the identified persona for the testing.

Preparation

Prototype

We had made click-through prototypes of both iterations on Figma. We tried to cover as many critical tasks flows as we can. We have also ramped the visuals with some brand elements so that it gives a finished feeling to our users. This is just to make sure the users do not have a very different experience from our existing product suite.

Script

We had created an interview script to maintain the flow of questions in the same way between different interviewers. This helped us to avoid any delta this could cause in the outcomes of the testing. We had added statements to make sure the user is comfortable about giving explicit feedback on the designs presented.

Tasks

From the identified user journey we had devised a set of tasks that they are expected to complete using the prototypes presented. These tasks are very situational and may not directly convey what they are expected to do. This helped us to understand the natural task flow which sometimes happens outside the product as well.

Metrics

We wanted to quantify the outcomes for a better understanding, here I have followed the criteria mentioned by Matej Latin in his article on Measuring and Quantifying User Experience

Success Criteria / Completion (C1)–0,1,3

For each task, we defined the success criteria to measure the completion of that task. This is rated based on the time taken to complete a task and on whether the participant was able to complete the whole flow of the given task.

Complexity(C2) — 1,2,3,4,5

This rating is to mark how complex the task is, was measured either based on the number of steps the user has to take to complete the given task or on the discoverability of the feature.

Criticality(C3) —1,2,3

This rating is based on the priority of the task, was derived from the feature framework we discussed before in this article.

:: During the session ::

Testing (30 mins)

We conducted these sessions as a combination of in-person and remote.

Having a recording of these sessions (with consent, of course) helps in post-session analysis.

After understanding the user’s business nature and the operations (as it was a B2B product), we presented the iterations and asked them to initially spend a couple of minutes to explore and understand the different sections of the application. At this stage, we asked the users to be vocal about what they think each section of the app did. We presented the tasks after this.

General Feedback (15 mins)

Once they had performed all the tasks, we asked them for overall feedback on the product and also asked them for three things that they wished they had in the application. This last point gave us inputs on which direction to take next.

Calculating Usability score

Post the session we had collected those scores as decided and calculated the usability score of a particular task with this formula.

Usability score = (Completion + Complexity) * Criticality

For example :

Task — To sign up and create an account

Completion = 1 (Total 7 steps, the user was able to reach till 4th) Complexity = 3 (Very linear process but the user has to add a lot of documents) Criticality = 3 (High)

Usability Score = 12

At the scale on which we conducted the exercise, this method did not add a lot of value as we were able to clearly see a pattern evolving in the ways the users have approached the tasks. But this might be very useful in cases where the product involves a lot of flows and it gets very subjective to evaluate with qualitative methods.