Should you reuse participants in your next usability study?
By: Nikita Chandawale
Tags: UX Research, Usability Testing, UX, Usability, User Research

Reusing participants certainly can be quite practical, convenient and honestly tempting, not just during current times. If you are working in an extremely niche or specialized product, it can difficult to recruit the right set of users. For example, it can be quite difficult to recruit participants for a product meant for law enforcement officers or paramedics or even wind turbine engineers.

A lot of people have the notion that reusing participants can reduce the participant’s anxiety as they know what to expect. Participants in this case will feel less stressed as they have been through one or more rounds of usability and know the drill. Besides, you would also have a rapport with them considering they have been moderated by you before. All of these factors would influence and improve data quality.

Despite all the advantages of reusing participants, however, the drawbacks are immense and outweigh the same.

The participants get comfortable with your product and research

Participants usually learn the struggles they went through the first time they participated in your research. In the following research, they would then look out for those learnt pieces and modify their actions accordingly. Participants will also recall the tasks you wanted them to perform and what was the workflow and predict the task based on that. Hence, the data you collect will not be accurate to the user’s natural behaviour and would be accounted for under learned behaviour. Further, you won’t be able to access the usability for a first time or an infrequent user.

Participants stop being users

The feedback from participants who have been part of more than one usability studies, tend to be more pinpointed. They evolve from being just your users to design critics. During repeat usability testing, these participants tend more towards providing an opinion than actually using your product/design. This results in collecting unactionable data.

Creating a biased view of users to your stakeholders

Stakeholders who attend your research usually recall the most vocal participant. The stakeholders, especially ones who are not familiar with user research, may start believing that these are the representative users. Further, if you continually test with the same participants, it's highly likely that the design is optimised for them than for a larger audience. The stakeholders will also develop a biased view of users, and will not be observant about the diversity in a larger set of users.

It goes against usability as a methodology

Usability testing usually has small sample sizes based on the assumption that the design process is done iteratively. The philosophy behind it is for each redesign, the design is tested with a new set of users. Hence, the sample size is then gradually built up through the iterations. For example, for the first iteration, the design was tested with 8 users. The following two iterations were tested with 8 respectively. Hence, the total number of users you tested the flow with is 24. On other hand, reusing participants would limit your sample size to just 8 in this case, which would be a very small user set.

Reusing participants can be quite pragmatic, considering you already have a pool of participants. In other situations, you may not have much choice in case it is quite difficult to find the right set of users. However, the pitfalls of using participants are overwhelming. Reusing participants doesn't affect just the current design but has implications for the future of the product as well. Hence, in my opinion, avoid reusing participants for your next usability study.