# Remote User Testing

Lars Bo Larsen, Tina Øvad, Lene Nielsen, Marta Larusdottir

► To cite this version:

Lars Bo Larsen, Tina Øvad, Lene Nielsen, Marta Larusdottir. Remote User Testing. 18th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.579-583, 10.1007/978-3-030-85607-6_81. hal-04291186

HAL Id: hal-04291186

https://inria.hal.science/hal-04291186v1

Submitted on 17 Nov 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
Remote User Testing
- Experiences and Trends

Lars Bo Larsen<fnref n="1" />, Tina Øvad<fnref n="2" />, Lene Nielsen<fnref n="3" />, and Marta Larusdottir<fnref n="4" />

**Abstract.** This paper describes a one-day INTERACT 2021 workshop on remote testing with users as participants. Remote user testing has been around since the mid-nineties, but the Covid-19 pandemic has boosted the interest in remote working in general and thus also remote testing. The workshop aim is to present and discuss the current state-of-the-art of remote testing methods and identify emerging trends. Subjects that will be discussed include, but are not restricted to: Remote testing methods and platforms; capturing of quantitative as well as qualitative data; moderated, unmoderated and remotely moderated tests; testing of physical products; and remote testing of participants with special needs.

**Keywords:** Remote User Testing · Usability · Remote User Studies.

# 1 Introduction

Remote user testing has been around for quite a long time. Early studies were carried out about 25 years ago in the mid-nineties [1] [2] [3] and have since evolved significantly. While the 00'ies witnessed many academic studies and methods, this is particularly evident over the recent decade with the arrival of numerous commercial tools and platforms for automated remote testing. However, the Covid-19 pandemic has put an even greater focus on remote work and the paradigm is more relevant now than ever before.

The basic idea of remote user testing is that a facilitator is not physically present at the same place as the participant in the user test. Instead, the participant accesses a product, a service or a prototype via a web browser or an app on a computer or a mobile device. In some cases, a facilitator may be (remotely) available to guide the user through the test scenarios, debriefing, etc. This is called supervised remote testing or remote moderated testing [4]. An alternative is to carry out the remote test unsupervised. This will in part save manpower and in part make the logistics of the test much easier as the test participant can perform the test any time and place s/he wishes. Unsupervised tests tend to mostly collect quantitative data, whereas supervised tests can capture a larger degree of qualitative data.

<footnote n="1">¹ Aalborg University, Aalborg, Denmark tbl@es.aau.dk</footnote>

<footnote n="2">² Preely, Denmark tina@preely.com</footnote>

<footnote n="3">³ IT University, Copenhagen, Denmark lene@itu.dk</footnote>

<footnote n="4">⁴ Reykjavik University, Reykjavik, Iceland marta@ru.is</footnote>
## 1.1 Supervised Remote Testing

Supervised remote testing is also referred to as synchronous testing, as it requires the test participants and the facilitator to be present at the same time. Supervised remote testing was described by e.g. Dray and Siegel in 2004 [5]. They discussed various possibilities for remote usability testing and made a comparison to usability test conducted in a lab setting. 10 years later, Brush et al. investigated differences in the participants’ and facilitator’ qualitative experience between lab and remote tests [6]. They had assumed that “participants would be more comfortable talking to the facilitator and would find it easier to think aloud and concentrate on tasks in the local condition”. Contrary to the assumption, they found that most participants preferred the convenience of the remote testing approach. The findings suggested the remote test participants contributed to the results as much as in the lab-based test [6]. A special case is supervised testing with remote conductor - i.e. when an assistant or moderator participates at the user’s side. This could e.g. be when the participant have special needs, such as children, persons with physical or cognitive impairments, the elderly, or those not comfortable with the digital solutions surrounding us. In such cases a local conductor, typically from the participant’s household may be recruited to help execute the test.

## 1.2 Unsupervised Remote Testing

Unsupervised user testing is also referred to as asynchronous testing, as it does not require the test participants and the facilitator to be present at the same time. Bruun et al. compared three different methods in an unsupervised remote testing scenario: user-reported critical incidents; forum-based online reporting and diary-based user reporting [7]. They found that the results gathered by conducting the unsupervised testing were significantly sparser compared to results gathered by conducting lab-based usability tests. Despite this, unsupervised remote testing has gained a large foothold during recent years, due to its ease of deployment and corresponding cost-efficiency. As with supervised remote testing, this could also be carried out with a remote conductor.

## 1.3 Remote Testing Platforms

A number of companies have built platforms to facilitate remote user testing over the recent decade. One such platform supplier is the Danish company Preely [8]. The platform lets the interaction designer develop prototypes using a variety of tools, such as Sketch, Figma, Invision or Adobe XD [9] [10] [11] [12] and then deploy them for test via the platform. Preely will import and execute the prototype and do all the bookkeeping and logging of user test data etc. for any number of test participants.

Other similar platforms are online services such as: Maze, UserTesting and UseBerry [13] [14] [15] for unsupervised testing. Similarly, consultancy services, like UserTribe, UserZoom and LookBack offer supervised remote tests [16] [17]
[18]. These platforms are insight-based and qualitative and demands more re-
sources post-test to make the analysis than the unsupervised ones.

1.4 Remote testing of physical products

However, the present state-of-the-art of remote user testing methods and plat-
forms are restricted to software products. At the present time it is relatively
straight-forward to distribute software products, but if the product being tested
is a physical device or a service, a number of new issues arise. Many have to do
with the costs and logistics of distributing and later collecting the test products
or prototypes. Others are directly concerned with the test session itself, such as
making sure the user's camera is pointed towards the product being tested.

## 2 Workshop Scope and Aim

The purpose of the workshop is to assess the latest trends within remote user
testing. In particular, to record and review the present state-of-the-art of meth-
ods and tools and assess the experiences gained during the Covid-19 pandemic.
We will seek to answer the following questions:

- Which new experiences have the Covid-19 pandemic brought with regards to remote testing?

- Has the greatly increased focus on remote work during the pandemic also brought new ideas and methods to the field of remote testing?

- Have new requirements emerged, to which no solution yet exists?

- Are existing platforms and tools sufficient for the future demands?

- Will remote testing extend from software to physical products?

- How can unsupervised remote testing to a higher degree than now provide qualitative insights?

- Does remote testing cater sufficiently for user groups with special needs – such as children, persons with physical or cognitive impairments, the elderly, or those not comfortable with the digital solutions surrounding us?

The workshop invites papers sharing remote testing experiences as well as
position papers bringing up a particular issue or focus. Contributions from in-
dustry are particularly welcome and could be in the form of a paper or a one-two
page industry abstract. The workshop layout will be a mix of presentations from
participants and group and plenary discussions (as described below).

The goal of the workshop will be to achieve an overview of the present state
and challenges and provide a roadmap for the coming years' advances of remote
testing.

3 Expected Outcome

We expect that the workshop will identify the current state-of-the-art and trends
of remote user testing. In particular, the workshop will aim to pinpoint the
challenges and unresolved problems facing the remote testing paradigm in the
future and provide a proposal for a roadmap to address these.
**3.1 Review process and proceedings**

The paper contributions to the workshop will be peer reviewed by the organis-
ers. Abstracts and industry contributions will be screened for relevance to the
workshop topic. These will form the proceedings of the workshop and will be
shared among participants prior to the workshop via the workshop website. The
proceedings will be made public after the workshop for those authors who want
that. The organisers will invite the participants who are interested to collaborate
to publish a joint paper on the workshop results in a recognised peer-reviewed
open access journal.

4 Organisation of the Workshop

The workshop will be based on active participation and discussions. Presenta-
tions will be organised in two main sessions, with a priority for discussions.

The aim is for 10 to 15 participants, to ensure a workshop atmosphere and
informal style, where all have a chance to share viewpoints and engage actively in
discussions. The workshop will take place in mixed-mode physically and virtually,
due to expected Covid-19 travel restrictions, at the time of the workshop. The
workshop will be split into two main sessions:

- **The morning session** will be devoted to presentations by the participants and discussions based on the shared experiences and challenges given by the facilitators. All participants are expected to take active part in this and contribute with stories about good/bad experiences of remote user testing.

- **The afternoon session** will focus on identifying trends and creating a set of expectations and challenges for remote user testing for the upcoming years.

The participants will form smaller groups for parts of the discussions with one representative from each group to give a brief account of what has been discussed. To ensure the interactions between the participants, the groups will present their conclusions in plenum and be reformed after each session. To accommodate those present virtually, emphasis will be made to draw them into discussion and tools such as Miro [19] will be employed for notes and documentation instead of whiteboards and post-its.

The workshop will be moderated by the organisers, who are all experienced
facilitators and expect to be present at the conference. They will act as presenters
and session chairs, moderate the discussions, etc. As a follow-up, the findings
and conclusions reached at the workshop will be published.

5 Workshop organisers

Brief introduction of the organisers:

- Lars Bo Larsen is associate professor at Aalborg University. His main re-
search and teaching interests are interaction design and user experience re-
search, particular-ly in industrial contexts.
- Tina Øvad has a background in Engineering Psychology and a PhD in Information Systems with focus on agile UX. Tina is CXO at Preely and External Lecturer at Aarhus University.

- Lene Nielsen is associate professor at IT university Copenhagen, department of Business IT. Her main research and teaching are in user experience research, in particular personas.

- Marta Lárusdóttir is an associate professor in the Computer Science Department at Reykjavik University, Iceland. Her main research area has been user centred design, especially focusing on usability and user experience evaluation.

# 6 References (all links visited June 11 2021)

## References

1. Castillo, J. C., Hartson, H. R., and Hix, D.: REMOTE USABILITY EVALUATION AT A GLANCE * (Tech. Rep.) (1997). https://doi.org/10.1145/286498.286736

2. Hammontree, M. L., Weiler, P., and Nayak, N. P.: Remote usability testing. Interactions, 1, pp.21-25 (1994).

3. Hartson, H.R., Castillo, J.C., Kelso, J., Neale, W.C., and Kamler, J.: Remote evaluation: the network as an extension of the usability laboratory. In Conference on human factors in computing systems - proceedings (pp. 228-235). New York, New York, USA (1996): ACM Press. https://doi.org/10.1145/238386.238511

4. Safire, M.: Remote moderated usability: What, why and how. 12th Annual Usability Professionals Association Conference, Phoenix, AZ (2003).

5. Dray, Susan and Siegel, David: Remote possibilities? International usability testing at a distance. Interactions. 11. 10-17 (2004).

6. Bernheim Brush, A. J., Ames, M., and Davis, J.: A comparison of synchronous remote and local usability studies for an expert interface. Conference on Human Factors in Computing Systems - Proceedings (April 2015), 1179-1182. https://doi.org/10.1145/985921.986018

7. Bruun, A., Gull, P., Hofmeister, L., and Stage, J. (2009). Let your users do the testing: A comparison of three remote asynchronous usability testing methods. In Conference on human factors in computing systems - proceedings (pp. 1619-1628). New York and NY: ACM Press. https://doi.org/10.1145/1518701.1518948

8. Preely homepage: https://preely.com

9. Sketch homepage: https://www.sketch.com/

10. Figma homepage: https://www.figma.com/

11. InVision homepage: https://www.invisionapp.com/

12. Adobe XD homepage: https://www.adobe.com/products/xd.html

13. Maze homepage: https://maze.co/

14. UserTesting homepage: https://www.usertesting.com/

15. UseBerry homepage: https://www.useberry.com/

16. UserTribe homepage: https://usertribe.com/

17. UserZoom homepage: https://www.userzoom.com/

18. LookBack homepage: https://lookback.io/

19. Miro homepage: https://miro.com/online-whiteboard/