# How Do People Position Assistive Information During Physical Tasks? Insights for Designing Cross-Domain AR Systems

Yue Shin Koh, Gilles Bailly, Ignacio Avellino

► To cite this version:

Yue Shin Koh, Gilles Bailly, Ignacio Avellino. How Do People Position Assistive Information During Physical Tasks? Insights for Designing Cross-Domain AR Systems. IHM'25 - 36e CONFÉRENCE INTERNATIONALE FRANCOPHONÉE SUR L'Interaction Humain-Machine, Nov 2025, Toulouse, France. 10.1145/3765712.3765733. hal-05294646

HAL Id: hal-05294646

https://hal.science/hal-05294646v1

Submitted on 2 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# How Do People Position Assistive Information During Physical Tasks? Insights for Designing Cross-Domain AR Systems
## Comment positionner les informations d'assistance pendant les tâches manuelles : perspectives pour la conception de systèmes de RA inter-domaines

Yue Shin Koh

Sorbonne Université, CNRS, INSERM,
Institut des Systèmes Intelligents et
de Robotique, ISIR
Paris, France

yue-shin.koh@sorbonne-universite.fr

Gilles Bailly

Sorbonne Université, CNRS, Institut
des Systèmes Intelligents et de
Robotique, ISIR
Paris, France

gilles.bailly@sorbonne-universite.fr

Ignacio Avellino

Sorbonne Université, CNRS, INSERM,
Institut des Systèmes Intelligents et
de Robotique, ISIR
Paris, France

ignacio.avellino@sorbonne-universite.fr

## Abstract

We report on an observation study with 12 participants investigating task characteristics that affect how people position assistive information during bi-manual physical tasks without Augmented Reality (AR). We observed a range of physical tasks, including cooking, assembly, woodworking and crochet, with our inclusion criteria being that they required the use of both hands and access to non-AR assistive information for successful completion. Our main findings are that during bi-manual physical tasks (1) AR systems should consider displaying assistive information using multiple methods at the same time or at different times and (2) task characteristics such as type of mobility (predictable vs unpredictable), and frequency of information access affected the position that people chose. We discuss how these findings encourage the composition of existing positioning methods in space and time to inspire the design of AR systems that display assistive information.

## Résumé

Nous rendons compte d'une étude d'observation menée auprès de 12 participants et portant sur les caractéristiques des tâches qui influent sur la manière dont les personnes positionnent les informations d'assistance au cours de tâches physiques bi-manuelles sans Réalité Augmentée (RA). Nous avons observé une série de tâches physiques, notamment la cuisine, l'assemblage, et le travail du bois, nos critères d'inclusion étant qu'elles nécessitaient l'utilisation des deux mains et l'accès à des informations d'assistance non-RA. Nous concluons que lors de tâches physiques bi-manuelles : (1) les systèmes de RA devraient afficher des informations d'assistance en utilisant plusieurs méthodes, soit en même temps, soit à des moments différents et (2) les caractéristiques des tâches telles que le type de mobilité (prévisible ou imprévisible) et la fréquence d'accès

à l'information affectent la position choisie par sujets. Nous expliquons comment ces résultats encouragent la combinaison de méthodes de positionnement existantes dans l'espace afin d'inspirer la conception de systèmes de RA affichant des informations d'assistance.

## CCS Concepts

* **Human-centered computing** → **Empirical studies in interaction design; Mixed / augmented reality; Field studies.**

## Keywords

Augmented Reality, View Management, Observation, Content Placement

## Mots clés

Réalité augmentée, gestion des vues, observation, placement de contenu

## ACM Reference Format:

Yue Shin Koh, Gilles Bailly, and Ignacio Avellino. 2025. How Do People Position Assistive Information During Physical Tasks? Insights for Designing Cross-Domain AR Systems. In *The 36th Conference on l'Interaction Humain-Machine (IHM '25), November 03-07, 2025, Toulouse, France*. ACM, New York, NY, USA, 14 pages. https://doi.org/10.1145/3765712.3765733

## 1 Introduction

Augmented Reality Head-Mounted Displays (ARHMDs) enable users to physically interact with the real world while providing digital information on-demand [3]. This is especially promising for complex bi-manual physical tasks where assistive information (e.g., instructions, demonstrations, technical diagrams) is essential to the task's successful performance. For example, we might expect a bike mechanic to refer to part schematics during a repair, or a surgeon to medical imagery during surgery. However, during these tasks, people can experience difficulty interacting with assistive information, delivered on paper or through digital devices, because their hands are encumbered in some way, e.g., dirty whilst cooking [19], sterile during surgery [41] or busy during assembly [52]. In addition to facilitating hands-free interaction, ARHMDs also allow users to position assistive information *everywhere*, free from

<figure><img src="image_2.png" /></figure>

This work is licensed under a Creative Commons Attribution 4.0 International License.
IHM '25, Toulouse, France

© 2025 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-1522-8/25/11
https://doi.org/10.1145/3765712.3765733
real-world constraints, i.e., mid-air. Furthermore, the digital nature of the information allows it to be more dynamic, with the cost of changing its content and position being significantly lower than that of analogue mediums. Given these possibilities, it remains uncertain how best to display assistive information while performing complex physical tasks.

Existing work has explored a wide range of methods for displaying UIs in AR including: on the users' wrist [38], in peripheral vision [37], next to [16, 47, 56] or directly overlaid on the task area [6, 17, 52], and anchored to movable objects of interest such as a surgical needle [53], kitchen implements [26] or everyday items [50]. These methods have been evaluated against AR and non-AR display techniques, demonstrating improved task performance time [43], error [6] and interaction count [38], amongst other metrics. However, the underlying assumption in these works is that for one task there is one “best” method. To the best of our knowledge, research considering situations where multiple methods at the same or different times might be useful during a bi-manual physical task is limited. To explore the idea of there being multiple “best” methods over the course of a task, our approach is to address the gap in understanding how task characteristics affect the way people display of non-AR assistive information during physical tasks. We use this to inform the design of an adaptive AR system, with the assumption that these factors may also lead users to change methods in AR. We conducted an observational study where we observe 12 people performing bi-manual physical tasks without AR technology. We then identify the factors that motivate how they display assistive information and examine how they can be applied to novel AR setups. We choose to observe tasks across several domains in order to identify non-domain-specific triggers across a range of bi-manual physical tasks that lead people to change the position of assistive information.

Our findings suggest that for the display of assistive information during complex bi-manual tasks, when tasks share certain characteristics, people exhibit similar positioning behaviors in response. Given this, our work finds that depending on whether the **type of mobility** in a task was predictable, people were more likely to use Relative-to-World (RtW) positioning, while unpredictable movement gave rise to more Relative-to-Body (RtB) and Relative-To-Object (RtO) positioning. Furthermore, we found that the higher the anticipated **frequency of information access** the more likely people were to place their information at a lower access cost, therefore favouring RtB positioning. While at a lower frequency of access their other priorities were revealed, notably a desire to reduce clutter.

These findings emphasize that the position of assistive information is a consideration that is in constant flux, a balance between many factors that people constantly prioritize and re-prioritize as they work. The implications of this on the design of cross-domain systems are that (1) AR systems should consider using multiple methods at the same or different times, and, (2) Task characteristics such as **type of mobility** and **frequency of information access** influence the positioning method that people use. As such, this research contributes to disentangling the complexity of human behavior when performing a physical task to better design AR systems that can detect and accommodate them.

# 2 Related Work

We first examine literature tackling the **interaction constraints during bi-manual physical tasks**, showing that while novel touchless or handsfree interaction techniques can address the physical cost of interaction, they do not address the cognitive cost of managing system interaction during the task. We then further examine work on **cognitive motor interference**, highlighting that the cognitive requirements of the task can interfere with motor performance. Lastly, we posit that work on the **display of assistive information in AR** has focused on comparing strategies for displaying information in AR to find the “optimal” strategy rather than considering that the optimal strategy may change *during* tasks.

## 2.1 Interaction Constraints during Bi-manual Physical Tasks

The cost of interacting with assistive information during bi-manual physical tasks can be broadly categorised into **physical cost** and **cognitive cost**.

The physical cost of interaction comes from the amount of hand encumbrance presented by the task, affecting people's ability to interact with any assistive information they might need with their hands, digital or otherwise. A surgeon would need to leave the sterile field to look through pre-operative images, a cook would have to wash their hands before referring to a recipe, a mechanic lifting a heavy part with both hands might be unable to refer to its installation instructions at the same time. Existing work has attempted to address this cost through novel handsfree or touchless interaction techniques, exploring the use of voice [41], eyes [24], hand gestures [25] and feet [23] for interacting with systems. However, while these techniques reduce the physical cost of interaction by delegating away from the hands, they do not address the cognitive cost of thinking about executing the interaction and the mental workload of switching between understanding the assistive information and applying that understanding to their physical task.

The cognitive cost has been explored by existing work showing that simultaneous cognitive and motor tasks can negatively impact each other's performance, a phenomenon called cognitive-motor interference [2]. This is supported by existing work highlighting the negative impact of visual and cognitive focus shifts on physical task performance in surgery [5], manufacturing [52] and whilst walking [44]. Rzayev et al. [48] showed that the position and alignment of notifications in AR significant affected the balance of attention between virtual and physical environments. Other work [27, 32] has shown that changing the display position of notifications can affect users' ability to maintain focus on primary tasks while receiving secondary information in AR. However, they have focused on the delivery of secondary information *optional or unrelated* to the task, rather than assistive information *essential* to the successful completion of the task as we will investigate in this paper.

## 2.2 Displaying Assistive Information in AR

Despite the potential of ARHMDs to reduce the physical and cognitive cost of managing both the primary task (physical task) and the secondary task (accessing assistive information) [43], it remains uncertain how to display assistive information in AR when performing complex physical tasks. We first examine how work on displaying
of non-AR assistive information during bi-manual physical tasks supports the potential of adaptive and in-situ positioning in AR. Next, we examine work that proposes frameworks to describe the position of 2D UIs in 3D space. We then outline different strategies that have been used to display of assistive information in AR for physical tasks, highlighting the literatures' focus on comparing them to find the "optimal" strategy, without considering that this may change during tasks. Finally, we consider how adaptive interfaces might leverage this understanding of a users' task to reduce the physical and cognitive cost of system interaction during it. As such, a better understanding of the specificities of the task that an AR system should accommodate is required.

### 2.2.1 Displaying Non-AR Assistive Information during Bi-manual Physical Tasks.
When examining the display of non-AR assistive information during bi-manual tasks, we see that position is considered more as a limitation to be worked around rather than an opportunity for optimisation [29, 40]. This is likely because of the limited possibilities for adaptive positioning in the real world. However, even with non-AR assistive information, some studies show the value of adaptive positioning during a task. Mentis [40] found that the arrangement of fixed displays showing medical images, affected the positions that surgical staff took in the operating room and created different information access behaviours. This highlights how the position of assistive information is essential to the task and impacts how the task is performed. Kolbeinsson et al. [29]'s work investigating information display preferences for non-AR assembly instructions in industrial settings, found that if accessing instructions required extra work, such as frequent to and frohs, workers would rely on memory and experience to reduce this inconvenience. These works highlight the potential for in-situ AR instructions to have an impact on task execution behaviour by allowing information to be present and accessible without additional movement. This reinforces the value of supporting adaptive positioning with AR.

### 2.2.2 Displaying 2D UIs in 3D Space.
ARHMDs let users position information anywhere in 3D space, facilitating positions that would be difficult or otherwise impossible to achieve in the real world, i.e., mid-air, attached to the user, attached to objects. As such, the description of these new possibilities for the position of 2D user interfaces in 3D space is a challenge that existing literature has tackled, proposing a variety of frameworks [4, 11, 14, 35]. Most commonly, they are defined relative to something, which is then the frame of reference. As such in this work, we use the term Reference Frames (RF) as a general term to describe the way 2D UIs are positioned in 3D space in terms of the entity to which they are anchored. In the literature, the terminology used to describe Reference Frames (RFs) is fragmented, using a variety of terms to describe similar concepts. In our work, we will use the same terminology as Cockburn et al. [9]. Relative-to-World (RTW) methods position information affixed to static elements in the environment. Relative-to-Body (RTB) methods position information anchored to the users' movement. Relative-to-Object (RTO) methods position information pinned to an object, such that when it moves, the information also moves.

### 2.2.3 Displaying AR Assistive Information for Physical Tasks.
Existing work on the display of assistive information in AR has considered a range of physical tasks: industrial assembly [51], LEGO assembly [16], surgical needle insertion [53], cooking [26], car repair [18]. They have explored a range of positions for the information, spanning all three reference frames. RtB strategies include: in the centre of their field of vision [16], on the user's wrist [38], in their peripheral vision [37]. RtW strategies include: next to [16, 47, 56], or directly overlaid [6, 17, 52] on the task performance area. RtO strategies include: relative to the moving components of the task, i.e., surgical needle [53], kitchen implements [26] and welding gun [10]. Furthermore, displaying assistive information in AR has been shown to improve error incidence [6, 55], task performance time [43], cognitive load [16] and usability [30], when compared to non-AR display techniques.

Although this literature has evaluated and compared different positions for assistive information across different physical task domains, they have afforded limited consideration to how the most effective position might change *during* the task. Instead, the underlying assumption is often that there is one “best” position per task, validated experimentally by comparing their performance according to a certain metric *i.e.* task completion time. For example, Robertson et al. [47] found that for a LEGO assembly task, graphical instructions displayed aligned to the task outperformed displaying them next-to the task area, fixed to the users’ view and in their peripheral view. However, we can see that a user wearing an ARHMD displaying assembly instructions may want instructions to be positioned differently when they are moving around retrieving parts than then they are stationary assembling them. As such, to the best of our knowledge, research considering situations where multiple methods might be useful during a bi-manual physical task is limited and correspondingly an understanding of the task characteristics that might change the “optimal” position is lacking.

### 2.2.4 Adaptive Interfaces to tackle Physical and Cognitive Cost.
Adaptive interfaces that automatically adjust how information is displayed in AR according to certain contextual factors have seen users report feeling less distracted (cognitive cost) [34] and been effective in reducing the number of interactions users make (physical cost) to achieve their information goal [38]. Existing context-adaptive interfaces have adapted information position [12, 33, 42] to factors of user context (e.g., cognitive load [34], or eye gaze [15]) and environmental context (e.g., desk layout [42], object geometry [22], room and social configuration [33]). However, adaptation to characteristics of “what the user is doing” or task context is less explored, despite being a category of context in several taxonomies [1, 20, 28, 49, 57]. For example, work on object-anchored interfaces [10, 50, 53] that display assistive information, demonstrates a level of adaptation to the movable components of the task, but this represents just one component of a wider task context that could be leveraged. To the goal of making information available “*at the right place, at the right time and at the right level of detail*” [34], this work aims to contribute to a better understanding of what “right” is. As such, a greater understanding of how to deconstruct and operationalise the task context of physical tasks into a usable contextual input for a context-adaptive system is needed.
<table><caption>**Table 1:** Overview of observed tasks. Methods used by participants to display information are indicated by the reference frames (RF) they used. We use Cockburn et al. [9]'s definition of reference frames for 3D UIs: relative to world (W), relative-to-user (U) an relative-to-object (O). Some participants used multiple methods within the same reference frame.</caption><thead><tr><td>ID</td><td>Task</td><td>Description</td><td>Display Technology</td><td>Mobility</td><td>Methods Used (by RF)</td><td>Participant Expertise</td><td>Duration (mins)</td></tr></thead><tbody><tr><td>1</td><td>Lego Assembly</td><td>Assembled the internal structure of a LEGO model</td><td>Paper</td><td>Stationary</td><td>W</td><td>Competent</td><td>29</td></tr><tr><td>2</td><td>Bed Assembly</td><td>Assembled a bed frame using its manual</td><td>Paper</td><td>Mobile</td><td>W, U</td><td>Competent</td><td>46</td></tr><tr><td>3</td><td>Cooking</td><td>Cooked a gratin dauphinoise following a recipe</td><td>Smartphone</td><td>Mobile</td><td>W, U</td><td>Proficient</td><td>82</td></tr><tr><td>4</td><td>Bike Maintenance 1</td><td>Performed routine maintenance on a road bike</td><td>Laptop</td><td>Mobile</td><td>W, U</td><td>Expert</td><td>44</td></tr><tr><td>5</td><td>Bike Maintenance 2</td><td>Recalibrated the electronic gears of a high performance road bike</td><td>Paper, Tablet, Smartphone</td><td>Mobile</td><td>W, U</td><td>Expert</td><td>63</td></tr><tr><td>6</td><td>Crochet</td><td>Crocheted a granny square following an online pattern</td><td>Smartphone</td><td>Stationary</td><td>W</td><td>Advanced Beginner</td><td>28</td></tr><tr><td>7</td><td>Needle Felting</td><td>Built the wire skeleton and felted the body of a model bat.</td><td>Paper</td><td>Stationary</td><td>W</td><td>Competent</td><td>43</td></tr><tr><td>8</td><td>Woodworking</td><td>Cut and joined wood to build a storage box.</td><td>Paper</td><td>Mobile</td><td>W, U, O</td><td>Proficient</td><td>144</td></tr><tr><td>9</td><td>Laser Cutting</td><td>Cut a digital design onto plywood with a laser cutter</td><td>Paper, Desktop</td><td>Mobile</td><td>W, U, O</td><td>Advanced Beginner</td><td>103</td></tr><tr><td>10</td><td>Inventory</td><td>Checked warehouse stock of cables against their records.</td><td>Paper, Desktop</td><td>Mobile</td><td>W, U</td><td>Advanced Beginner</td><td>26</td></tr><tr><td>11</td><td>Electronics Testing</td><td>Tested performance of newly assembled motherboards</td><td>Specialised device</td><td>Stationary</td><td>W</td><td>Competent</td><td>33</td></tr><tr><td>12</td><td>Gym workout</td><td>Followed a preset workout plan at the gym</td><td>Smartphone, Smartwatch</td><td>Mobile</td><td>W, U</td><td>Proficient</td><td>56</td></tr></tbody></table>

## 3 Method

**Participants.** We recruited 12 unpaid participants (7 male, 5 female), aged 18-44 through word of mouth, leveraging our university network, as well as approaching local businesses. A purposive sampling method was employed. Across a variety of domains, we reached out to potential participants known to have experience with them and discussed tasks they perform that might match our interest. Participants were required to self-identify as a non-beginner, having prior experience in the task they were being observed performing. This experience was not required to be professional. We define two criteria for tasks: (1) require the use of both hands, and (2) require the use of non-AR assistive information (digital i.e., smartphone, tablet, or, non-digital i.e., paper) during the task for successful completion. The variety of tasks that meet this criteria is large and we could not observe all of them. As such, the tasks we were able to observe for our study were limited by our recruitment capability. In a similar vein, all the assistive information we observed being used were 2D, this was not an intentional choice, but a function of the tasks and participants we were able to recruit. Table 1 summarises participants and tasks observed in this study. This study received approval from our institution's ethics committee.

**Observation & Protocol Design.** We chose to conduct a contextual inquiry [46], interleaving observation with focused, situated interview questions concerning the task at hand and the role of assistive information. This was because leveraging our participants' domain expertise was especially important in a study that spans various domains. The researcher took an overt, non-participant role, taking systematic observation notes in the format of a predefined observation grid, whilst observing the participant in their natural context. We designed the observation grid to help the researcher focus their

notes on capturing the contextual elements relevant to any observed task action or interaction with assistive information. It had the following headings: (1) Action/Interaction, (2) Environmental Context, (3) User Context, (4) Task Characteristics, (5) Information/Device characteristics. This aligns with existing categorisations of context [1, 20, 28, 49, 57] that have environment, user and task as categories. We add a field for information/device characteristics. Observational data was supplemented by a semi-structured interview (SSI) conducted immediately post-observation, in order to understand the explicit reason behind participants' observed behaviour. The SSI was also designed to continue the researcher-participant partnership of a contextual inquiry, leveraging each participant's domain expertise to help the researcher contextualise and interpret their observations.

**Procedure.** The procedure was as follows: (1) A declaration of consent; (2) Observation of the participant performing the task; (3) A post-observation semi-structured interview. We observed until the task reached a natural conclusion. Observations lasted 26-144 minutes (M = 58 minutes, SD = 35.64 minutes).

**Data Collection & Analysis.** For each observation, we took field notes by filling out the predefined **observation grid** digitally on the researcher's laptop. We took **photos** using a smartphone to document the task environment and key moments of the participants' task performance. We recorded **audio of the semi-structured interviews** which were then transcribed. The interviews consisted of three parts. First, we asked any clarifying questions about the task, giving the participant an opportunity to explain what they were doing in more detail. Second, we raised examples of their information positioning behaviour during the task and questioned their underlying thoughts and motivations. Finally, we gave the
<figure><img src="image_3.png" /></figure>

(a) P10 demonstrates RtW positioning by putting their reference sheets on a stack of boxes and two rolls of coiled wire that they had stacked to elevate the papers. As they moved around the warehouse they would regularly return to these locations to refer to the sheets.

<figure><img src="image_4.png" /></figure>

(b) P12 demonstrates RtB positioning of their phone displaying workout videos by constantly adjusting its position to be in their eyeline as they exercised

<figure><img src="image_5.png" /></figure>

(c) P9 demonstrates RtO positioning when they wrote the measurements of a deck of cards onto a post it that they stuck onto the deck

<caption>**Figure 1:** Examples of assistive information (outlined in red) positioned in the different reference frames from our observations.</caption>

participant a more comprehensive overview of our research subject, followed by direct questions pertaining to our interest in how and when assistive information might be positioned using different methods during tasks. The goal of this was to elicit relevant anecdotes from participants' experience where they saw frictions or opportunities that could be addressed, harnessing the participants' domain expertise to generate insight. The interview guide with the structure, example questions and question templates is available in Appendix A.

The first author conducted data analysis. Observation grids and interview transcripts for each task were considered in tandem. Each observation grid was coded bottom-up, creating short descriptions of any instruction positioning and movement that occurred. Then we read through the corresponding interview, augmenting our codes with any insight into the context or motivation that they contained. This left us with one set of augmented codes for each task. Next, we used the Affinity Diagram method [39] gathering codes across all tasks into a digital whiteboard as individual post-its and clustering first by direct similarity, then by conceptual relevance. At this stage, the first author went through several iterations of clustering, trying to develop a useful higher-level framework with which to describe the data. First, we created groups based on whether codes described positioning information relative to the user, task or environment. Then, we tried arranging codes along two dimensions: readability and reachability. Next, we tried organising codes by the type of movement (eye, head, hand or walking) required to engage with it. Finally, we settled on differentiating our codes based on the reference frame that they used.

# 4 Results

During our observations, participants performing stationary and seated tasks exhibited much more limited positioning behaviours compared to those performing mobile tasks where they had to walk around a space. Given this and the nature of our research question, we focus on reporting the more diverse positioning behaviours

exhibited by participants whose tasks were not fully stationary. In all the mobile tasks, we observed assistive information being moved and displayed in different positions using all three reference frames. Without AR, RtB positioning methods that dynamically adjusted to user movement—i.e., following head/eye gaze or hand movement—were not observed. Instead, RtB positioning was only observed insofar as the participant made the effort to bring the information with them as they performed their task. Figure 1 depicts how we considered the information to be positioned in each reference frame using examples taken from our observations.

Our main finding is that participants changed the position of assistive information throughout the course of complex bi-manual physical tasks. This supports our initial premise that there is rarely one best position for this information during the whole task. We further identified that participants adjusted the position of assistive information in response to the (1) Frequency of Information Access and (2) Type of Mobility. Table 2 summarises our findings, indicating the preferred reference frame or key decision-making factor in each case.

## 4.1 Information Position Changes According to Frequency of Information Access

The frequency with which participants anticipated needing assistive information at any given point during their task affected how they decided to position it.

4.1.1 *Information needed frequently is positioned at a lower access cost.* The more frequently users anticipated needing the assistive information, the more accessibly they would try to position it (low access cost). In our observations, during mobile tasks, this frequency of access affected whether a participant chose to (in ascending order of access cost): attach the instructions to task objects (RtO), carry the instructions around with them (RtB) or leave them somewhere fixed and move back to them (RtW).
<table><caption>Table 2: Summary of findings indicating the factors of the task properties we identified (frequency of information access, type of mobility) and for each factor the relevant considerations for RF preferences.</caption><thead><tr><th colspan="2">Frequency of Information Access</th><th colspan="2">Type of Mobility</th></tr><tr><th>Frequent</th><th>Infrequent</th><th>Predictable</th><th>Unpredictable</th></tr></thead><tbody><tr><td>RtB or RtO positioning preferred, with participants prioritising a lower access cost —i.e., available handsfree, in line of sight.</td><td>Highlights participants' constant trade-off between a lower access cost and a cluttered workspace.</td><td>RtW positioning preferred, with participants choosing to leave/duplicate information where it was needing rather than taking it with them.</td><td>RtB or RtO positioning preferred, with participants using user/task specific positioning to accommodate uncertainty about where they might need the information.</td></tr></tbody></table>

When the frequency at which users anticipated needing assistive increased, participants were more motivated to reduce access cost, whether that be by bringing the assistive information around with them, or, ensuring it was positioned readably or within reach according to their need. P8 for example found themselves constantly needing to refer to their notebook to check measurements as they marked and cut their wood (see Figure 2). They were constantly losing track of their notebook as part of constantly moving it around to be near them (RtB). Eventually, they wrote the measurements directly onto the wood (RtO) so they did not have to worry about interrupting their often safety-critical tasks of sawing and nailing wood to keep the notebook nearby. This also helped identify different planks after they had been cut. Here, P8 switched from RtB to RtO positioning in response to a high frequency of access, reducing access cost. Similarly, P2 sought to reduce access cost when they saw a frequent need for their manual while assembling a bed, making the effort to move the manual with them as they moved from corner to corner assembling the bed frame (RtB), placing the manual in their direct line of sight so they could refer to it *at a glance* while they worked *without having to lift their hands* from their task (see Figure 3a).

<figure><img src="image_6.png" /><figcaption>Figure 2: P8 refers to their notebook at the assembly station and the wood cutting station. Red outlines indicate the assistive information. Green outlines indicate where the task is taking place.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(a) P2 placed the manual near the task area (outlined in green) when they anticipated a high frequency of need and took the effort to move it around with them as they moved (RtB)</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(b) P2 placed the manual on a nearby chair a few steps away from the task area (RtW) when they anticipated needing the manual less frequently.</figcaption></figure>

<caption>Figure 3: P2 alternated between RtW and RtB positioning while assembling the bed as the frequency at which they anticipated needing the manual changed.</caption>
However, as the frequency with which users anticipated needing
assistive information decreased, so did their inclination to bother
ensuring it would be accessible at a low cost i.e., by bringing it with
them (RtB), instead defaulting to RtW positioning. For example,
when P2 did not anticipate needing the manual as frequently, they
put the manual in their pocket (RtB) so that it would be accessible
wherever they were as they moved around the room. To read the
manual they would have to free a hand to get it out, and we saw
it used more when they were looking for a part, rather than when
they were assembling them. Here, even though both methods were
RtB, when the manual was in their pocket, the tolerated access cost
was higher because they anticipated a reduced frequency of access.
Later, when the frequency of access decreased, for example during
parts of the task that were more intuitive or that they were more
familiar with, they even had the manual on a nearby chair (RtW)
(see Figure 3b) such that they had to get up and go get it (high
access cost). Similarly, P9 referred to the laser cutter's instruction
manual regularly at the beginning of the task, taking it with them
as they moved between the laser cutter and the computer (RtB).
However, this frequency diminished progressively as they gained
familiarity of the steps and specific settings needed to operate the
machine. They transitioned to leaving it near the computer (RtW)
where they needed it more and referring to it occasionally, until
they eventually put the manual down and never looked at it again.
P3 demonstrated this transition in the opposite direction, starting
with a higher access cost and adjusting their own position to create
a lower cost at certain points in their cooking. They initially placed
their phone with the recipe on it at a relatively high access cost,
on the far side of the kitchen counter (RtW) so it would be "out
of the way", walking over to it whenever they need to refer to it.
However, during the measuring-out-ingredients stage of the task,
P3 chose to work on the induction stovetop closer to where their
phone was located rather than on the designated counter space a
few steps away. This made it slightly easier for them to look over to
their phone to clarify quantities (lower access cost). P3 anticipated
a different frequency of access at different steps of the task and
changed where they decided to position themselves relative to the
recipe accordingly, thereby adjusting the access cost.

Overall, we see a trend where the high anticipated frequency
of need corresponds to participants positioning information at a
lower access cost. Then, as this need decreases, the access cost they
would tolerate would increase. That is not to say they would not
still prefer a lower access cost, but the participants demonstrated
and clearly expressed how ensuring a lower access cost reduced
in priority when it came to balancing their other needs, especially
since in this real-world non-AR scenario, there was a significant
cost to keeping things accessible.

4.1.2 *Information needed occasionally highlighted trade-off between access cost and clutter.* When users anticipated a less frequent need for assistive information, we observed that participants often tolerated having information positioned with a higher access cost in order to have a less cluttered workspace. P2 demonstrated this trade-off between access cost and clutter when they switched from taking the manual with them as they worked on the bed (RtB) to placing it on a low chair a few steps away (RtW) (see Figure 3). They explained that “when the task [was] less important or difficult

... you don't have to check the instructions every minute, [so] it was not necessary to keep it with me". They further explained that the chair was somewhere "out of the way" compared to on the floor or balanced on the bed, where it had been when they were moving it around with them (RtB). This was important because they wanted to "keep the floor clear otherwise it's dangerous". Here, the higher access cost of a few extra steps was an acceptable trade-off for less cluttered workspace because they did not anticipate a high access frequency.

Reducing clutter was described by several participants as motivation for positioning assistive information with a slightly higher access cost, even amongst those performing stationary tasks. P3 worked in a small kitchen and wanted to maximise counter space, P8's home workshop had very few flat surfaces to work on, and P7 was needle felting on a small mat on a small table. As such, they tolerated having the assistive information positioned with a higher access cost, i.e., P7 had to put down their needle felting to read their instructions because it was placed further away (Figure 4, P8 put their notebook on the floor to keep the table clear for wood cutting). There were alternative positions with a lower access cost that they did sometimes choose, such as on top of the mat or on top of the box of yarn, but at other times they chose to prioritise a clutter-free working environment. Interestingly, we also observed participants trying to compensate for a higher access cost by memorising information. For example. P3 tried to remember multiple ingredient quantities at once, and P8 tried to remember the measurements they needed, all to reduce the frequency at which they would need to refer to their assistive information.

<figure><img src="image_9.png" /><figcaption>Figure 4: P7 worked on a small table, with limited space for the needle felting mat and the paper instructions. To reduce clutter would move the instructions off the mat or put it off the side in order to make space for their work.</figcaption></figure>

Overall, we see that the anticipated frequency of information
access during the task was an important consideration that often
outweighed other considerations when users were deciding where
to position assistive information. However, it was only one of the
many components that people constantly weighed against each
other during the course of their task, adjusting the assistive infor-
mation position accordingly when their priorities changed.

## 4.2 Information Position Changes According to the Type Of Mobility

The level of mobility during a task affected how participants chose to position their assistive information. RtO and RtB positioning was prevalent when participants anticipated needing information on the move or at multiple, often unpredictable places, while RtW positioning arose when tasks were less mobile or happened across multiple defined locations.

<figure><img src="image_10.png" /><figcaption>Figure 5: P12 left their phone near the machine (RtW) while they went to fetch the weights. This was a predefined location where P12 knew they would need it.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 6: P5's tablet was normally on the worktop (RtW), but when cataloguing parts, they were not sure where they would need it so they picked it up and carried it with them (RtB)</figcaption></figure>

4.2.1 RtW positioning was preferred when task locations were pre-defined. When participants moved between multiple defined locations during a task, information was often positioned RtW in those locations as that was where the information was needed. This aligns with our finding that the more frequently users' anticipated needing the information, the closer participants positioned it. Here, participants deemed that information was only needed in certain locations and so placed the information in those locations. P12 left their phone near the machine they would be using while going to fetch weights because they knew they would only be needing their phone during the exercise (see Figure 5). P8 had two defined workstations in their home workshop, one for cutting wood and one for assembly (see Figure 2). At first, they brought the notepad with them as they moved between workstations (RtB), but found themselves constantly losing track of where it was. Eventually, they **copied** the measurements they needed from their notepad directly onto the wood it related to, so they would not have to constantly refer to their notepad. After this, they left their notepad at the assembly station (RtW) so they could refer to it during assembly and referred to the measurements on the wood (RtO) at the cutting station. In the interview, P8 further specified that their preferred solution would have been to have "multiple copies of the instructions so I could have a copy at each station". This use of duplication (RtW) instead of moving the information around (RtB) highlights that the information was only useful in fixed locations. This is further supported by the fact that during the tasks we observed, when instructions were moved, they were rarely referred to in transit, they were moved so that they could be in a new position. As such, it follows that having prior knowledge of relevant locations to have assistive information would affect participants' positioning behaviour.

4.2.2 RtB positioning was preferred when task locations were unpredictable. When participants moved frequently during their task to locations that were not fixed or otherwise predictable in anyway, information was often positioned RtB so that it could available no matter where. P5's bike workshop had a communal tablet used to input and access information about the bikes they were working on. When they were working on a bike and only referring to it sporadically, it was propped up on a box on the worktop behind them (RtW). In this case, the small interruption of taking a few steps over to use it was acceptable. When they were working around the workshop cataloguing parts, they carried the tablet around with them (RtB). In this case, because their location was unpredictable and constantly changing, they could not simply move the tablet to a more suitable place. However, having to repeatedly return to the worktop (or any fixed location) was sufficiently inconvenient that they made the effort of taking it with them (see Figure 6). P11 exhibited similar behaviour changing from a default RtW positioning to RtB positioning when they started moving around looking for something. The inventory lists they used were homed in two locations on top of two wire coils and on top of a stack of boxes (RtW) located in the general area of the warehouse they were inventorying (see Figure 1a or Figure 7a). Most often they located the item, and brought it to the lists for comparison, but when they were not sure where the item was or anticipated needing to stray a little further from the papers to find it, they would pick up the papers
and carry them around during their search (RtB) (see Figure 7b). Here, the trade-off between effort required to bring the information with them (RtB) was balanced against the effort of moving back and forth between the task and where the information had been left (RtW).

<figure><img src="image_12.png" /><figcaption>(a) P10 put their reference sheets on a stack of boxes and walked over to refer to them when needed (RtW)</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>(b) P10 picked up the reference sheets and walked around the warehouse with them when they were not sure where they would end up while looking for items</figcaption></figure>

**Figure 7:** P10 switched between positioning their reference sheets RtW and RtB depending on the how unpredictable their movement was going to be.

4.2.3 RtO positioning when task locations were multiple or unpredictable. An interesting alternative approach to deal with uncertainty about where the assistive information would be needed was RtO positioning. P9 measured a deck of cards as part of their task and stuck a post-it with the measurements on it directly on the deck (see Figure 1c) because they were not sure when they would need to refer back to this information. The alternative would have been

to make a note on their phone, but they explained that they often “lost” the information when they did that. This way, the information would be there so long as the deck of cards was there, and since the task involved the deck, they could be sure it would be. P8 wrote the measurements for the wood they were cutting directly on the wood, so they would need to refer to their notebook. After the wood had been cut, this writing then helped distinguish different planks of similar size when they were being assembled. In both cases, by using RtO positioning the assistive information was available in a location-agnostic and task-relevant way.

## 5 Discussion

In this paper, we explored the question: *What task characteristics affect how people display non-AR assistive information during physical tasks?* Our findings raise several interesting considerations for the design of an adaptive system to display assistive information using an ARHMD during such tasks. First, we found that people performing complex bi-manual physical tasks display assistive information using **multiple methods** over the course of the task. This supports our initial premise that the most effective position for assistive information might change or be multiple during bi-manual physical tasks. We discuss the system design implications of the way participants used multiple methods over the course of their task in Sections 5.1 and 5.2. Second, our findings further show that a key factor causing the use of multiple methods was participants’ desire to accommodate changing task characteristics, the most pertinent ones being **frequency of information access** and **level of mobility**. We discuss how these characteristics might be operationalised and accommodated for by an ARHMD assistive information system in Section 5.3. Finally, in Sections 5.4 we discuss the limitations of our approach of seeking insights for AR system design by observing real-world non-AR behaviour.

### 5.1 Same Information Positioned using Multiple Methods

5.1.1 *At Different Times.* Seeing assistive information moved and displayed in different locations using different methods at different times during the task is a strong argument for the location of assistive information to be dynamic. At some point in their task, P2, P5 and P10 all tried to have their assistive information follow them around as they moved. But at other times, they also had the same assistive information in a *home* location. In this case, we see that the same assistive information was positioned not only with different methods but also according to different RFs. To the best of our knowledge, transitioning between RFs is under-explored in 3D UI literature. Pei et al.'s [45] survey of 113 commercial applications with 3D UIs showed that only 6.2% allowed users to transition UIs between RFs. As such, future work could consider how and when an ARHMD system could enable users to position the same information not only using multiple methods in the same RF but according to multiple RFs.

5.1.2 *At the Same Time.* During the tasks we observed, instructions were rarely referred to in transit when they were moved. They were moved so they could be in a new position. In Lu & Xu's [38] exploration of spatial UI transition methods, participants needed different UI widgets to complete tasks depending on their location in
the VR environment, but while moving between rooms, the widgets were “collapsed” into the wrist. Here the assistive widgets were less accessible whilst the user was in transit between locations. This type of strategy works for tasks where participants move information because they needed it at defined place A and place B, but have no need for it in between. However, simply having the information available both at place A and place B could also be a valid approach. In this vein, P8 expressed a desire for multiple copies of their instructions and attempted to achieve this by copying the measurements they needed from their notebook onto the wood they were working with. This way they would not have to keep referring to their notebook whenever they needed the measurements. It is important to note that this behaviour was not extensively observed as all participants only had access to one copy of their assistive information. However, duplicating information in the digital world has a much lower cost than having to print out a new manual or having multiple devices available. Therefore, content duplication is a display method that is more accessible in AR and to our knowledge has yet to be extensively explored, with existing systems [13, 31] focusing on the smooth transition of information between locations.

## 5.2 Different Information Positioned Using Multiple Methods

When participants referred to multiple types of assistive information in their task (i.e., paper, tablet, smartphone), it was common for them to use different methods to display the information. In our previous example, with P8 copying a subset of their notes onto the wood it related to, they had essentially created another, more succinct version of the assistive information. Here the part of the information related to the wood was displayed using a relative-to-object method while the notepad used a relative-to-world method, with P8 switching between them depending on their information needs. P5 pinned the paper receipts they were working with on the wall (relative-to-world), while taking their tablet with them around the workshop (relative-to-body). How to integrate multiple devices into one cohesive user experience is a challenge documented in literature about multi-device ecosystems [21]. Transposing this into AR, all the assistive information might be available via one device, an ARHMD, but like in a multi-device ecosystem we are in theory able to have different subsets of the information displayed on independent displays, freely transitioning them between multiple positioning methods and multiple RFs according to what best suits our needs. This is what we envision, but how to integrate the many content placement affordances of ARHMDs into a cohesive and effective user experience for complex bi-manual task performance is a question that warrants further investigation.

## 5.3 Task Characteristics Affect Position Choice

The development of an operationalised understanding of task characteristics suitable for input into an AR system is an ongoing challenge. Such an understanding would be invaluable to the design of an AR assistive information system that wanted to have the “best” positioning method available in response to changing task characteristics. There are certainly other task characteristics than **type of mobility** and **frequency of information need** that play a role in users’ choice of positioning method. We intend to work on

identifying them in future work with further analysis of the same data corpus. We discuss below the implications of considering the task characteristics we identified in our observations on the design of an ARHMD assistive information system.

5.3.1 *Type of Mobility.* The mobile tasks we observed demonstrated the use of many methods within and across RFs primarily to ensure that participants were able to access information wherever they were. This supports the premise that motivates many AR assistive information systems proposing display methods where the information follows the user while they walk around [31, 37]. Interestingly, some participants used RtW positioning to place information in a fixed “home” location while they moved around. This might leverage spatial memory, documented to speed up information retrieval [8, 36], to help participants consistently know where to look for information. Therefore, a better understanding of how might a system determine when a user would prefer that the assistive information dynamically “follow” them versus stay in a home location is needed.

One determining factor we observed was the **type of mobility** involved in the task. Was it predictable movement or unpredictable movement? Over how many locations did the task place? Could the user predict where all these locations would be beforehand? Depending on the answer to these questions, we saw users change the position of their assistive information. Existing work that explores dual-tasking while walking [31, 37] mostly explore unpredictable motion as they have participants follow a path that is revealed progressively. As such, future work could investigate how a system might adapt to the type of movement that the user was performing over the course of a task. For example, upon detecting that a user was regularly visiting the same place, it could automatically (or prompt the user to) place a copy of the assistive information in that space. This is not unlike systems that prioritise access to recently or frequently visited items.

5.3.2 *Frequency of Information Access.* When participants encountered different **types of information needs** e.g., detailed comparison, clarification, they used different methods to position information. When participants had a constant or regular need for assistive information during task performance they put it next to or “as close as possible” to the task area. Within that, tasks requiring extensive visual comparison between task objects and the instructions also saw a higher incidence of them being brought closer together. This proximity was reminiscent of RtO positioning, had it been easily achievable, and supports the premise that motivates many AR assistive information systems proposing visualisations overlaid on or as close as possible to the task space [47, 52]. On the other hand, when users only anticipated needing brief glances for clarification or reassurance, information reverted to being positioned relative-to-world by default as participants were not as inclined to go to the effort of bringing it with them “just in case”. It would be interesting to see how this behaviour evolved in response to the reduced cost of RtB and RtO positioning in AR. Would users want information that they only occasionally glanced at to be overlaid on their workspace?

Another way to look at this is to distil the more abstract concept of “type of need” i.e. comparison, clarification, etc. into how it translated to the **frequency at which the information was**
needed. This frequency can change during the task according to a variety of factors, including but not limited to: type of information need, user familiarity with the task, and type of information (i.e. images, video, text). Interestingly, we saw instances of participants exhibiting behaviour designed to reduce this frequency by memorising information. In general, people understand that accessing assistive information represents a cost. However, we observed that while they might seek to reduce this cost by making it easier to access (i.e., nearby, in line of sight), this was often achieved at the cost of cluttering their workspace.

An ARHMD system might solve the **clutter vs. low access cost trade-off**, by lifting the information out of real world space into virtual space and thereby not taking up any actual "space". However, virtual displays still take up visual attention and occlude the real world in a way that represents physical and cognitive clutter.[7] As such, a better understanding of how an ARHMD system could distinguish between when displaying assistive information to prioritise a low access cost was desirable, and when it represented clutter. In this vein, a consideration of the **type of information need** might again be interesting, for example for detailed comparison of a low access cost might trump clutter, whereas for quick glances for reassurance or clarification minimising clutter might take priority over minimising access cost.

## 5.4 Positioning Method Choice Needs to Be Evaluated in AR

AR opens up a large panel of positioning methods for information used during bi-manual physical tasks. This is especially the case for RtO positioning methods, whose incidence was low in our observations but was also the most difficult to achieve using non-AR methods. Without AR technology, having information attached to movable objects can be intrusive to task performance. Having information follow the user is also difficult to achieve without them having a free hand. In AR, this is less of a concern. This is a limitation of our approach that we are aware of, as the cost of enabling RtB and RtO positioning noticeably inhibited its use in our observations. However, despite a high access cost, participants still chose to use RtB and RtO positioning, highlighting their potential in AR where access costs are lower.

Seeking inspiration from real-world strategies for AR is not always desirable or optimal as technology provides affordances that are not available in the real world. However, our findings confirm that existing behaviours can still inspire the design and exploration of new AR methods. Given this, a natural follow-on to the work presented in this paper would be an exploration of the factors affecting positioning method choice for assistive information displayed using ARHMDs. It would be interesting to see how peoples' method choice within and across RFs changed when provided the wider range of options available to an AR system. For example, since our study was only able to observe the use of 2D assistive information, it could be interesting to investigate whether RF preferences differ when displaying 2D information versus the spatially registered 3D information that AR makes more accessible?

Our findings also showed that, even though RtO positioning was arguably difficult to achieve in the real world, participants still chose to use it. This highlights the potential value of RtO positioning under certain conditions.

In AR where RtO and RtB positioning are just as accessible as RtW, it could be interesting to investigate whether the access cost vs. clutter trade-off discussed in Section 4.1.2 still applied. For example, Warden et al. [54]'s work on the scan-clutter trade-off when using head-mounted displays in safety-critical environments found that while displaying information overlaid or adjacent to critical aspects of the task could reduce the cost of repetitive visual scanning, increasing clutter caused an increasing cost to both speed and accuracy. As such, for tasks where scene information is required, it is important to minimise clutter in the design of HMD interfaces. It follows that if we considered assistive information positioned RtB (i.e., attached to the torso, head, arm) might represent a potential source of physical and cognitive clutter that could see users preferring positions other than RtB. Therefore, would the prevalence of RtW methods actually decrease compared to other RFs when they were all equally achievable? What are the factors that might affect the use of RtO and RtB methods in AR? These are the questions that highlight the value of further investigating the choice of positioning method in AR.

## 6 Conclusion

We presented an observation study of how people interacted with assistive information when no AR technology was available. We found that during complex bimanual physical tasks (1) assistive information was positioned using multiple methods at different times or multiple methods at the same time and, (2) task characteristics such as type of mobility and frequency of information access impacted position choice. These findings in a non-AR context open novel research directions for the design of adaptive AR assistive systems. They invite researchers to explore how to operationalise 'what the user is doing' into dimensions usable as inputs to an AR system that adapts information position appropriately.

## Acknowledgements

This research was supported by the *Institut Universitaire d'Ingénierie en Santé* of Sorbonne Université. We thank all the participants for their time, and the HCI Sorbonne research group for their invaluable support and feedback. Special mention to Nour, César and all my friends, without whom this work would not have been possible.

## References

[1] Gregory D. Abowd, Anind K. Dey, Peter J. Brown, Nigel Davies, Mark Smith, and Pete Steggles. 1999. Towards a Better Understanding of Context and Context-Awareness. In *Handheld and Ubiquitous Computing*, Gerhard Goos, Juris Hartmann, Jan Van Leeuwen, and Hans-W. Gellersen (Eds.). Vol. 1707. Springer Berlin Heidelberg, Berlin, Heidelberg, 304–307. https://doi.org/10.1007/3-540-48157-5_29

[2] Emad Al-Yahya, Helen Dawes, Lesley Smith, Andrea Dennis, Ken Howells, and Janet Cockburn. 2011. Cognitive motor interference while walking: A systematic review and meta-analysis. *Neuroscience & Biobehavioral Reviews* 35, 3 (Jan. 2011), 715–728. https://doi.org/10.1016/j.neubiorev.2010.08.008

[3] Ronald T. Azuma. 1997. A Survey of Augmented Reality. *Presence: Teleoperators and Virtual Environments* 6, 4 (Aug. 1997), 355–385. https://doi.org/10.1162/pres.1997.6.4.355

[4] M. Billinghurst, J. Bowskill, M. Jessop, and J. Morphett. 1998. A wearable spatial conferencing space. In *Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215)*. IEEE, Pittsburgh PA USA, 76–83. https://doi.org/10.1109/ISWC.1998.729532

[5] Manuel Birlo, P.J. Eddie Edwards, Matthew Clarkson, and Danail Stoyanov. 2022. Utility of optical see-through head mounted displays in augmented reality-assisted surgery: A systematic review. *Medical Image Analysis* 77 (April 2022),
102361. https://doi.org/10.1016/j.media.2022.102361

[6] Jonas Blattgerste, Benjamin Strenge, Patrick Renner, Thies Pfeiffer, and Kai Essig. 2017. Comparing Conventional and Augmented Reality Instructions for Manual Assembly Tasks. In *Proceedings of the 10th International Conference on Pervasive Technologies Related to Assistive Environments*. ACM, Island of Rhodes Greece, 75–82. https://doi.org/10.1145/3056540.3056547

[7] Maxime Cauz, Thibaut Septon, and Bruno Dumas. 2023. Interaction Techniques to Control Information Clutter in a Pervasive Augmented Reality Scenario. In *HCI International 2023 – Late Breaking Papers*, Jessie Y. C. Chen, Gino Fragomeni, and Xiaowen Fang (Eds.). Springer Nature Switzerland, Cham, 3–21. https://doi.org/10.1007/978-3-031-48050-8_1

[8] Andy Cockburn and Bruce McKenzie. 2002. Evaluating the effectiveness of spatial memory in 2D and 3D physical and virtual environments. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '02)*. Association for Computing Machinery, New York, NY, USA, 203–210. https://doi.org/10.1145/503376.503413

[9] A. Cockburn, P. Quinn, C. Gutwin, G. Ramos, and J. Looser. 2011. Air pointing: Design and evaluation of spatial target acquisition with and without visual feedback. *International Journal of Human-Computer Studies* 69, 6 (June 2011), 401–414. https://doi.org/10.1016/j.ijhcs.2011.02.005

[10] Florian Echtler, Fabian Sturn, Kay Kindermann, Gudrun Klinker, Joachim Stilla, Jörn Trilk, and Hesam Najafi. 2004. The Intelligent Welding Gun: Augmented Reality for Experimental Vehicle Construction. In *Virtual and Augmented Reality Applications in Manufacturing*, S. K. Ong and A. Y. C. Nee (Eds.). Springer London, London, 333–360. https://doi.org/10.1007/978-1-4471-3873-0_17

[11] Barrett Ens, Juan David Hincapié-Ramos, and Pourang Irani. 2014. Ethereal planes: a design framework for 2D information space in 3D mixed reality environments. In *Proceedings of the 2nd ACM symposium on Spatial user interaction*. ACM, Honolulu Hawaii USA, 2–12. https://doi.org/10.1145/2659766.2659769

[12] Barrett Ens, Eyal Ofek, Neil Bruce, and Pourang Irani. 2015. Spatial Constancy of Surface-Embedded Layouts across Multiple Environments. In *Proceedings of the 3rd ACM Symposium on Spatial User Interaction (SUI '15)*. Association for Computing Machinery, New York, NY, USA, 65–68. https://doi.org/10.1145/2788940.2788954

[13] João Marcelo Evangelista Belo, Mathias N. Lystbæk, Anna Maria Feit, Ken Pfeuffer, Peter Kän, Antti Oulasvirta, and Kaj Grønbæk. 2022. AUIT – the Adaptive User Interfaces Toolkit for Designing XR Applications. In *Proceedings of the 35th Annual ACM symposium on User Interface Software and Technology*. ACM, Bend OR USA, 1–16. https://doi.org/10.1145/3526113.3545651

[14] Steven Feiner, Blair MacIntyre, Marcus Haupt, and Eliot Solomon. 1993. Windows on the world: 2D windows for 3D augmented reality. In *Proceedings of the 6th annual ACM symposium on User interface software and technology (UIST '93)*. Association for Computing Machinery, New York, NY, USA, 145–155. https://doi.org/10.1145/168642.168657

[15] Andreas Fender, David Lindlbauer, Philipp Herholz, Marc Alexa, and Jörg Müller. 2017. HeatSpace: Automatic Placement of Displays by Empirical Analysis of User Behavior. In *Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17)*. Association for Computing Machinery, New York, NY, USA, 611–621. https://doi.org/10.1145/3126594.3126621

[16] Markus Funk, Thomas Kosch, and Albrecht Schmidt. 2016. Interactive worker assistance: comparing the effects of in-situ projection, head-mounted displays, tablet, and paper instructions. In *Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing*. ACM, Heidelberg Germany, 934–939. https://doi.org/10.1145/2971648.2971706

[17] Markus Funk, Lars Lischke, Sven Mayer, Alireza Shirazi, and Albrecht Schmidt. 2018. Teach Me How! Interactive Assembly Instructions Using Demonstration and In-Situ Projection. In *Assistive Augmentation. Cognitive Science and Technology*. Springer, Singapore, Singapore, 49–73. https://doi.org/10.1007/978-981-10-6404-3_4

[18] Steffen Gauglitz, Benjamin Nuernberger, Matthew Turk, and Tobias Höllerer. 2014. World-stabilized annotations and virtual scene navigation for remote collaboration. In *Proceedings of the 27th annual ACM symposium on User interface software and technology (UIST '14)*. Association for Computing Machinery, New York, NY, USA, 449–459. https://doi.org/10.1145/2642918.2647372

[19] Yalda Ghasemi, Allison Bayro, Justin MacDonald, Heejin Jeong, Joel Reynolds, and Chang S. Nam. 2024. Embedding Spatial Augmented Reality in Culinary Training: A Comparative Evaluation of sAR Kitchen and Video Tutorials. *IEEE Transactions on Learning Technologies* 17 (2024), 765–775. https://doi.org/10.1109/TLT.2023.3343788

[20] Ayse Goker and Hans Myrhaug. 2002. User Context and Personalisation. In *Workshop on Case Based Reasoning and Personalization*. Aberdeen, Scotland, 1–7. https://openaccess.city.ac.uk/id/eprint/624/

[21] Jens Grubert, Matthias Kranz, and Aaron Quigley. 2016. Challenges in mobile multi-device ecosystems. *mUX: The Journal of Mobile User Experience* 5, 1 (Aug. 2016), 5. https://doi.org/10.1186/s13678-016-0007-y

[22] Violet Yinuo Han, Hyunsung Cho, Kiyosu Maeda, Alexandra Ion, and David Lindlbauer. 2023. BlendMR: A Computational Method to Create Ambient Mixed Reality Interfaces. *Proceedings of the ACM on Human-Computer Interaction* 7, ISSN

(Oct. 2023), 217–241. https://doi.org/10.1145/3626472

[23] Benjamin Hatscher, Maria Luz, and Christian Hansen. 2018. Foot Interaction Concepts to Support Radiological Interventions. *i-com* 17, 1 (April 2018), 3–13. https://doi.org/10.1515/icom-2018-0002

[24] Benjamin Hatscher, Maria Luz, Lennart E. Nacke, Norbert Elkmann, Veit Müller, and Christian Hansen. 2017. GazeTap: towards hands-free interaction in the operating room. In *Proceedings of the 19th ACM International Conference on Multimodal Interaction (ICMI '17)*. Association for Computing Machinery, New York, NY, USA, 243–251. https://doi.org/10.1145/3136755.3136759

[25] Mithun George Jacob and Juan Pablo Wachs. 2014. Context-based hand gesture recognition for the operating room. *Pattern Recognition Letters* 36 (Jan. 2014), 196–203. https://doi.org/10.1016/j.patrec.2013.05.024

[26] Janet G Johnson, Tommy Sharkey, Iramuali Cynthia Butarbutar, Danica Xiong, Ruijie Huang, Lauren Sy, and Nadir Weibel. 2023. UnMapped: Leveraging Experts' Situated Experiences to Ease Remote Guidance in Collaborative Mixed Reality. In *Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23)*. Association for Computing Machinery, New York, NY, USA, 1–20. https://doi.org/10.1145/3544548.3581444

[27] Naohiro Kishishita, Kiyoshi Kiyokawa, Jason Orlosky, Tomohiro Mashita, Haruo Takemura, and Ernst Kruijff. 2014. Analysing the effects of a wide field of view augmented reality display on search performance in divided attention tasks. In *2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)*. IEEE, Munich, Germany, 177–186. https://doi.org/10.1109/ISMAR.2014.6948425

[28] Anders Kofod-Petersen and Jörg Cassens. 2006. Using Activity Theory to Model Context Awareness. In *Modeling and Retrieval of Context*, Thomas R. Roth-Berghofer, Stefan Schulz, and David B. Leake (Eds.). Vol. 3946. Springer Berlin Heidelberg, Berlin, Heidelberg, 1–17. https://doi.org/10.1007/11740674_1

[29] Ari Kolbeinsson, Emmie Fogelberg, and Peter Thorvold. 2023. Information display preferences for assembly instructions in 6 industrial settings. In *Cognitive Computing and Internet of Things: Proceedings of the 14th International Conference on Applied Human Factors and Ergonomics and the Affiliated Conferences, San Francisco, USA, 20-24 July, 2023*. AHFE International Open Access, New York, NY, USA, 152–161. https://doi.org/10.54941/ahfe1003292

[30] Sri Kolla, Andre Sanchez, and Peter Plapper. 2021. Comparing Effectiveness of Paper Based and Augmented Reality Instructions for Manual Assembly and Training Tasks. *SSRN Electronic Journal* (Jan. 2021), 1–6. https://doi.org/10.2139/ssrn.3859970

[31] Wallace S. Lages and Doug A. Bowman. 2019. Walking with adaptive augmented reality workspaces: design and usage patterns. In *Proceedings of the 24th International Conference on Intelligent User Interfaces (IUI '19)*. Association for Computing Machinery, New York, NY, USA, 356–366. https://doi.org/10.1145/3301275.3302278

[32] Hvunini Lee and Wontack Woo. 2002. Investigating Display Position of a Head-Fixed Augmented Reality Notification for Dual-task. In *2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)*. IEEE, Christchurch New Zealand, 584–585. https://doi.org/10.1109/VRW55335.2022.00143

[33] Zhipeng Li, Christoph Gebhardt, Yves Inglin, Nicolas Steck, Paul Streli, and Christian Holz. 2024. SituationAdapt: Contextual UI Optimization in Mixed Reality with Situation Awareness via LLM Reasoning. In *Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST '24)*. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3654777.3674706

[34] David Lindlbauer, Anna Maria Feit, and Otmar Hilliges. 2019. Context-Aware Online Adaptation of Mixed Reality Interfaces. In *Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST '19)*. Association for Computing Machinery, New York, NY, USA, 147–160. https://doi.org/10.1145/3332165.3347945

[35] Jiazhou Liu, Barrett Ens, Arnaud Prouzeau, Jim Smiley, Isobel Kara Nixon, Sarah Goodwin, and Tim Dwyer. 2023. DataDancing: An Exploration of the Design Space For Visualisation View Management for 3D Surfaces and Spaces. In *Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23)*. Association for Computing Machinery, New York, NY, USA, 1–17. https://doi.org/10.1145/3544548.3580827

[36] Jiazhou Liu, Arnaud Prouzeau, Barrett Ens, and Tim Dwyer. 2022. Effects of Display Layout on Spatial Memory for Immersive Environments. *Proceedings of the ACM on Human-Computer Interaction* 6, ISSN (Nov. 2022), 576:468–576:488. https://doi.org/10.1145/3567729

[37] Feiyu Lu, Virginia Tech, United States, Shakiba Davari, Virginia Tech, United States, Lee Lisle, Virginia Tech, United States, Yuan Li, Virginia Tech, United States, Doug A Bowman, Virginia Tech, and United States. 2020. Glanceable AR: Evaluating Information Access Methods for Head-Worn Augmented Reality. In *IEEE Conference on Virtual Reality and 3D User Interfaces (VR)*. IEEE, Atlanta, GA, USA, 930–939. https://doi.org/10.1109/VR46266.2020.00113

[38] Feiyu Lu and Yan Xu. 2022. Exploring Spatial UI Transition Mechanisms with Head-Worn Augmented Reality. In *Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22)*. Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/3491102.3517723
[39] Andrés Lucero. 2015. Using Affinity Diagrams to Evaluate Interactive Prototypes. In *Human-Computer Interaction – INTERACT 2015*, Julio Abascal, Simone Barbosa, Mirko Fetter, Tom Gross, Philippe Palanque, and Marco Winckler (Eds.). Springer International Publishing, Cham, 231–248. https://doi.org/10.1007/978-3-319-22668-2_19

[40] Helena M. Mentis. 2017. Collocated Use of Imaging Systems in Coordinated Surgical Practice. *Proceedings of the ACM on Human-Computer Interaction* 1, CSCW (Dec. 2017), 1–17. https://doi.org/10.1145/3134713

[41] Helena M. Mentis, Kenton O'Hara, Gerardo Gonzalez, Abigail Sellen, Robert Corish, Antonio Criminisi, Rikin Trivedi, and Pierre Theodore. 2015. Voice or Gesture in the Operating Room. In *Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '15)*. Association for Computing Machinery, New York, NY, USA, 773–780. https://doi.org/10.1145/2702613.2702963

[42] Aziz Niyazov, Barrett Ens, Kadek Ananta Satriadi, Nicolas Mellado, Loïc Barthe, Tim Dwyer, and Marcos Serrano. 2023. User-Driven Constraints for Layout Optimisation in Augmented Reality. In *ACM Conference on Human Factors in Computing Systems (CHI 2023)*. ACM, Hambourg, Germany, 1–17. https://doi.org/10.1145/3544548.3580873

[43] Alessandra Papetti, Marianna Ciccarelli, Matteo Claudio Palpacelli, and Michele Germani. 2023. How to provide work instructions to reduce the workers’ physical and mental workload. *Procedia CIRP* 120 (2023), 1167–1172. https://doi.org/10.1016/j.procir.2023.09.143

[44] P. Patel, M. Lamar, and T. Bhatt. 2014. Effect of type of cognitive task and walking speed on cognitive-motor interference during dual-task walking. *Neuroscience* 260 (Feb. 2014), 140–148. https://doi.org/10.1016/j.neuroscience.2013.12.016

[45] Siyou Pei, David Kim, Alex Olwal, Yang Zhang, and Ruofei Du. 2024. UI Mobility Control in XR: Switching UI Positionings between Static, Dynamic, and Self Entities. In *Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '24)*. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3613904.3642220

[46] Mary Elizabeth Raven and Alicia Flanders. 1996. Using contextual inquiry to learn about your audiences. *SIGDOC Asterisk J. Comput. Doc.* 20, 1 (Feb. 1996), 1–13. https://doi.org/10.1145/227614.227615

[47] Cindy M. Robertson, Blair MacIntyre, and Bruce N. Walker. 2008. An evaluation of graphical context when the graphics are outside of the task area. In *Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR '08)*. IEEE Computer Society, USA, 73–76. https://doi.org/10.1109/ISMAR.2008.4637328

[48] Rufat Rzayev, Susanne Korbely, Milena Maul, Alina Schark, Valentin Schwind, and Niels Henze. 2020. Effects of Position and Alignment of Notifications on AR Glasses during Social Interaction. In *Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society*. ACM, Tallinn Estonia, 1–11. https://doi.org/10.1145/3419249.3420095

[49] Albrecht Schmidt, Michael Beigl, and Hans-W Gellersen. 1999. There is more to context than location. *Computers & Graphics* 23, 6 (Dec. 1999), 893–901. https://doi.org/10.1016/S0097-8493(99)00120-X

[50] Adwait Sharma, Alexander Ivanov, Frances Lai, Tovi Grossman, and Stephanie Santosa. 2024. GraspUI: Seamlessly Integrating Object-Centric Gestures within the Seven Phases of Grasping. In *Designing Interactive Systems Conference. ACM, IT University of Copenhagen Denmark*, 1275–1289. https://doi.org/10.1145/3643834.3661551

[51] Eleanor Smith, Gordon Semple, Dorothy Evans, Kenneth McRae, and Paul Blackwell. 2020. Augmented Instructions: Analysis of Performance and Efficiency of Assembly Tasks. In *Virtual, Augmented and Mixed Reality, Industrial and Every-day Life Applications*, Jessie Y. C. Chen and Gino Fragomeni (Eds.). Vol. 12191. Springer International Publishing, Cham, 166–177. https://doi.org/10.1007/978-3-030-49698-2_12 Series Title: Lecture Notes in Computer Science.

[52] Antonio E. Uva, Michele Gattullo, Vito M. Manghisi, Daniele Spagnulo, Giuseppe L. Cascella, and Michele Fiorentino. 2018. Evaluating the effectiveness of spatial augmented reality in smart manufacturing: a solution for manual working stations. *The International Journal of Advanced Manufacturing Technology* 94, 1–4 (Jan. 2018), 509–521. https://doi.org/10.1007/s00170-017-0846-4

[53] Frank K. Wacker, Sebastian Vogt, Ali Khamene, John A. Jesberger, Sherif G. Nour, Daniel R. Elgort, Frank Sauer, Jeffrey L. Duerk, and Jonathan S. Lewin. 2006. An Augmented Reality System for MR Image-guided Needle Biopsy: Initial Results in a Swine Model. *Radiology* 238, 2 (Feb. 2006), 497–504. https://doi.org/10.1148/radiol.2382041441

[54] Amelia C. Warden, Christopher D. Wickens, Daniel Rehberg, Benjamin A. Clegg, and Francisco R. Ortega. 2025. Clutter costs in head-mounted displays: a study examining trade-offs between overlay and adjacent presentation of information. *Cognitive Research: Principles and Implications* 10 (Aug. 2025), 47. https://doi.org/10.1186/s41235-025-00650-5

[55] Yumeng Yang, Joyce Karreman, and Menno De Jong. 2020. Comparing the Effects of Paper and Mobile Augmented Reality Instructions to Guide Assembly Tasks. In *2020 IEEE International Professional Communication Conference (ProComm)*. IEEE, Kennesaw, GA, USA, 96–104. https://doi.org/10.1109/ProComm48883.2020.00021

[56] Faisal Zaman, Craig Anslow, and Taehyun James Rhee. 2023. Vicarious: Context-aware Viewpoints Selection for Mixed Reality Collaboration. In *29th ACM Symposium on Virtual Reality Software and Technology*. ACM, Christchurch New Zealand, 1–11. https://doi.org/10.1145/3611659.3615709

[57] Andreas Zimmermann, Andreas Lorenz, and Reinhard Oppermann. 2007. An Operational Definition of Context. In *Modeling and Using Context*, Boicho Kokinov, Daniel C. Richardson, Thomas R. Roth-Berghofer, and Laure Vieu (Eds.). Vol. 4635. Springer Berlin Heidelberg, Berlin, Heidelberg, 558–571. https://doi.org/10.1007/978-3-540-74255-5_42
# A Post-Task Semi-Structured Interview Guide

The idea of this semi-structured interview is to take examples of behaviour that the researcher noticed during the observation and ask further questions to gather relevant details.

As such, there will be four groups of questions: 1) Clarifying questions about the task, 2) Questions about specific occurrences during the observation, 3) questions to be asked to every participant to gather further contextual information, and 4) questions that may be relevant to ask post-task. Group 1 questions will be different across observations and cannot be predefined. Below we outline template questions for Group 2 and questions for groups 3 and 4.

## Group 2: Template questions with key words

* You placed the instructions *insert location* while you performed the task, why? **Position**

* You moved the instructions *insert time*, why? **Movement**

* You referred to the instructions *insert time*, why? **Reference**

* You interacted with the device *insert behaviour*, why? **Device**

* You did *insert action* during the task, how would this have affected your position preferences if the information could be placed anywhere? **Position**

* What feature of the device made your experience better/worse? **Device**

* If you could have placed/moved it elsewhere, where would it have been? **Position/Movement** Why would that be better?

* What affected your ability to position the information where you wanted it? **Position**

* If device did not have *insert limitation*, how would this have affected your positioning/movement/reference behaviour? **Device**

* How did *insert contextual factor* affect the position/movement of instructions? **Position/Movement**

* How do you feel *insert contextual factor* affected how you performed the task?

## Group 3: Mandatory questions

(1) Could this task be performed without the technological assistance task instructions?

* If yes: How? What would change?

* If no: Why?

(2) What format do the task instructions usually take? Device + medium

* What is your preference?

* Would another type of instructions be better i.e. video, images, written instructions?

* Would this change how you **positioned/moved** the information?

## Group 4: Optional questions

* How would you describe the error tolerance of your task?

* How would you describe the risk associated with your task?

* How would you describe your expertise in this task?

* How representative was the task to usual?

* How do you feel your environment affected how you performed the task?

Blue-sky hypotheticals

* What changes would you make to the instructions/device to improve your experience?

* If information could be positioned anywhere...

* If information could move dynamically...