# Wearable Touchscreens to Integrate Augmented Reality and Tablets for Work Instructions?

Jan Van den Bergh, Florian Heller

► To cite this version:

Jan Van den Bergh, Florian Heller. Wearable Touchscreens to Integrate Augmented Reality and Tablets for Work Instructions?. 8th International Conference on Human-Centred Software Engineering (HCSE), Nov 2020, Eindhoven, Netherlands. pp.199-206, 10.1007/978-3-030-64266-2_13.hal-03250499

HAL Id: hal-03250499

https://inria.hal.science/hal-03250499

Submitted on 4 Jun 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Wearable Touchscreens to Integrate Augmented Reality and Tablets for Work Instructions?

Jan Van den Bergh[0000-0002-3054-0628] and Florian Heller[0000-0002-3265-3570]★

Hasselt University - tUL - Flanders Make
Expertise Centre for Digital Media
Diepenbeek, Belgium
`firstname.lastname@uhasselt.be`

**Abstract.** Manual assembly in high variety - low volume production is challenging for the operators as there might be small changes from one product to the next. The operators receive adapted instruction sets to be aware of these small differences, currently either printed or presented on a computer terminal. Augmented Reality (AR) is considered the future of work instruction display as it promises hands-free interaction, but interacting with AR interfaces can be problematic in production environments.

In this paper, we report on an exploratory study where we compare tablet interaction with three different ways of touch interaction for AR glasses: on the table, on the glasses, and the lower arm. While keeping the instructions the same in all conditions, we measured how placement affects the overall workload while performing an assembly task. Results indicate that combining additional wearable touch input with AR glasses is at least a promising direction for future research and, at best, an enabler for more widespread AR usage in manual assembly.

**Keywords:** Work Instructions · Augmented Reality · Tablet · User-Centered Design · Assembly · End-User Feedback.

## 1 Introduction

Despite all automation achieved in manufacturing, a large part of the work is still performed by manual assembly operators, called just operators from now on. Especially in high variety, low volume production, they perform the majority of the work. In many cases, different models are assembled on the same line in mixed-model assembly systems (MMAS). This poses challenges for operators regarding cognitive load as these models might only differ in small details. Operators, thus, increasingly get (digital) work instructions that guide them during the assembly. However, given the challenging environment that a production line is, how can the operators interact best with these instructions.

Wearables — computing devices worn at specific places on the body (e.g., arms, legs, belt) or head — have been described as promising to support operators in their tasks [7,8,10], as, e.g., head-mounted displays can superimpose

<footnote>★ Both authors contributed equally.</footnote>
information directly onto the assembly [1]. The issue of presenting the work in-
structions at the right time and offering a feedback channel for the operators to
flag potential defects in the instructions [4] makes an input capability for the
AR interface a necessity. Body-based touch input can be given under various cir-
cumstances, such as standing, sitting, or kneeling [12]. This paper discusses an
exploratory study on how operators can interact with work instructions on the
shop floor. As the work environment is likely to interfere with voice input and
gestures, we focus on touch interaction, which is already supported by several
such devices. We ran an experiment to evaluate the impact of the touch input
location on the perceived workload while performing a primary assembly task.
We presented the instructions on a tablet and an AR headset, and tested touch
input on the tablet, the headset, or the lower arm.

## 2 Related Work

Augmented reality is frequently realized through HMDs. Koelle et al. investi-
gated user attitudes towards these glasses and recommended being task-focused
and using a least capabilities approach [6]. In the case of operators, this implies
that if they are not expected to contribute pictures or videos, HMDs without a
camera may be recommended. People have concerns about being recorded and
may not even notice indicator LEDs when present, as shown in earlier research
with laptops [9].

While gesture interaction with an HMD without a camera is limited, such
interaction may not be desirable anyway [13]. Also, other research found that
hand gestures that can be detected by such a camera may not be desirable; e.g.,
Hsieh et al. [5] found in their experiment on a gesture system for use in public
space (using gloves) that most people naturally performed the gestures roughly
around the lower torso. Most participants considered the gestures appropriate
for use in the workplace (and other public conditions).

Nonetheless, a recent survey of 12 augmented reality glasses [11] showed
that all but one have a camera built-in, although it can only take pictures on
some devices. Control options mostly include a limited set of buttons and/or a
touchpad on the glasses or a wired handheld device. Many reviewed AR glasses
also offer voice or gesture control. Taking these studies together may imply that
using other input than that offered by the devices may need to be explored. All
devices should have this possibility as all glasses in the survey by Syberfeldt et
al. [11] provide wireless connectivity options, which could be used to provide
these alternative means of interaction.

## 3 Alternative Touch Interactions

To assess the potential of new multimodal interaction techniques on assembly
lines, we performed an experiment where we compared various placements of
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Left: the assembly instructions are shown with a minimal navigation interface that allows to go to the next/previous step, give feedback on the current step, and get additional help. Middle: The feedback menu allows to record audio, to flag an unclear description or an unclear picture, and to activate the camera with zoom (right).</figcaption></figure>

visual output and touch input and their impact on the perceived workload. Visualizing assembly instructions on a screen, for example, is robust and straightforward, yet cumbersome for the worker if the assembly area is large and she has to carry the tablet with her, for example. Moving the visualization to an AR headset is promising as it allows the worker to look at the instructions at all times if needed. The instructions can also be laid over the real-world components to provide an even more seamless experience [1]. Our goal is to provide a robust touch input surface with basic control of the worker's interface.

We evaluated how the visualization of the instructions (Fig. 1) and the placement of the control interface affects the perceived workload while assembling components. We wanted to investigate the effect of instruction placement (AR or tablet) and control (tablet, forearm, or head) on workload and performance.

## 3.1 Interface Description

Fig. 1 shows the user interface, as shown on the tablet or the AR-headset. The instructions take up most of the screen with the feedback and navigation controls placed vertically on the right side. Feedback regarding the current instruction step can be given at different levels of granularity: to draw attention to a specific step in the instructions, the user can flag that step by pressing the topmost button. This button is intended for fast interaction if the problem is obvious, or the user is unable to provide more information due to time constraints, situational disability, or limited motivation. The light-bulb button leads to the submenu where the user can provide more detailed information, such as adding dedicated flags for *unclear description* or *unclear image*, or an audio recording or a picture. These options would enable motivated operators to provide richer input during assembly.

We implemented our interface in Unity and ran it on an HTC Nexus 9 as a tablet and a Microsoft Hololens as AR glasses. Our wearable device was a 2.8-inch touchscreen (Robopeak RPUSBDISP) attached to a Raspberry PI Zero W running Raspbian, and the control interface was implemented using Python and PyQt. The control interface sent its commands to the visualization application using OSC messages.
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Experiment conditions used in the experiment (A.-D.). We added explicit feedback instructions (1) and AR interface minimization button (2).</figcaption></figure>

## 3.2 Experimental Setup and procedure

Similar to the experiment by Funk et al. [2], participants had to assemble LEGO Models according to given instructions. To level the difficulty of the assembly, we used models from the LEGO Racers Tiny Turbos series (8642 (hereafter referred to as *Red*), 8644 (*Yellow*), 8657 (*Blue*), 8658 (*Black*)). The models had between nine and 11 instruction steps, with the number of parts to be added in a single step ranging from one to seven (M=3.1, SD=1.6) and the total number of pieces to assemble ranging from 27 (*Black*) to 38 (*Red*) (M=32, SD=4.5). All instruction sets contain a phase where the assembled model has to be flipped around to add pieces to the bottom side.

We defined four conditions, shown in Fig. 2. We used the tablet as a baseline condition (A). In the remaining cases, we visualized the instructions on the AR-headset and ran the control interface on a tablet (B), and a wearable device fixed to the glasses (C), or the participant's lower non-dominant arm (D).

Due to the limited display resolution and field-of-view (FOV) of the AR-headset, the instructions covered a large part of the visible area. Participants could, therefore, minimize the instructions to the upper left corner of the FOV using a dedicated button in the control interface (Fig. 2, 2). We deliberately did not fix the instructions to a particular physical location in the AR conditions, as this would basically mimic a dedicated screen at that position, which we already covered by the tablet condition.

We added requests to press specific buttons on the control interface to some of the instruction steps to simulate workers giving feedback (Fig. 2, 1). This allows the participant to decide whether to react to this request immediately or finish the assembly of the components first and provides the same amount of triggers to all participants.

The order of the conditions and the order of the models to be assembled was randomized using Latin squares. After each condition, we asked the participants
<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Left: Mean RTLX by condition Right: Mean task completion time in minutes by condition (Fig. 2). Both graphs indicate mean, quartiles, and whiskers. The only significant difference was between condition C and the other conditions for both workload (RTLX) and completion time.</figcaption></figure>

to fill out a pen-and-paper NASA TLX workload assessment questionnaire. We used the raw scores, frequently referred to as RTLX method [3].

## 3.3 Results

We recruited 10 participants (all male, mean age 31.1 years, SD=6.4) from our lab. We removed one trial for the *blue* and one for the *black* model from quantitative analysis, as the task completion times deviated more than two standard deviations from the mean.

**Workload** The mean TLX scores are reported in Fig. 3. We ran a repeated-measures ANOVA on the mean TLX scores with *condition* and *model* as fixed effects. *User* was modeled as random effect and the TLX scores were normally distributed. *Condition* had a significant effect on the average workload ($F_{3,30.03} = 10.32, p < .0001$). A *post-hoc* Tukey HSD-test showed that the workload in condition C (AR glasses and interaction on glasses) was significantly higher than in all other conditions ($p < 0.039$), but there was no significant difference between the other pairs. The type of model to be assembled did not have a significant effect on the workload ($F_{3,30.03} = 2.75, p = .0603$)

We ran a repeated-measures ANOVA on the mean TLX subscale ratings with *condition* as fixed effect and *user* as random effect. Pairwise *post-hoc* comparisons were made using Tukey HSD tests. We can observe the difficulties with condition C on the individual subscales of the TLX questionnaire. *Condition* had a significant effect on *Mental Demand* ($F_{3,33} = 5.5984, p = .0032$) and *Frustration* ($F_{3,33} = 6.3473, p = .0016$). Only condition C was significantly more mentally
demanding than conditions A and B ($p$ < .0444) and frustrating ($p$ = .0007) than condition A.

**Task Completion Time** Similar to the workload, just having the tablet alone (condition A) leads to the shortest task completion times (Fig. 3 right). We performed a repeated-measures ANOVA on the log-transformed task completion times with *condition* as fixed effect and *user* as random effect. Condition had a significant effect on task completion time ($F_{3,33.01} = 3.3, p$ = .0323). A *post-hoc* pairwise comparison with a Tukey HSD-test showed only the difference between conditions C and A to be statistically significant ($p$ = .032).

**Qualitative Feedback and Observations** After having completed all conditions, we asked the participants which interface they preferred, and seven chose tablet only without the AR glasses. They justified their choice with the well-known look-and-feel and that they could place it at a fixed position of their choice, making it easy to reach. One participant mentioned that he would prefer this interface even more than the original paper instructions because he does not have to look for the step he was at when he shifted his attention to the construction. Three participants said they preferred the combination of AR-Headset and the arm-worn wearable because the instructions are always in the field-of-view

One major issue with the AR-visualization was the Microsoft HoloLens that we used. When justifying their choice for the tablet visualization, participants complained about the weight and the visual quality. This includes the low physical resolution and color depth, but also the low contrast and the fact that it is obstructing the view in general. Six of the seven participants who opted for the tablet as preferred interface mentioned one of the AR-based combinations as second choice with high chances of moving to first choice if the headset was more comfortable.

From the seven participants that opted for the AR visualization as a second choice, five also discussed which control interface they would like to use along with it. Three mentioned the wearable on the arm as of their interest and a request to make it more polished as they did not like the hardware's prototype appearance.

Overall, participants had trouble with the red model's instructions in the AR conditions because it contains many dark pieces that are difficult to perceive with the HoloLens due to low contrast. Two participants had one non-critical piece left over after assembling the *red* model which they forgot in step five of the assembly. With this model, most of the participants had to return a few steps, but all were successful in recovering from that mistake.

We observed that during simple assembly steps, participants overlooked feedback instructions. Participants quickly grasped which pieces were to assemble and how, and then hit the *next* button while returning to the instructions. In a real-world setting, the trigger to give feedback will not be external as in this experiment, but intrinsic to the worker. However, it indicates that additional information needs to be prominent to grab attention from the primary task.
## 3.4 Discussion

In this experiment, we gained insights on how AR visualization and various control interfaces influence how people perceive work on an assembly task. The assembly task we had the users perform was constrained to a small space in front of the user, which allowed them to place the tablet right next to the components and leave it there. This may not always be possible in practice; the components might be located on shelves from which the user has to collect them before the assembly. In some cases, the workspace is much larger than a desk, requiring movement. There might not be free space to place the tablet next to the working area. These factors might tip the preference more towards an AR headset or wearable displays that are more mobile and leave the hands free (most of the time). Nevertheless, more light-weight headsets with a good visual quality are required for a tool that can be worn throughout an 8-hour shift.

The screen placement only had a minor impact on both workload and task completion time compared to the placement of the feedback control panel. Once we move from a very local assembly task to larger items, this input device needs to be carried around, making the wearable control panel much more interesting. This also opens possible workarounds for the problem of the heavy headset, such as a combined input and output on a small touchscreen attached to the lower arm. To visualize assembly instructions, the screen needs to be bigger than the control interface. Smartphones or small tablets are already used to visualize instructions, and a similar size may be appropriate.

Some aspects of the experiment limit generalizability of the results. Given the exploratory character, we opted for a limited amount of participants. This means an experiment with a larger sample size is needed. This experiment is preferably also carried out with assembly operators and AR glasses, rather than an AR headset.

On the practical level, we would pay more attention to the limited contrast of AR devices as it might affect readability more than visible on a regular screen. Ideally, the devices in the different conditions also have a similar level of refinement. Possible changes in the presentation include presenting the instructions on the wearable controller and making it easy to put it on a table. This could create a hybrid between the tablet and AR glasses.

## 4 Conclusion

We studied where assembly instructions should be presented and how interaction with a minimal feedback interface affects workload. Most participants preferred the tablet solution, but they also stated that a more comfortable AR setup, especially with wearable touch-interaction, might be preferred.

Wearable, easily removable, touch controllers, possibly integrating display of instructions in combination with less bulky AR glasses, seem promising. The latter are already available [11]. Before definitive conclusions can be drawn on which interface technologies are best suited in such settings, more refined prototypes and evaluation with a large group of operators are still needed.
**Acknowledgements** The work has been funded by the FlandersMake project OperatorKnowledge (grant number: HBC.2017.0395). The committee of ethics of Ghent University has approved the study (approval 2018/0439).

References

1. Barfield, W. (ed.): Fundamentals of Wearable Computers and Augmented Reality. CRC Press (2015). https://doi.org/10.1201/b18703

2. Funk, M., Lischke, L., Mayer, S., Shirazi, A.S., Schmidt, A.: Teach Me How! Interactive Assembly Instructions Using Demonstration and In-Situ Projection, pp. 49–73. Springer Singapore, Singapore (2018). https://doi.org/10.1007/978-981-10-6404-3_4

3. Hart, S.G.: Nasa-task load index (nasa-tlx); 20 years later. In: Proceedings of the human factors and ergonomics society annual meeting. vol. 50, pp. 904–908. Sage publications Sage CA: Los Angeles, CA (2006)

4. Haug, A.: Work instruction quality in industrial management. International Journal of Industrial Ergonomics 50, 170 – 177 (2015). https://doi.org/10.1016/j.ergon.2015.09.015

5. Hsi eh, Y.T., Jylhä, A., Orso, V., Gamberini, L., Jacucci, G.: Designing a willing-to-use-in-public hand gestural interaction technique for smart glasses. pp. 4203–4215. CHI ’16, ACM, New York, NY, USA (2016). https://doi.org/10.1145/2858036.2858436

6. Koelle, M., Kranz, M., Möller, A.: Don’t look at me that way!: Understanding user attitudes towards data glasses usage. pp. 362–372. MobileHCI ’15, ACM, New York, NY, USA (2015). https://doi.org/10.1145/2785830.2785842

7. Kong, X.T.R., Luo, H., Huang, G.Q., Yang, X.: Industrial wearable system: the human-centric empowering technology in industry 4.0. Journal of Intelligent Manufacturing (Apr 2018). https://doi.org/10.1007/s10845-018-1416-9

8. Lukowicz, P., Timm-Giel, A., Lawo, M., Herzog, O.: WearIT@Work: Toward Real-World Industrial Wearable Computing. IEEE Pervasive Computing 6(4), 8–13 (Oct 2007). https://doi.org/10.1109/MPRV.2007.89

9. Portnoff, R.S., Lee, L.N., Egelman, S., Mishra, P., Leung, D., Wagner, D.: Somebody’s watching me?: Assessing the effectiveness of webcam indicator lights. pp. 1649–1658. CHI ’15, ACM, New York, NY, USA (2015). https://doi.org/10.1145/2702123.2702164

10. Stanford, V.: Wearable computing goes live in industry. IEEE Pervasive Computing 1(4), 14–19 (Oct 2002). https://doi.org/10.1109/MPRV.2002.1158274

11. Syberfeldt, A., Danielsson, O., Gustavsson, P.: Augmented reality smart glasses in the smart factory: Product evaluation guidelines and review of available products. IEEE Access 5, 9118–9130 (2017). https://doi.org/10.1109/ACCESS.2017.2703952

12. Thomas, B., Grimmer, K., Zucco, J., Milanese, S.: Where Does the Mouse Go? An Investigation into the Placement of a Body-Attached TouchPad Mouse for Wearable Computers. Pers. and Ubiqu. comp. 6(2), 97–112 (Jan 2002). https://doi.org/10.1007/s007790200009

13. Werrlich, S., Nguyen, P.A., Daniel, A.D., Yanez, C.E.F., Lorber, C., Notni, G.: Design recommendations for hmd-based assembly training tasks. In: SmartObjects@ CHI. pp. 58–68 (2018)