# Augmented Reality Technology for Displaying Close-Proximity Sub-Surface Positions

Kasper Hald, Matthias Rehm, Thomas B. Moeslund

► To cite this version:

Kasper Hald, Matthias Rehm, Thomas B. Moeslund. Augmented Reality Technology for Displaying Close-Proximity Sub-Surface Positions. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.641-659, 10.1007/978-3-030-29384-0_38. hal-02544610

HAL Id: hal-02544610

https://inria.hal.science/hal-02544610

Submitted on 16 Apr 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Augmented Reality Technology for Displaying Close-Proximity Sub-Surface Positions

Kasper Hald, Matthias Rehm, and Thomas B. Moeslund

Department of Architecture, Design & Media Technology
Aalborg University, Aalborg, Denmark

kh@create.aau.dk

**Abstract.** When designing human-system collaboration to assist in strenuous manual tasks we need to develop methods of communication between the system and the human. In this paper we are evaluating augmented reality (AR) technologies for displaying task-relevant information when the target is on a work surface for a typically standing work operation. In this case we are testing AR interfaces for displaying sub-surface positions. To do this we compare four types of AR interfaces, a head-mounted see-through display, a mounted see-through display, top-down surface projection and graphical overlays on a static monitor. We performed the experiment with 48 participants. Data analyses show significant difference between the AR interfaces in terms of task completion times and user satisfaction with the projection-based display being the fastest and most satisfying to the participants.

**Keywords:** Augmented Reality · Usability Testing.

## 1 Introduction

Repetitive strenuous movements involved in production work can lead to musculoskeletal diseases in the long term [9]. This issue can be addressed by introducing assistive and collaborative systems that can relieve some of the strain. This also has the potential to increase productivity. In order to do this we need to develop communication methods between the system and the human, from here referred to as the operator. The communication from the system to the operator in the production context will involve conveying the details pertaining to the current task, and for this study we seek to utilize augmented reality (AR) interfaces for this purpose.

The studies and development are done in the context of industrial meat production in which employees stand on the meat processing lines. In this case the term operator will refer to a single production employee collaborating with an instance of the system's assisting agents. The system will be assisting in tasks involved in sequential meat processing where each employee performs one task on each piece of meat for a period of time. These tasks include positioning the meat, cutting it down to size, trimming fat layers or picking out impurities. For this test we focus on the latter task and develop methods for the system to show
the position of an impurity, whether it be on or under the surface of the meat, that it can be addressed and removed by operator.

Because the operators will be working in close proximity with potentially hazardous hardware it is critical that task-relevant information can be displayed non-obstructively to the operator in order to communicate the current objective of the system. This will allow the operator to anticipate the actions of the system, leading to safer collaboration and improve human trust in the system as the communication is developed further.

We focus on evaluating AR interfaces since these can be used hands-free. The goal is to evaluate and compare four different types of AR interfaces in terms of effectiveness, ergonomics and user acceptance when showing sub-surface positions in an opaque mass acting as the analog for the meat. The four types of AR interfaces are a head-mounted see-through display, a tablet-based see-through display mounted to an adjustable arm, top-down surface projection and graphical overlays on a static monitor.

## 2 Related Research

The communication from the system to the operator in the manufacturing context will involve conveying the details pertaining to the current task. Novak-Marcinin et al. [8] define augmented reality-aided manufacturing (ARAM) as the overlap of AR-aided robot control, AR-aided testing, AR-aided assembly and AR-aided transport and storage. The experiment in this paper is to evaluate interfaces for AR-aided assembly, because of the meat production context, where it will be used to aid production staff.

Regarding preliminary evaluation of AR devices, Elia et al. [3] proposed a 4-step model to be applied in specific manufacturing processes. The first step is a multi-criteria analysis for ranking the most effective AR systems for the purpose, which is the current stage of this project. The ranking is performed by comparing the hardware options in terms of output modalities, reliability, responsiveness and agility. The ranking is done using pair-wise comparison followed by analysis and ranking of the AR devices. Elia et al. [3] categorize types of AR hardware as head-mounted displays (HMD), handheld devices, projectors and haptic force feedback systems. The second step is obtaining a judgment matrix using pairwise comparison between criteria, followed by evaluation of local weights and consistency of comparison in step 3, with final ranking of devices as step 4.

Kruijff et al. [6] classified potential issues with AR caused by a combination of the environment of use, capturing the environment, the method of augmentation, the types of display device and user. They also point out whether these issues are predominant with particular display types which are categorized similarly to Elia et al. [3]: Head-mounted displays (video see-through or optical see-through), handheld mobile devices or projector-camera system (stationary or mobile). Relevant issues for this study include wearable see-through displays having limited field of view (FOV) and vergence-accommodation conflict for virtual objects and surface-based distortions for projector-based setups.
For this experiment we consider the environment of use be recreating the relevant working conditions pertaining to freedom of movement and posture allowed in a standing task. The four categories outlined by Elia et al. and Kruijff et al. have all been considered for the experiment. However, seeing as all of them are primarily visual aids as opposed to haptic force feedback systems, the latter is not included in this experiment. Since the context of the study allows for handheld devices to be implemented in combination with existing production tools, we would consider haptic feedback as a possible addition to visual augmented reality, so it may be evaluated as an addition at a later stage.

Human-system collaboration enabled by AR has been studied previously, often in the context of human-robot collaboration. However, often these tests have not been performed in the context of a close-proximity task with the user standing at a table. Green et al. [4] tested a human-system collaboration system utilizing an HMD where the operator was sitting at a table. However, the headset used in this case, rather than being a see-through AR display, was an eMagin Z800 headset using OLED displays with the augmented video fed from a mounted webcam. With the potential to have the webcam pointed downwards toward the table the operator would be relieved from bending their neck to look directly down at the tracking markers. The paper does not specify any angle adjustment in the implementation. This solution is not considered for this comparison due to the potential hazard or limiting the operators field of view as opposed to see-through HMDs which allow the user to still see outside of the display field. Even-though not pertaining to a task specifically, Vogel et al. [13] proposed using projection-based AR to show an outline of a safe working area in relation to the collaborative system.

Schwerdtfeger et al. [10] go into depth describing the projected AR, specifically using lasers. They points out the cons of HMD AR devices, those being narrow field of view, limited resolution, swimming effect, multiple focus planes as well as eye fatigue. While laser-projected AR can address some of these issues it is limited to displaying information on surfaces in the environment and the image must be distorted to compensate for environment geometry and viewing angle, whether the projector is head-mounted or stationary. In addition, it also introduces the challenge of occlusion by either the user or other objects. In order to avoid surface distortion for this test we use an even surface for this comparison.

Swan et al. [12] studied how depth perception is affected while using AR devices in that subjects tended to underestimate distance in AR when they are projected at less than 23 meters distance to the user, after which the bias switches to overestimation. Comparing this to short distance error, Singh et al. [11] estimated an error of -5.5 cm at most for distances less than 50 cm. From these finding we should expect our participants to underestimate the target distances in our test. However, we can not know if this is true when the user can judge distance in relation to a real surface as opposed to judging a target hanging in the air.
Similarly to showing sub-surface positions, augmented reality has previously been used to imitate x-ray vision. Avery et al. [1] emphasized that when showing the content beyond the surface using an graphical overlay it should include an edge overlay representing the surface as a depth cue, using occlusion as a depth cue so the object does not appear to float in front of the surface. We are implementing the same method in our test applications by projecting a graphical grid overlay on the surface of our meat analog.

The main contribution of this experiment to the fields of human-robot collaboration and ARAM is in the comparison between the AR devices, but the significance is in the environment and conditions it will be utilized, as comparing the systems in a low-distance setup while standing at a work surface has rarely been done.

## 3 Setup

The experiment is performed using a tray of sand as the analog for a cut section of meat, as it allows the test subject to poke into it with a tool to address the impurities that will be displayed in it using the AR devices. We use dry loose sand with low density so allow easy entry and to prevent visible entry points to stay throughout a test session. The sand also allows for the surface to be smoothed out by hand.

The participant's performance is measured using an HTC Vive controller with a nail mounted to the bottom as shown in Figure 1. Using the six degrees of freedom tracking capabilities of the controller the participants are using the tip of the nail to poke into the body of sand as closely as they can while holding the controller in their dominant hand. A second Vive controller is held in the non-dominant hand and the participants will use a button on it to confirm when they have hit a target. Confirming with the non-dominant hand prevents shaking the tip of the nail during a button press.

The HTC Vive tracking space is running using a Windows 10 PC which also acts as the host for the test application, developed in Unity 3D. In order to show the target position on the HMD and the mounted tablet, both of which run Android, the PC acts as network host and sends target positions to the devices acting as clients over a wifi connection throughout the test sessions.

The surface position of the sand is calibrated in the tracking space using the tracked nails on the Vive controller along with a printed piece of calibration paper which also fits in-between the tracking markers used for the HMD and mounted tablet. The tracking markers are positioned at either side of the testing area with the goal of having at least one visible to the tracking camera regardless of where the user is looking and to prevent occlusion of the trackers. The full test setup is shown in Figure 2.
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** The needle used for the experiment, made from a 3D printed mount and a nail attached with a bolt in the loop designed for the wrist strap.</figcaption></figure>

# 4 AR Devices

We are testing four different types of AR devices. Three of them are based on three of the types outlined by both Elia et al. [3] and Kruijff et al. [6]; head-mounted see-through display from here referenced as HMD, see-though mobile display and projection-based AR. In addition, we are comparing a video feed on a monitor augmented with graphical overlays because this is currently a typical way of displaying information in meat production settings. The software for the four AR devices are developed using Unity 3D. Similar to all of the versions is that a green grid is projected aligned with the surface of the sand, similarly to what is described by Avery et al. [1].

The targets are shown as a red sphere or circles at 10 mm in diameter projected into the sand. Due to the nature of the devices the display methods for the targets differ between the displays. Because the image for the top-down projector and the stationary screens are limited to 2D projections on the surface of the sand, the red dot is shown along with a number indicating the depth of the target in millimeters. We considered using similar labels for HMD and tablet devices, but because the user will see the target projected from different angles depending on their viewing angle as opposed to always seeing it from the top and directly down. Both the HMD and tablet show 3D-rendered images, allowing for occlusion and motion parallax as occlusions by the green grid as cues, which is not possible for the remaining displays without using head-coupled perspective.
<figure><img src="image_3.png" /><figcaption>Fig. 2. The full setup for the AR test [5].</figcaption></figure>

Because of the varying nature and performances of the displays, accurate calibration between all devices proved very difficult. Because of this, this study focuses on the accuracy spread between each device. In practice this means that the accuracy for each device is measured by the offsets from the median offset from the targets for each participant. By doing this we assume that the median hit is an accurate hit as adjusted to the user.

## 4.1 Head-Mounted See-Through Display

The HMD using an Epson Moverio Bt-300 which is an Android-based device and is equipped with 0.43 inch wide panel 720p displays at a 30 Hz refresh rate. The software is implemented using Unity 3D with the Vuforia AR plugin. The HMD is shown in Figure 3. The impurities are projected into the sand on the glasses while also utilizing the overlay grid.

Using Vuforia along with the build-in 5 megapixel camera on the right-hand side of the headset the system is tracking using the AR markers at either end of the surface of the sand. The goal is to avoid the participants occluding the trackers by only having to use one hand for the tasks, leaving the tracker on the opposite side exposed. The feed from the camera itself is not displayed on the HMD, only the grid targets are displayed. The overlay is manually offset and
rotated to best fit the surface of the sand. This process includes adjusting the rendering FOV to fit the display area of the glasses. In this case the FOV is set to 21 degrees, despite the manufacturer advertising 23 degree FOV for the device.

Since we are not able to perform eye tracking using the hardware to determine the convergence point, rendering the position in stereo would leave the user with difficulty focusing. Because of this the target is not rendered in stereo, but rather as a 2D overlay similarly to the see-through display, and when analyzing the data the participants' dominant eye must be taken into consideration.

<figure><img src="image_4.png" /><figcaption>Fig. 3. The Epson Moverio Bt-300 glasses used for the experiment. [5]</figcaption></figure>

## 4.2 Mounted See-Through Display

Similarly to the HMD, the see-through tablet display is running Android and Unity with Vuforia. The display is shown in Figure 4. The tablet is mounted to an adjustable stand, so it can be positioned according to the user's height while oriented to show the entire surfaces area of the sand and all the targets in frame. For this test the tablet is manually adjusted to each subject and remains stationary where an AR tracking marker is visible.

We focus on the marker opposite the user's dominant hand to prevent occlusion. The participant also has to be able to reach underneath the tablet and their hands are visible throughout the test sessions and they can coordinate their movement with the targets in the camera feed. Since the tablet remains mostly stationary during each session, Vuforia is mainly used when the tablet is
initially adjusted for the user, this device is less susceptible to errors and latency
introduced by Vuforia.

<figure><img src="image_5.png" /><figcaption>**Fig. 4.** The arm-mounted see-through display setup with impurity projected into the sand as a red dot assisted by a grid overlay aligned with the surface. [5]</figcaption></figure>

4.3 Top-Down Projection

This device is set up using a projector mounted to a tripod with a 3D printed adapter and pointed down towards the sand. The projection is shown in Figure 5, bottom left. The projection is adjusted using a piece of paper with a grid printed on it matching the green grid overlay. The projected overlay features the green grid and the targets are shown as red dots with depth in millimeters shown next to them.

4.4 Stationary Screen

The monitor setup utilizes a 25 inch PC monitor with an aspect ratio of 16 by 10.
The monitor is shown in Figure 6, bottom right. We use a Logitech C920 on an
adjustable mount pointed straight down to the surface of the sand. Similarly to
the top-down projection setup, the camera feed on the monitor features the green
grid overlay and the targets are displayed along with label indicating their depth.
Similarly to the tablet setup, the user will be seeing their hands in the video
<figure><img src="image_6.png" /><figcaption>**Fig. 5.** The grid projected onto the sand with the targets shown as a red dot. [5]</figcaption></figure>

feed during the test sessions. The camera position is calibrated using the printed
d grid and by placing an HTC Vive controller in the video feed and matching its
position with the representation on the monitor.

5 Experiment

Each participant is introduced to the topic at the beginning of the experiment and they are asked to sign a consent form, followed by a questionnaire pertaining to their age, sex, height, dominant hand and dominant eye. In cases where participants do not know their dominant eye, it is determined with the Miles Test [7].

**5.1 Pointing Tasks**

Each participant performs a set of pointing tasks with each AR device. The order
of the devices is counterbalanced between participants in order to counter bias
and initial confusion about the tasks with the first devices. With four systems
we have 24 possible combinations. We do each combination twice to reach a total
of 48 participants.

While wielding the HTC Vive controller the participant is asked to point
the tip of the nail in the center of the target as they appear, as quickly and
precisely as they can. To do this they will be asked to have the HTC Vive
controller rest on the side of their dominant hand between the thumb and index
finger as shown in Figure 1. When the participant believes they have reached
<figure><img src="image_7.png" /><figcaption>Fig. 6. The monitor and top-down camera. [5]</figcaption></figure>

the target center, they must confirm by pressing the trackpad on the controller in the non-dominant hand. To prevent accidental double-presses there is a delay of one second from when a target is confirmed to when the next target is shown, during which the confirm button is disabled. This also allows the participants to return their hand to a natural position close to their body, but they are not instructed to do so.

The sand is held in a rectangular plastic tray and has a depth of 5 centimeters. The targets are shown at depths of 0, 5 and 10 millimeters, meaning that participants can not expect to drive the nail to the bottom of the tray and get a precise hit. At each of the three depths the targets are distributed on two rows and four columns with 10 centimeters of spacing in both dimensions, so the participants will get to both reach across and away from the center of their body. With a 24 targets per device per user, we get a total of 4608 samples. The rectangular shape for the tray and the sand is appropriate for the experiment considering that at the points of fine operation in meat production, such as picking out impurities, the meat has been cut down to these shapes.

For each target we measure the task completion time from when a new target appears until the participant confirms a hit. This includes time spent searching for the targets. The accuracy is measured as the offsets from the target center to the tip of the nail at confirmation for the three dimensions individually as well as the absolute distance from target to tip. After using each of the four devices
the participants get to evaluate the device in terms of acceptance and ease of use with System Usability Scale (SUS) [2].

## 5.2 Hypotheses

With the experiment we aim to prove that there are significant differences in the effectiveness and user experience between the AR display types, specifically in terms of the following hypotheses:

- H1: Spotting and hitting sub-surface targets when using different AR devices will yield different task completion times.

- H2: Hitting sub-surface targets using the different AR devices will yield different hit accuracy.

- H3: The different AR devices have different usability based on the Standard Usability Score.

# 6 Results

As described in Section 4, the four devices can be split into two categories, the AR glasses and mounted tablet showing depth through perspective, occlusion and parallax while the remaining are limited to 2D overlays. In this section these categories are described as the 3D and the 2D enabled devices, respectively.

The experiment was performed with 48 participants, average age 24 years, ranging between 20 and 34. Nine participants were female, ten were left-handed. 21 reported having left-eye dominant, 25 reported right, one reported both eyes to be dominant and one could not be determined. Eight participants had positive eyeglass prescriptions, ten had negative prescriptions and two had unknown prescriptions.

## 6.1 Data Analyses

Performing analyses of variance with significance threshold at .05 shows significant difference among the AR devices in task completion times when comparing them individually (F(2,3926) = 333.3, p < .001) and when comparing the 3D and 2D enabled devices (F(1,3926) = 124.7, p < .001) with Tukey's HSD post-hoc analysis showing significant difference between all devices. The projection-based display yielded the lowest average time (2.3 seconds), followed by the stationary screen (3.55 seconds), the mounted see-through display (4.5 seconds) and the AR glassed (9.92 seconds) had the longest average task time. A summary of the task time data is shown in the box plot in Fig. 7. These results confirm the first hypothesis.
<figure><img src="image_8.png" /><figcaption>**Fig. 7.** Box plot of the task completion times between the four AR interfaces. Next to each box plot are their mean values and confidence intervals.</figcaption></figure>

To analyze the accuracy between AR devices, in order to compensate for dominant eye and calibration inaccuracies between test participants, the samples are corrected according to their per-session median hit value. By doing this we make the assumption that the median offsets are for a precise hit, making the median values zero for the data sets used. Figure 8 shows the per-session median hit values for each condition and participants. The high median offsets for depth suggest a calibration error.

Performing a multiple analysis of variance, dependent values being the hit offsets on three separate dimension, there is significant difference between the four AR devices (F(2,3926) = 6, p < .01) as well as when comparing the 3D and 2D enabled devices (F(1,3926) = 3, p < .002), confirming the second hypothesis. The handedness of the user did not yield significant difference in hit offsets.
<figure><img src="image_9.png" /><figcaption>**Fig. 8.** Per-session surface median hit offset values for each participant (left) and histogram for depth offset medians in bins of 5 mm (right) for each condition.</figcaption></figure>

Looking at hit spread, the AR glasses have the largest average standard
deviation at 51 millimeters in target hit offsets for all three dimensions. In com-
parison, the stationary screen, projection and tablet displays have averages of
10, 28 and 30 millimeters, respectively.

Investigating the difference in sideways offsets in relation to the user for the
four devices (F(2,3926) = 3.36, p < .05), a Tukey's HSD post-hoc analysis shows
no significant difference between any pair of devices, suggesting the difference
depends on whether the interfaces were 3D or 2D enabled (F(1,3926) = 9.26,
p < .003). Similarly for offsets going towards or away from the body is only
significantly difference between the two groups (F(1,3928) = 8.27, p < .005).
The average offsets along with standard deviations along the surface are shown
in Figure 9.
<figure><img src="image_10.png" /><figcaption>**Fig. 9.** The average offsets along with standard deviations along the surface between the four AR devices.</figcaption></figure>

For the depths offsets as well there is only significant difference between the two categories (F(3,3926) = 3.4, p= .017) and all of the average offsets are within one millimeter of each other. The depth offsets are shown in Figure 10.

The SUS scores show significant difference between the four devices (F(3,186) = 92.03, p < .001) with Tukey's HSD post-hoc analyses showing significance between all four devices. There is not significant difference in user acceptance when comparing 2D and 3D devices. However, two of the four devices do not have normally distributed scores according to the Shapiro-Wilk normality test,
<figure><img src="image_11.png" /><figcaption>**Fig. 10.** Average depth offsets between the four AR devices along with confidence intervals.</figcaption></figure>

the AR glasses ($p = .062$) and the mounted see-through display ($p = .16$), making the third hypothesis harder to retain. The SUS scores are summarized in Figure 11.

## 6.2 Observations

The AR glasses had the longest overall task completion time. The long task times are likely due to the narrow FOV, for multiple reasons. Firstly, the FOV does not allow for the user to see the augmentation overlay on the entire surface at once, requiring additional search time to the task. Despite them being able to see all of the surface at once, the AR glasses only cover a small segment of their FOV, creating a letterbox effect. Despite the mounted tablet display tracking and showing targets in similar way, its position and FOV did allow for user to see the entire surface at once, eliminating the need for search time.

The limited FOV of both the display and the camera used for the tracking in combination with the short distance to the surface of the sand made it difficult for the participants to inspect the entire surface area while also keeping the tracking marker in view of the camera. This in combination with errors and latency introduced by Vuforia made a sub-par user experience.

In addition, the glasses' FOV does not allow the user to glance downwards, leading them to turn their head downwards to an uncomfortable degree as they leaned in over the tray, while also having to turn their head to look around and
<figure><img src="image_12.png" /><figcaption>**Fig. 11.** Box plot of the SUS scores between the four AR interfaces and a line at the cut-off value of 68. Next to each box plot are their mean values and confidence intervals.</figcaption></figure>

search for targets. A few test participants commented on this, stating that using
the glasses was starting to give them neck pains. Many participants who tried
other devices after the glasses would comment out loud on how much easier it
was.

# 7 Discussion

Despite the accuracy of the AR devices proved significantly different, it is hard to define a concrete set of tendencies, seeing as significance differs between devices and axes and the average hit positions differ as seen in Figure 9. Also, the average hit accuracy in depth, though significantly different, are within one millimeter of one another, making it less relevant in a real-world context. It does show, however, that the projection based AR and the monitor were the two devices with the lowest standard deviations along the surface.

The analyses show significant differences when comparing the 2D and 3D
enabled interfaces, both for task completion time and accuracy. For the latter
this grouping is more consistently significant, because the post-hoc analyses show
no significant difference in pairwise comparison. This may be due to both of
them being static and showing the positions from a top-down perspective as
opposed to the see-through display which required interpretation of perspective
and occlusion as depth clues and the glasses that required constant tracking as
the user moves, introducing noise. The tracking noise would also be introduced
for the mounted displays in a real-world setting as the display would be moved
around. Nevertheless, the inaccuracy for the projection and the monitor can
likely be fixed with hardware and software adjustment, where the remaining
devices have the challenge of tracking and depths communication.

The SUS results show that the projection-based AR and the monitor-based
AR were the only two devices with averages scores reaching above the standard
cut-off point of 68. The analysis of variance was used despite not all of the groups
being normally distributed because due to the nature of the SUS scale where
groups will be tailed in different directions dependent on their average position
on the scale, making them hard to fit in any statistical model.

The low task completion time combined with the high SUS score for the
projection-based AR system is likely due to the direct connection between the
display and the target, the interfaces being on the target itself. Eliminating the
requirement of coordinating hands with a display offset from the target seems to
make the interface more accessible. This is despite the problem with projection-
based AR that the user will occlude the projection when interacting with it.

The issues with the AR glasses described in Section 6.2 make the Epson
Moverio non-viable for near-distance tasks. An alternative would be the tech-
nique used by Green et al. [4] where the user's viewing direction would be shifted
downwards, compensating for the FOV. However, whether this approach, occlud-
ing part of the user's FOV during a potentially hazardous task, would work is
uncertain. It would be beneficial to repeat the experiment using a HMD specifi-
cally designed for the context of close-proximity tabletop operations. This would
involve expanding the display FOV, allowing the user to search by scanning with
their eyes rather than turning their head straight at the target. Another required
feature is eye-tracking in order to properly implement stereo display.

It is worth considering after this experiment whether 2D and 3D enabled AR devices are comparable in terms of accuracy and user experience due to differences in affordances. Because the projection-based overlay and the screen are static displays, the only tracking involved in a real-life scenario is of the meat and the targets as the subject is moved around work surface. As such, these two displays potentially introduce less tracking noise, which is also likely to have affected the test results.

The results and conclusions to this experiment are mainly valid in the sim-
ulated context and would benefit from repeating in a setup with real meat. In
that case, the results and observations illustrate the limitations of the AR glasses
as implemented in this experiment, which should be addressed before they are
assessed with real meat, either by different hardware or tracking solutions.
# 8 Conclusion

This paper presents a comparative study of the usability of four types of AR displays for showing sub-surface impurities in meat by having the participants point to targets inside an analog made of sand. The four display types are wearable AR glasses, a mounted see-through display, projection-based AR and a monitor displaying a top-down video feed with graphical overlay. The goal is to determine suitable interfaces for augmented meat production.

After performing the evaluation with 48 participants our three hypotheses were retained with significantly different task completion time, accuracy and user acceptance depending on the AR display type used. Data analysis indicates that projection-based AR yields the second-lowest variance in combination with the lowest task completion times and the highest SUS score, making it the most suited for the task with the mounted see-through display and stationary screen being viable alternatives, while the AR glasses showed to be non-viable for near-distance tasks as they were implemented for the experiment.

Even though the research is aimed at the meat production industry, the results are relevant to any industry that utilized manual processes while standing at a work surface. Whether the results would pertain to sitting tasks at a desktop might require further testing due to the difference in distance and postures.

As stated in Section 7, for further comparison between the devices, the limitations of some of the devices have to be addressed with more suitable hardware and tracking solutions before they are tested further in a real-world context. That is if the 2D and 3D-enabled devices can be considered comparable from an accuracy perspective due to the difference in affordances. Even so, at early stages of development it is relevant to compare them in terms of the usability and user acceptance that stems from the different affordances.

# Acknowledgement

This research is sponsored by Innovation Fund Denmark and is part of the project, Augmented Cellular Meat Production.

# References

1. Avery, B., Sandor, C., Thomas, B.H.: Improving Spatial Perception for Augmented Reality X-Ray Vision. In: 2009 IEEE Virtual Reality Conference. pp. 79-82. IEEE (mar 2009). https://doi.org/10.1109/VR.2009.4811002, http://ieeexplore.ieee.org/document/4811002/

2. Brooke, J., et al.: Sus-a quick and dirty usability scale. Usability evaluation in industry 189 (194), 4-7 (1996)

3. Elia, V., Gnoni, M.G., Lanzilotto, A.: Evaluating the application of augmented reality devices in manufacturing from a process point of view: An AHP based model. Expert Systems with Applications 63, 187-197 (nov 2016). https://doi.org/10.1016/j.eswa.2016.07.006, http://linkinghub.elsevier.com/retrieve/pii/S0957417416303505
4. Green, S.A., Chase, J.G., Chen, X., Billinghurst, M.: Evaluating the Augmented Reality Human-Robot Collaboration System. In: 2008 15th International Conference on Mechatronics and Machine Vision in Practice. pp. 521–526. IEEE (dec 2008). https://doi.org/10.1109/MMVIP.2008.4749586, http://ieeexplore.ieee.org/document/4749586/

5. Hald, K., Rehm, M., Moeslund, T.B.: Testing Augmented Reality Systems for Spotting Sub-Surface Impurities. pp. 103–112. Springer, Cham (aug 2019). https://doi.org/10.1007/978-3-030-05297-3-7

6. Kruijff, E., Swan, J.E., Feiner, S.: Perceptual issues in augmented reality revisited. In: 2010 IEEE International Symposium on Mixed and Augmented Reality. pp. 3–12. IEEE (oct 2010). https://doi.org/10.1109/ISMAR.2010.5643530, http://ieeexplore.ieee.org/document/5643530/

7. Miles, W.R.: Ocular dominance demonstrated by unconscious sighting. Journal of Experimental Psychology 12(2), 113 (1929)

8. Novak-Marcincin, J., Barna, J., Janak, M., Novakova-Marcincinova, L.: Augmented Reality Aided Manufacturing. Procedia Computer Science 25, 23–31 (jan 2013). https://doi.org/10.1016/j.procs.2013.11.004, https://www.sciencedirect.com/science/article/pii/S187705091301209X

9. Punnett, L., Wegman, D.H.: Work-related musculoskeletal disorders: the epidemiologic evidence and the debate. Journal of Electromyography and Kinesiology 14(1), 13–23 (feb 2004). https://doi.org/10.1016/j.jelekin.2003.09.015, https://www.sciencedirect.com/science/article/pii/S1050641103001251?via%3Dihub

10. Schwerdtfeger, B., Pustka, D., Hofhauser, A., Klinker, G.: Using laser projectors for augmented reality. In: Proceedings of the 2008 ACM symposium on Virtual reality software and technology - VRST '08. p. 134. ACM Press, New York, New York, USA (2008). https://doi.org/10.1145/1450579.1450608, http://portal.acm.org/citation.cfm?doid=1450579.1450608

11. Singh, G., Swan, J.E., Jones, J.A., Ellis, S.R.: Depth judgment measures and occluding surfaces in near-field augmented reality. In: Proceedings of the 7th Symposium on Applied Perception in Graphics and Visualization - APGV '10. p. 149. ACM Press, New York, New York, USA (2010). https://doi.org/10.1145/1836248.1836277, http://portal.acm.org/citation.cfm?doid=1836248.1836277

12. Swan, J., Livingston, M., Smallman, H., Brown, D., Baillot, Y., Gabbard, J., Hix, D.: A Perceptual Matching Technique for Depth Judgments in Optical, See-Through Augmented Reality. In: IEEE Virtual Reality Conference (VR 2006). pp. 19–26. IEEE. https://doi.org/10.1109/VR.2006.13, http://ieeexplore.ieee.org/document/1667622/

13. Vogel, C., Poggendorf, M., Walter, C., Elkmann, N.: Towards safe physical human-robot collaboration: A projection-based safety system. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems. pp. 3355–3360. IEEE (sep 2011). https://doi.org/10.1109/IROS.2011.6094550, http://ieeexplore.ieee.org/document/6094550/