# Testing Augmented Reality Systems for Spotting Sub-Surface Impurities

Kasper Hald, Matthias Rehm, Thomas B. Moeslund

► To cite this version:

Kasper Hald, Matthias Rehm, Thomas B. Moeslund. Testing Augmented Reality Systems for Spotting Sub-Surface Impurities. 5th IFIP Working Conference on Human Work Interaction Design (HWID), Aug 2018, Espoo, Finland. pp.103-112, 10.1007/978-3-030-05297-3_7. hal-02264613

HAL Id: hal-02264613

https://inria.hal.science/hal-02264613

Submitted on 7 Aug 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Testing Augmented Reality Systems for
## Spotting Sub-Surface Impurities

Kasper Hald, Matthias Rehm, and Thomas B. Moeslund

Department of Architecture, Design and Media Technology
Aalborg University, Aalborg, Denmark
kh@create.aau.dk

**Abstract.** To limit musculoskeletal disorders we are working towards implementing collaborative robotics in strenuous or repetitive production work. Our objective is to evaluate augmented reality (AR) devices for assisting in near-distance tasks before applying and testing the displays in the context of human-robot collaboration in a production setting. This chapter describes the hardware setup and procedure for testing AR systems for showing sub-surface positions of foreign elements in an opaque mass. The goal is to test four types of setup in terms of user accuracy and speed, the four setups being a head-mounted see-through display, a mounted tablet-based see-through display, top-down surface projection and overlays on a static monitor. The experiment is carried out using a tracked HTC Vive controller with a needle attachment. Precision tasks are performed by 48 participants and each display is evaluated using the System Usability Scale and the NASA Task Load Index.

**Keywords:** Augmented Reality · Usability Testing · Human-Robot Collaboration.

# 1 Introduction

Repetitive actions at work can cause musculoskeletal disorders (MSDs), whether it is due to non-neutral body posture in an office environment or repetitive motion patterns on a production line [7]. Introducing human-robot interaction (HRI) and augmented reality (AR) can assist in strenuous physical tasks and present data in a seamless and non-obstructive ways, improving work flow and employee welfare by addressing and reducing causes of MSDs.

The main context of this study is cellular meat production in which employees stands on the meat processing line, performing one or few tasks each on every piece of meat. This type of work involves lots of repeated strenuous movement. The theories and experiments will, however, pertain to creating an efficient and sustainable workplace in the general sense, regardless of the type of working environment.

The robots will act as assistants, aiding in production by pointing out details such as impurities in the meat before moving on to human-robot collaboration where the robot will aid in physical tasks, such as flipping over the meat or
bending it for easy trimming and deboning of pork bellies. The overall goal is
safe and efficient communication and collaboration between human and robot,
aided by AR technologies.

Human-robot collaboration is an emerging field within HRI, as can be seen
from dedicated workshops and track at HRI conferences[5]. A particular chal-
lenge challenge in HRI is the asymmetrical communication capabilities of the
human and the robot partner, creating a need for a multi-level coordination
between them to achieve successful interaction: The communication level, the
physical level, the social level and the task level, the communication and physi-
cal level being the main focuses of this experiment.

The objective of this experiment is to evaluate the utility of AR systems
for close-distance tasks relevant to the meat production context before applying
them to human-robot communication. In this case the task is finding impurities
in pork bellies in the context of cellular meat production. In more general terms
this means showing positions in space within and opaque mass. The display
types used are an arm mounted see-through display, head-mounted see-through
display, top-down surface projection and overlays on a static monitor.

We propose a method of helping workers maintain a healthy posture implic-
itly while directing their attention towards a target. One way of doing this is to
use the arm-mounted display and placing it in a position that would require the
user to correct their posture in order to properly utilize the display. This poster
describes the procedure for an upcoming preliminary study in AR systems, the
results of which will be used for further study in HRI.

## 2 Related Research

The communication from the robot to the human in the manufacturing context
will involve conveying the details pertaining to current task, and for this study
we seek to utilize the AR for this purpose. Elia et al. proposed a 4-step model for
preliminary evaluation of AR devices to be applied in specific manufacturing pro-
cesses, starting with a multi-criteria analysis for ranking the most effective AR
systems for the purpose. This involves considering hardware and output modal-
ities as alternatives and evaluating them in terms of reliability, responsiveness
and agility using pair-wise comparison followed by analysis and ranking of the
AR alternatives. Elia et al. categorize AR hardware as head-mounted displays
(HMD), handheld devices, projectors and haptic force feedback systems[3]

Kruijff et al. classified potential perceptual issues with AR between environ-
ment, capturing the environment, augmentation, display device and user while
pointing out whether they are predominant with particular display types: Head-
mounted displays (video see-through or optical see-through), handheld mobile
devices or projector-camera system (stationary or mobile)[6]. For this experiment
each display type is evaluated.

Schwerdtfeger et al. go into depth describing the projected AR, specifically
using lasers, pointing out the cons of HMDs; small field of view, limited res-
olution, swimming effect, multiple focus planes and eye fatigue. While laser-
projected AR can address some of these issues it is limited to displaying in-
formation on surfaces in the environment and the image must be pre-distorted
to compensate for geometric surface distortion, whether the projector is head-
mounted or stationary. In addition, it also introduces the challenge of occlusion
by either the user or other objects[8]. In order to avoid surface distortion for this
test an even and level surface is used.

Studies have shown that depth perception while using AR is affected by a
bias; subjects tended to underestimate distance in AR when they are projected
at less than 23 meters distance to the user, after which the bias switches to
overestimation[10]. Singh et al. estimated a distance judgment error of -5.5 cm
at most for distances less than 50 cm[9]. However, this was tested in 2010 using
an nVisor ST display, so modern see-through displays may lead to different per-
formances. Augmented reality has previously been used to imitate x-ray vision:
Avery et al. emphasized that when showing the content beyond the surface using
an graphical overlay it should include an edge overlay representing the surface
as a depth cue, so the object does not appear to float in front of the surface[1].

3 Setup

Spotting sub-surface impurities is tested using a tray of sand where the impuri-
ties are virtually projected beneath the surface using each of the AR displays.
Between test sessions the surface of the sand is smoothed out to hide the previous
entry points.

The main component for measuring the user's performance is a six degree
of freedom tracking space. For this an HTC Vive setup is utilized: One of the
controllers is used to point to the virtual impurities with a needle mounted to it.
The other controller is used to confirm when they are touching the impurity.
This is to prevent shaking the dominant hand when confirming a target. The
setup for the test is shown in full in Figure 1.

In order to display the targets on the mobile AR devices, the head-mounted
display and the see-through display, the PC that runs the software and tracking
also acts as network host while the mobile units act as clients and receive target
positions as the tests are running.

The position and orientation of the sand's surface are calibrated using the
needle mounted to the Vive controller along with a tracking guide printed on
paper. This is also used to align the positions of the printed tracking markers
used for the mounted see-through display and the HMD.

## 3.1 Mounted See-Through Display

The concept of the arm-mounted display is that it should be held by a robot
arm, allowing it to orient the screen towards the impurity, thereby guiding the
attention of the user. However, for this test will be mounted to a manually
adjustable mount. The see-through display is tracking the tray and the targets
<figure><img src="image_2.png" /></figure>

*4/4*4/4

<caption>Fig. 1. The full setup for the AR test.</caption>

using the Vuforia AR framework running in the Unity game engine compiled to Android.

The tracking markers are printed on paper and are placed at either end of the tray so that they are not obscured by the participants' hands during the test. When positioning the screen for the participant, its camera is pointed toward the tracking opposite to the participant's dominant hand, making it unlikely for them to reach over it.

The display is an Android tablet showing the back-facing camera feed where the impurities are shown as a red sphere of 10 mm diameter projected into the sand. In addition, this view will feature an overlay aligned with the surface of the sand to assist in depth perception similarly to [1]. This setup is illustrated in Figure 2.

## 3.2 Head-Mounted Display

For testing see-through head-mounted displays an Epson Moverio Bt-300 is used. These are shown in Figure 3. Similarly, the AR glasses are running using and Android-based mobile unit acting as a client, receiving target positions from the host PC running the tracking space while tracking its own position using Vuforia and Unity.
<figure><img src="image_3.png" /></figure>

*4/4*4/4

<caption>**Fig. 2.** The arm-mounted see-through display setup with impurity projected into the sand as a red dot assisted by a grid overlay aligned with the surface.</caption>

The position of the impurity are shown on the glasses in a similar fashion to arm-mounted display while also utilizing the overlay grid. However, due to the limitation of not being able to track the gaze of each eye of the participants and thereby determining the convergence point, attempting to render the position in stereo would leave the user with difficulty focusing. Therefore, the target is rendered as a 2D overlay similarly to the see-through display, and when analyzing the data the participants' dominant eye must be taken into consideration.

## 3.3 Top-Down Projection

The top-down display is implemented using a projector mounted to a tripod using a custom 3D-printed mount. It projects the position of the impurity in the surface of the sand using a 10 mm red. Because the projection does not allow for perspective rendering like the see-through display and the HMD, the red dot is rendered along with a number indicating the depth of the impurity.

The camera used is a Logitech C920. During the calibration of the setup, the position and orientation of the camera is calibrated using the needle on the Vive controller. The mounted projector is shown in Figure 4 and the top-down projection is shown in Figure 5.
<figure><img src="image_4.png" /></figure>

*4/4*4/4

<caption>**Fig. 3.** The Epson Moverio Bt-300 glasses used for the experiment.</caption>

## 3.4 Monitor

The static display involves a monitor positioned at the opposite side of the tray to the user showing a top-down camera feed of the surface of the tray while showing the position of the impurity, similarly to the top-down projection. Similarly to the mounted see-through display the user will be looking at their hands and the impurities through the monitor. This setup us shown in Figures 6.

# 4 Procedure

At the beginning of each test the participant is introduced to the subject of the experiment, the setup and the tasks they will be performing. User information gathered before the test starts will include sex, age, height, dominant hand and dominant eye.

## 4.1 Pointing Tasks

The experiment is run with 48 participants. This is to counterbalance the order in which the participant tries the display in order to counter bias or possible con- fusion at the start of the experiment, either pertaining to the pointing task or the questionnaires between test sessions. Four systems yield 24 possible combina- tions and going through them twice will yield an acceptable number of samples for analysis.

For each AR system the participant must point to 24 impurities projected into the sand by the AR display. They will do this by pointing into the sand
<figure><img src="image_5.png" /><figcaption>Fig. 4. The projector mounted for the top-down projection setup.</figcaption></figure>
<figure><img src="image_6.png" /></figure>

*4/4*4/4

<figure><img src="image_7.png" /><figcaption>Fig. 5. The grid projected onto the sand with the targets shown as a red dot.</figcaption></figure>

*4/4*4/4

<caption>Fig. 6. The monitor and top-down camera.</caption>
using a needle-like object mounted to an HTC Vive controller in their dominant hand, shown in Figure 7. The participants are instructed to point as quickly and precisely as they can with the goal of getting the tip of the needle as close to the center of the impurity as possible. To confirm that they are pointing at the impurity the participants must press a button on the controller in the non-dominant hand while the needle is in the target before moving on to the next target.

In order to prevent participants accidentally confirming two targets in suc-
cession, new targets appear with a one-second delay. This also allows the partic-
ipants to retract their hand to a neutral position before searching for the next
target.

<figure><img src="image_8.png" /></figure>

*4/4*4/4

<caption>**Fig. 7.** The needle used for the experiment, made from a 3D printed mount and a nail and attached with a bolt in the loop designed for the wrist strap.</caption>

The body of sand is 25 cm wide, 55 cm long and 5 cm thick. Impurities in the meat most often occur on the surface, but can occur as deeply as 3 mm under the surface. The impurities will be projected at 24 predetermined position distributed between three layers, two row and four columns in order to test the effectiveness of the systems at various depths and in relation to dominant hand. The layer will be at depth 0 mm, 5 mm and 10 mm, and the layers and columns will be spaced by 10 cm on a square grid.
## 4.2 Measurements

The data gathered for each target in the experiment will be the distance between the needle tip and the center of the target as well as the time between target spawn and user confirmation. The distance measurement will include the distances on the X, Y and Z-axes as well as the direct distance.

In addition to the data gathered from the software, between testing the displays the participants are asked to fill in two questionnaire regarding their user experience: A System Usability Scale[2] and a NASA Task Load Index[4] are used to evaluate each of the AR displays in terms of user acceptance and ease of use.

## 5 Conclusion

This chapter presents the hardware setup and procedure for a preliminary comparative study of the usability of a collection of AR displays for showing subsurface impurities in meat by having the participants point to targets inside an analog made of sand. The goal is to determine suitable interfaces for augmented meat production while being able to implicitly correct the worker's posture while directing their attention towards the target. To this end, we expect that the collaborative robot arm will show the greatest impact, as it allows for dynamic display adjustments in relation to the workers posture.
References

1. Avery, B., Sandor, C., Thomas, B.H.: Improving Spatial Perception for Augmented Reality X-Ray Vision. In: 2009 IEEE Virtual Reality Conference. pp. 79–82. IEEE (mar 2009). https://doi.org/10.1109/VR.2009.4811002, http://ieeexplore.ieee.org/document/4811002/

2. Brooke, J., et al.: Sus-a quick and dirty usability scale. Usability evaluation in industry **189**(194), 4–7 (1996)

3. Elia, V., Gnoni, M.G., Lanzilotto, A.: Evaluating the application of augmented reality devices in manufacturing from a process point of view: An AHP based model. Expert Systems with Applications **63**, 187–197 (nov 2016). https://doi.org/10.1016/j.eswa.2016.07.006, http://linkinghub.elsevier.com/retrieve/pii/S0957417416303505

4. Hart, S.G., Staveland, L.E.: Development of nasa-tlx (task load index): Results of empirical and theoretical research. In: Advances in psychology, vol. 52, pp. 139–183. Elsevier (1988)

5. Hayes, B., Kruijff-Korbayova, I., Sierhuis, M., Shah, J.A., Scassellati, B., Gombolay, M.C., Jung, M.F., Hindriks, K., de Greeff, J., Jonker, C., Neerincx, M., Bradshaw, J.M., Johnson, M.: HRI Workshop on Human-Robot Teaming. In: Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts - HRI’15 Extended Abstracts. pp. 255–256. ACM Press, New York, New York, USA (2015). https://doi.org/10.1145/2701973.2714396, http://dl.acm.org/citation.cfm?doid=2701973.2714396

6. Kruijff, E., Swan, J.E., Feiner, S.: Perceptual issues in augmented reality revisited. In: 2010 IEEE International Symposium on Mixed and Augmented Reality. pp. 3–12. IEEE (oct 2010). https://doi.org/10.1109/ISMAR.2010.5643530, http://ieeexplore.ieee.org/document/5643530/

7. Punnett, L., Wegman, D.H.: Work-related musculoskeletal disorders: the epidemiologic evidence and the debate. Journal of Electromyography and Kinesiology **14**(1), 13–23 (feb 2004). https://doi.org/10.1016/J.JELEKIN.2003.09.015, https://www.sciencedirect.com/science/article/pii/S1050641103001251?via%3Dihub

8. Schwerdtfeger, B., Pustka, D., Hofhauser, A., Klinker, G.: Using laser projectors for augmented reality. In: Proceedings of the 2008 ACM symposium on Virtual reality software and technology - VRST ’08. p. 134. ACM Press, New York, New York, USA (2008). https://doi.org/10.1145/1450579.1450608, http://portal.acm.org/citation.cfm?doid=1450579.1450608

9. Singh, G., Swan, J.E., Jones, J.A., Ellis, S.R.: Depth judgment measures and occluding surfaces in near-field augmented reality. In: Proceedings of the 7th Symposium on Applied Perception in Graphics and Visualization - APGV ’10. p. 149. ACM Press, New York, New York, USA (2010). https://doi.org/10.1145/1836248.1836277, http://portal.acm.org/citation.cfm?doid=1836248.1836277

10. Swan, J., Livingston, M., Smallman, H., Brown, D., Baillot, Y., Gabbard, J., Hix, D.: A Perceptual Matching Technique for Depth Judgments in Optical, See-Through Augmented Reality. In: IEEE Virtual Reality Conference (VR 2006). pp. 19–26. IEEE. https://doi.org/10.1109/VR.2006.13, http://ieeexplore.ieee.org/document/1667622/