# Using the Smartphone as an Augmented Reality Device
in ETO Industry

Niklas Jahn, Axel Friedewald, Hermann Lödding

► To cite this version:

Niklas Jahn, Axel Friedewald, Hermann Lödding. Using the Smartphone as an Augmented Reality Device in ETO Industry. IFIP International Conference on Advances in Production Management Systems (APMS), Aug 2020, Novi Sad, Serbia. pp.538-546, 10.1007/978-3-030-57997-5_62. hal-03635624

HAL Id: hal-03635624

https://inria.hal.science/hal-03635624

Submitted on 20 Jun 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Using the Smartphone as an Augmented Reality Device in ETO Industry

Niklas Jahn(✉)[0000-0001-6557-4003], Axel Friedewald and Hermann Lödding

Hamburg University of Technology, Hamburg, Germany
{niklas.jahn, friedewald, loedding}@tuhh.de

**Abstract.** Industrial Augmented Reality (IAR) has proven its potential in ETO industry by improving productivity and acting as a key technology for consistent digital information flows. In order to leverage these potentials on a significant scale, broad distribution of IAR inside a company should be aimed for. However, most applications have not yet overcome a prototype state and failed to make the step into production. Primary limiting factors are inadequate device availability and acceptance of current IAR solutions, which focus on smart glasses and tablet devices. The aim of this paper is to examine whether smartphones have the potential to serve as a platform for IAR in ETO industry. First the user interface of an existing tablet IAR prototype is adapted to a smartphone in order to later evaluate user experience. Hence a survey to assess worker preferences between tablet and smartphone devices for common IAR supported tasks is developed. The survey design is tested on a group of workers from ETO industry.

**Keywords:** Industrial Augmented Reality (IAR) • Mobile Devices • Engineer-to-order (ETO) • User Experience

## 1 Introduction and Motivation

Digitalization initiatives in ETO industry transform the future way workers access information by focusing on consistent digital information flows between product design and shop floor and digitalized operator support [1]. Industrial Augmented Reality (IAR) is a special form of digitalized operator support, which visualizes information as virtual content superimposed with reality. It is broadly researched in manufacturing [2] and in particular in the ETO industry for maritime products [3, 4]. In this field mobile devices are the most common type of hardware due to native capabilities that foster the implementation of IAR, such as sensors, display technologies and software frameworks [4]. IAR can reduce the necessary amount of 2D drawings with their high creation costs by allowing for simple access to the underlying 3D CAD data and derived information, e.g. measurements [5].

For leveraging the potentials, it is implied that the predominant device and software need to be widely accepted in a company in terms of *user friendliness*, *reliability* and *scalability* [6]. These fundamental requirements are still not sufficiently fulfilled by head-mounted devices or wearable technology [7].
In contrast, smartphones and tablets are routinely used in private and business areas. Smartphones, in particular, are carried around by almost everyone due to their compactness, which makes them a highly available information carrier. While broad research on tablet-based IAR in ETO industry exists, smartphones still lack examination.

This paper identifies the challenges for smartphone-based IAR and introduces a prototypical smartphone adaption of an existing UI for a tablet-based IAR information system for workers in ETO industry, like e.g. on a shipyard. Subsequently a study design for determining user preferences in comparison with the tablet solution is proposed. The results of a preliminary test of the study design are finally presented.

# 2 Industrial Augmented Reality in ETO Industry

The basic principle of Augmented Reality is to overlay real environments with virtual content while maintaining a constant positional reference between those two and allowing the user to interact with the virtual content in real-time [8]. Industrial Augmented Reality is a subset of AR and aims at industrial fields of application and their distinct requirements. E.g. most environments in ETO industry do not offer consistent connectivity. Therefore, IAR applications must be designed to focus on daily updates of the most relevant data instead of live streaming it. The main reason for use of IAR, especially in ETO industry, is its positive impact on productivity and product quality.

## 2.1 Productivity Impact

In previous research, Halata, Friedewald et al. [5, 9] extensively studied the use of AR in one-of-a-kind production for products, which are produced in a prepared environment, like cruise ships in a yard and assessed the individual potentials of different IAR devices. They concluded that handheld devices bear the biggest potential to improve key activities found in productivity analysis for assembly processes in ETO industry, which requires more flexible work planning than in BTO or MTO industries. Although focusing on tablets for prototyping and evaluation, they anticipate similar potentials for smartphones. Studies with workers from ETO industry in a test environment revealed a significant labor productivity increase of up to 19% with a tablet-based prototype [9].

The productivity increase results from the impact of an IAR system on those activities which are summarized as information gathering. An example is gathering information about the final state of a work package in order to plan the task execution ad hoc and to check for engineering changes, which occur more often than in other industries due to the ETO approach. A target-actual comparison after a work package is finished, reveals deviations from target state. AR is not only a more intuitive way of completeness check in comparison to the use of drawings, evaluations have also shown an increased error detection rate and improved product quality, which reduces rework [9].
## 2.2 Standard User Interface for Tablet-based IAR

In the research project SUPER [10] an IAR prototype based on the fundamental concept of Halata et al. [5, 9] has been developed, which offers additional features like displaying a product structure and advanced filtering. Its UI is shown in Figure 1 and consists of three main areas. There are two collapsible 2D *content areas* on each side of the screen (1), which can hold different groups of information and associated functionalities, called modules. A module is for example the searchable product structure on the left side of the screen, which a worker uses to find and inspect relevant parts for a work package. A 3D *content area* is using nearly the entire screen space when 2D content areas are collapsed (2). It either displays an entirely virtual environment (CAD mode) or an augmented reality visualization (AR mode). Two constantly visible *control bars* at the top and bottom of the screen (3) contain buttons to control these visualization modes as well as the activation state of modules in the 2D content areas. Further buttons allow for user interactions affecting the 3D content area in terms of navigation, e.g. automatic movement of the camera towards a selected part in CAD mode.

<figure><img src="image_2.png" /><figcaption>Fig. 1. User interface of a tablet-based IAR prototype</figcaption></figure>

## 3 Adapting the User Interface to the Smartphone Screen

Since the hardware performance is comparable between smartphones and tablets, the smaller screen size of smartphones in particular makes the difference. In the following the focus therefore is on solving the resulting challenge of maintaining user experience despite the smaller screen size, when IAR is brought to smartphones.
In order to develop an adapted UI design and to implement it in a prototypical appli-
cation, requirements are formulated, that aim for good user experience.

**3.1 Fundamental Requirements**

A satisfying user experience is tightly coupled with user friendliness and reliability of an application. In order to incorporate these aspects in a UI design, requirements have been compiled, which are of special importance for a smartphone UI:

* **Device Orientation.** Unlike tablets, which are primarily used in landscape orientation, for smartphones portrait orientation is the more common one [11]. This is especially due to its allowance for one hand operation. Also there is higher relevance of device orientation due to the larger aspect ratio of the screen, resulting in a bigger difference in field of view when viewing a 3D scene. A smartphone UI therefore has to adapt to frequent changes in device orientation, since users tend to change it depending on the displayed content and situations requiring one-hand operation.

* **Restricted Screen Zones.** Smartphones include a variety of sensors, like cameras and brightness sensors, which are necessarily placed at the front of the device facing the user, e.g. to perform video calls. Many devices have display notches that integrate the sensors but make parts of the screen resolution invisible. On top of that the native operation systems of smartphones reserve other restricted areas to place controls such as a virtual home menu buttons. A smartphone UI therefore has to take into account both of these restricted screen zones.

* **Information Access.** The UI should allow easy and fast access to any information and functionality and therefore require the least possible amount of interactions. This can be challenging. A multi-level dropdown menu is an example for a common UI element, which requires little screen space, but tends to obstruct the information it is linked to, because it requires multiple inputs for information at a deeper menu level.

* **Predictability.** The UI should give the user the feeling of being in control by having a predictable, i.e. reproducible behavior. Automatic UI reactions should be kept at a minimum and should be simple enough for the user to understand. A menu that automatically sorts itself based on the elements a user often uses, might appear helpful in the first place, but it is not very predictable since the user will find himself in situations where buttons changed positions, resulting in a feeling of uncertainty.

* **Intuitive Interactions and Layout.** Smartphone operating systems have basic design guidelines for their target device which foster intuitive use, e.g. size of interaction items like buttons should be suitable to be reliably clicked without misses. Another aspect of these guidelines requires, that the UI layout should be organized in a fixed grid, so the user can easily internalize it.

* **User Focus.** The limited screen size of smartphones can lead to unavoidable conflicts about which areas of the UI should be in focus and therefore use more screen space. The UI should support the user in establishing focus by allowing him to maximize certain areas of interest and minimize others. Where possible the UI can taking control over this automatically under consideration of aforementioned *predictability*.
## 3.2 Design of the User Interface

These fundamental requirements have been taken into account to create a prototype optimized for smartphones. The underlying design activities and decisions can be consolidated into three main activities.

**Streamlining the UI.** Many common smartphones have space-limiting notches at the top of the screen. However, the two 3D mode control buttons fit in this area (1) as shown in Figure 2 a), where they are quickly accessible, but hardly pressed accidentally. The majority of buttons for controlling module visibility can be combined in a single menu strip large enough to view a reasonable number of buttons in parallel (2). It allows to access further menu items by horizontal scrolling.

Additionally both 2D content areas of the tablet UI can be combined into a single one to substantially reduce the need for screen space (3). Since most modules are designed to be used independently, the 2D content area can be restricted to only contain one module at a time. Lastly the bottom control bar is resized to fit larger buttons, which demand for less typing precision (4).

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** UI of the smartphone-based IAR prototype</figcaption></figure>

**Adding User Control over the Layout.** Reduction of the UI to two content areas already increases clarity, but can be further improved by allowing the user to decide which area to primarily work with. It leads to a reduction of the split-attention effect [12], that the user experiences when using the 2D and 3D area simultaneously as shown in Figure 2 a). Therefore the UI is extended with controls to expand and minimize the 2D area, hiding or expanding the 3D visualization area as a result (see Fig. 2 b).
If the smartphone's orientation changes from vertical to horizontal, the user interface automatically reacts by minimizing the 2D area and maximizing the 3D area to give the user a greater field of view for AR and CAD mode.

**Improving Interaction with Lists.** Lists will automatically react to changing selection in the 3D area by expanding items and scrolling to the relevant list entry (auto-scroll) as shown in Figure 3. When using the search bar on top of a list (1), the 2D area is automatically expanded to full screen (1*). Selecting a search result (2) also leads to the aforementioned auto-scroll behavior (2*).

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Optimized user interactions with lists (1, 2) by automatic UI reactions (1*, 2*)</figcaption></figure>

The UI design has been implemented in a prototypical IAR application. In the following, the goal is to examine the resulting user experience in comparison with the tablet-based prototype, which has been introduced in Sect. 2.2.

# 4 Comparative Study on User Experience

## 4.1 Methodology

To cover both task-related and task-independent aspects concerning the user experience, the proposed evaluation method combines three questionnaires. Participants perform different IAR related tasks with each device and then answer the questionnaires.
* The NASA Task Load Index (TLX) quantifies the overall workload of a person for performing a task based on six individual factors measured on a 20-step scale [13]. It serves as a standardized questionnaire to evaluate whether the device type effects the effort to achieve certain tasks performed with a designated software prototype.

* The User Experience Questionnaire (UEQ) captures the overall impression of a user interacting with a software [14]. This is represented by six different scales, which are assigned to two categories of either pragmatic, i.e. task-related, or hedonic, i.e. task-independent, quality aspects. The focus on user satisfaction, effectivity and efficiency is a difference to questionnaires that are orienting on norms like ISO 9241/110-S, which mainly emphasize the latter two aspects.

* An additional self-developed *custom questionnaire* is used to retrieve information about the user's device usage in private and professional context, his or her general device preferences and previous experience with IAR.

## 4.2 Results of a Preliminary Test

The evaluation method was tested on a group of three workmen in ETO industry in order to find out whether it is suitable and can be later applied on a larger group.

Results of the pre-test are of course not representative but deliver a helpful first indication about user experience of the smartphone prototype in direct comparison with the tablet prototype and needs for adjustment of the evaluation method. Based on the pre-test, the following relevant differences between the devices that affect the evaluation design can been found. Participants which prefer the smartphone-based IAR prototype often reason their decision on the compactness and form factor of the device or existing practice and habit. The evaluation design should take into account this disturbance factor by using a blocking concept, which separates experienced users from unexperienced ones for each device type.

In this pre-test participants were more satisfied with their task results on smartphones than on tablets. This might be explained with a higher amount of practice with smartphones and emphasizes the aforementioned measure. It also depends on the order in which the devices are used to perform the tasks, as this implies a disruptive factor due to a training effect on the task execution when the other type of device is used to repeat the task. In order to reduce this interference, the starting device type must also be taken into account when blocking.

## 5 Outlook

The presented IAR prototype presented in this paper is the foundation for process-related research on the use of smartphones as an IAR device for ETO industry. The pre-tested study allows to compare devices in terms of the user experience and is intended to be used on a larger scale in future research. The measurement of user experience aims at determining the device that mostly satisfies workers' needs.

However, an additional productivity analysis of both devices in a laboratory environment would enable to also take into account the economic impact to a company.
**Acknowledgments.** The authors would like to express their gratitude to the Research Council of Norway (MAROFF-2) for funding this research in the project "SmartYard" (No. 282365). We are also grateful for the dedicated contribution of the participants who took part in the pre-test.

**References**

1. Strandhagen, J.W., Buer, S.-V., Semini, M., Alfnes, E.: Digitalized manufacturing logistics in engineer-to-order operations. In: Ameri, F., Stecke, K.E., Cieminski, G. von, Kiritsis, D. (eds.) APMS 2019, IFIPAICT, vol. 566, pp. 579–587, Springer, Cham (2019), https://doi.org/10.1007/978-3-030-30000-5_71.

2. Bottani, E., Vignali, G.: Augmented reality technology in the manufacturing industry: A review of the last decade. *IISE Transactions* **51**(3), 284–310 (2019), https://doi.org/10.1080/24725854.2018.1493244.

3. Helle, S., Korhonen, S., Euranto, A., Kaustinen, M., Lahdenoja, O., Lehtonen, T.: Benefits achieved by applying augmented reality technology in marine industry. In: Bertram, V. (ed.) COMPIT 2014, pp. 86–97, Verlag Schriftenreihe Schiffbau, Hamburg (2014).

4. Fraga-Lamas, P., Fernandez-Carames, T.M., Blanco-Novoa, O., Vilar-Montesinos, M.A.: A Review on industrial augmented reality systems for the industry 4.0 shipyard. *IEEE Access* **6**, 13358–13375 (2018), https://doi.org/10.1109/ACCESS.2018.2808326.

5. Halata, P.S., Friedewald, A., Lödding, H.: Augmented reality supported information gathering in one-of-a-kind production. In: Bertram, V. (ed.) COMPIT 2014, pp. 489–503, Verlag Schriftenreihe Schiffbau, Hamburg (2014).

6. Navab, N.A.: Developing killer apps for industrial augmented reality. *IEEE Computer Graphics and Applications* **24**(3), 16–20 (2004), https://doi.org/10.1109/mcg.2004.1297006.

7. Merhar, L., Berger, C., Braunreuther, S., Reinhart, G.: Digitization of manufacturing companies: Employee acceptance towards mobile and wearable devices. In: Ahram, T.Z. (ed.) AHFE 2018, AISC, vol. 795, pp. 187–197, Springer, Cham (2019), https://doi.org/10.1007/978-3-319-94619-1_18.

8. Azuma, R.T.: A survey of augmented reality. Presence: Teleoperators and Virtual Environments **6**(4), 355–385 (1997), https://doi.org/10.1162/pres.1997.6.4.355.

9. Friedewald, A., Halata, P.S., Meluzov, N., Lödding, H.: Die Produktivitätswirkung von Augmented Reality in der Unikatfertigung. In: Schlick, C. M. (ed.): Megatrend Digitalisierung - Potenziale der Arbeits- und Betriebsorganisation, pp. 141–162, GITO, Berlin (2016).

10. Wanner, M.-C., Lödding, H. et al.: Schifftbauliche Unikatproduktion mit Erweiterter Realität (SUPER). Final Report, Center of Maritime Technologies e.V. (CMT), Hamburg (2019).

11. Smartphone and tablet portrait orientation usage 2017 Q3, https://www.scientiamobile.com/smartphone-tablet-portrait-orientation-usage-2017-q3, last accessed 2020/4/2.

12. Chandler, P., Sweller, J.: The split-attention effect as a factor in the design of instruction. *British J. of Edu. Psychol.* **62**(2), 233–246 (1992), https://doi.org/10.1111/j.2044-8279.1992.tb01017.x.

13. Hart, S.G., Staveland, L.E.: Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Hancock, P.A., Meshkati, N. (eds.) Human Mental Workload, Advances in Psychology, vol. 52, pp. 139–183, Elsevier, Amsterdam (1988), https://doi.org/10.1016/S0166-4115(08)62386-9.

14. Rauschenberger, M., Schrepp, M., Perez-Cota, M., Olschner, S., Thomaschewski, J.: Efficient measurement of the user experience of interactive products. How to use the user experience questionnaire (UEQ). Example: Spanish language version. *IJIMAI* **2**(1), 39-45 (2013), https://doi.org/10.9781/ijimai.2013.215.