# Investigating Impact of Augmented Reality on Game Design to Facilitate Learning Experiences in Logistics Operations Using Immersive AR Interfaces

Sundus Fatima, Jannicke Baalsrud Hauge, Prabahan Basu, Jakob Baalsrud Hauge, Anindya Chowdhury, Artem Schurig

► To cite this version:

Sundus Fatima, Jannicke Baalsrud Hauge, Prabahan Basu, Jakob Baalsrud Hauge, Anindya Chowdhury, et al. Investigating Impact of Augmented Reality on Game Design to Facilitate Learning Experiences in Logistics Operations Using Immersive AR Interfaces. 20th International Conference on Entertainment Computing (ICEC), Nov 2021, Coimbra, Portugal. pp.419-426, 10.1007/978-3-030-89394-1_34. hal-04144372

HAL Id: hal-04144372

https://inria.hal.science/hal-04144372

Submitted on 28 Jun 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
Investigating Impact of Augmented Reality on Game
Design to Facilitate Learning Experiences in Logistics
Operations Using Immersive AR Interfaces

Sundus Fatima<sup>1,3</sup> [00000002-4696-816X], Jannicke Baalsrud Hauge<sup>1,2</sup> [0000-0002-3747-0845],
Prabahan Basu<sup>1,3</sup>, Jakob Baalsrud Hauge<sup>1</sup> [0000-0001-7410-7511], Anindya Chow-
dhury<sup>1,3</sup> and Artem Schurig<sup>1,3</sup> [0000-0001-8379-0324]

¹ BIBA – Bremer Institut Für Produktion und Logistik GmbH, Bremen, Germany

² KTH-Royal Institute of Technology, Södertälje, Sweden

³ Universität Bremen, Bremen, Germany

fat@biba.uni-bremen.de

**Abstract.** Augmented Reality (AR) provides an immersive experience platform where players may interact with the real world. Virtual digital information details enhance this interaction with the physical environment. This paper focuses on investigating using Augmented Reality (AR) technology for player's interaction influences the game design (technical features and interactivity aspects mainly) that could facilitate its players to achieve its proposed Intended Learning Objectives when playing different scenarios for logistic operations. To achieve this, an initial mock-up is set up with simple AR interactions to analyse the overall immersive player's experience.

**Keywords:** AR, Game Design, Immersive Interactive Interfaces, Logistics operations

# 1 Introduction

There is an increasing trend in using Augmented Reality (AR) in serious educational games as AR provides interactive interfaces for engaging learners and promotes active learning. Due to active involvement, AR can be an effective technology to visualise pedagogical paradigms and gain its learners' attention [1]. An advantage of AR is that it keeps the real world and, therefore, can maximise the experience by adding virtual objects providing the player with an interactive and prosperous environment with the possibility to place virtual objects anywhere without restriction to computer points to create unique immersive player experiences [2]. Within warehouse operations, some applications are using Virtual Reality (VR) and/or AR in an assistive way [3-6], but so far, the uptake both in training as well as in operative processes are limited [7-9]. Comparing the advantages and disadvantages of VR and AR and our experience with using such technologies [7,10] in games, we will in this case study investigate AR in more detail, since AR allows the combination of both virtual and real, letting players sense
the real world. In contrast, VR provides a computer-generated immersive virtual envi-
ronment to be experienced over either a computer screen or stereoscopic displays. VR
enhances only a fictional reality, but AR enhances both the virtual and real world.
Player experiences in interactive, immersive and learning environments have become
relevant in past years [11]. To create an immersive environment to improve the reali-
sation of learning content, it is essential to make an appropriate selection of tools and
technological devices that could support the creation of such an immersive educational
environment where learners have first-person experiences [11]. To experience AR tech-
nology, proper devices and hardware are to be used.

The research problem encountered is how to select appropriate immersive interac-
tive and cost-effective technical devices to realise the desired immersive environment.
The selection of appropriate devices is necessarily crucial as to create such an interac-
tive environment using AR technology. As mentioned earlier, this paper's focus is to
see the impact of using AR technology on game design, mainly addressing technical
features and interactivity. In this regard, it is to investigate if using tablet devices could
provide users with the required immersive interactive environment for realising AR that
could impact game design influencing overall player immersive experiences? To an-
swer this question, an initial mock-up is set up with a tablet to realise basic AR inter-
actions to observe player immersive experiences in terms of technology and interactiv-
ity aspects.

## 2 Game Design

The motivation behind the game is to give the players an experience as close as possible to the real-life one in a forklift driven scenario.

At the beginning of the game, a short introduction will be given to describe the game briefly. Once the game starts, the player will now be directed to the tutorial part, where they will learn about the basic controls and movement of the forklift. Considering this might be the first time the player plays an AR-based game, the tutorial is critical. In the tutorial, the player will learn how to control and move the forklift around in the given arena and get accustomed to the AR equipment. Once they are well accustomed to the available tools, they will be asked to perform basic tasks such as moving from one point to the other, picking up stuff, moving stuff from one point to the other within the stipulated area. This will ensure that the player is now comfortable with the movement and other actions available to them. The player requires to complete this part to be able to play the game.

The goal is to introduce sensor assistance and let the player explore how this assistance
can enhance the performance or the quality of work in a warehouse scenario. With the
improvement of technology, sensors have become an everyday tool in our lives. The
target is to be able to integrate them into a forklift, benefit the logistics of a warehouse,
and introduce the players to the implementations of these benefits. To maximise player
possibilities and experiences that might help players while playing games using specific
sensors to perform particular tasks in the warehouse environment. The sensors used for
this purpose are 1. Ultrasonic sensors, 2. Vibration sensors, 3. Touch sensors.
The player will perform trivial actions of driving forward, backwards and steering the forklift. Different sensors have different purposes and are installed in the forklift to assist the performance. The player will be driving the forklift and performing tasks with sensor assistance and will experience the benefits of using sensors. Each assignment will comprise a sensor-specific action. For each of the sensors available in the game, the player will get to perform tasks specific to each of them to maximise their understanding of the working of the sensors, as mentioned above. The player will be able to use each sensor in real-time and understand the advantages of having a sensor assisted environment. The AR technology should be able to provide the players with a close to real-life experience.

## 3 Initial AR Prototype

The prototype needs proper testing. This will be held in early autumn 2021 as the prototype is still in the development phase. Here we will discuss the considerations, observations, initial testings, and its results during its development, which could provide a basic idea to understand how we intend to measure immersion in particular application when prototype would be implemented fully and tested then.

Since the goal is to measure the impact in player immersion, a sense of immersion in computer games is described for virtual environments, where the focus is to make the virtual world as authentic as possible. AR already gives a natural environment. The design focus is to make real interaction between virtual objects and the natural environment. By creating real interaction, we could measure the immersiveness based on the interaction between virtual and real environment which player would experience. Since development is in the initial phase with trivial UI and interactions not fully implemented, proper testing would be later in this mock-up. It was observed only to get an initial impression if the player could visualise and have the first impression of UI on the tablet screen, could control forklift using basic forward controls implemented, for the time being, initial interactivity impression and limitations in terms of usability been observed. However, it was pretty hard to measure or monitor immersiveness at this stage as proposed basic interactions are not entirely implemented.

The aim is to see immersive player experience using AR with a tablet which could increase engagement by using immersive learning techniques which could give players exciting new ways to interact with their environment. Since many years AR applications are developed, and the possibilities of AR applications are incredible, what we aim to achieve here with AR is more appropriate in terms of logistics operations as the knowledge could be offered in the space where learning or training needs occur using the cost-effective device (Tablet) to experience AR where the actual environment is augmented with information which can help the user learn or train for the specific skills within a safe environment avoiding the risks that real-world experiment may entail. The use of a cost-effective device to experience AR technology and the low probability of associated risks still make this approach appropriate and a good idea to be used for logistic operations.
A detailed introduction is provided in the pretest section to the players as the player will learn about the detail of the AR components of the game even if the player is experiencing the AR technology for the very first time. In the introduction, players will have some basic tasks that will teach them the basic controls of the forklift. During tutorial, the player will learn more about warehouse safety standards. This section should introduce the players to use the AR controllers to manage the forklift in the gaming scenario, which includes moving the forklift. The controllers will be on a tablet, which has the AR, and the player will be able to control the forklift through it. To develop that in the physical world, we use a model forklift and add sensors and motors. The forklift is operated, and the data collected by the sensors is transferred back. Players will be using different sensors which have various purposes and are installed in the forklift to assist the performance. While selecting a sensor, the player will receive basic information about the sensor, like how the sensor works in natural environments and how the sensor can be of assistance.

# 4 Prototype

This chapter describes the current considerations related to the architecture of the MR-environment and which AR technology will be most suitable to achieve the intended interaction between player and gameplay.

## 4.1 Architecture

The technologies were implemented according to the current state of the art, focusing on simplicity, modularity, and expandability. Based on these requirements, a Raspberry Pi 4 Model B is used as a server and input device simultaneously. The UI developed with Unity Engine is running on an Android tablet. Its camera is used for environment recognition. The software accesses the real-time sensor data wirelessly via the REST interface. This data is presented to the player visually on display as an overlay using augmented technology.

### Technical Architecture

<figure><img src="image_3.png" /><figcaption>Fig. 1. Overview of technical architecture</figcaption></figure>
## 4.2 Unity 3D Game Engine

Unity 3D game engine is used to develop interactive interaction as Unity is a multi-platform development tool used to generate 2D and 3D gaming and highly immersive experiences. It is used to make User Interface (UI) following the game art of simplicity for initial interactive mock-up. It would be a simple screen on tablet with basic controls where players could navigate the forklift.

## 4.3 Vuforia SDK

Vuforia is an Augmented Reality (AR) software development tool-kit is used to create dynamic user experiences. Vuforia utilises computer vision technology to see 3D objects in a real environment and is used to place virtual objects using camera viewfinders and positions objects in real surroundings. Using AR Camera GameObject from Vuforia is mainly used to realise augmented reality.

<figure><img src="image_4.png" /><figcaption>Fig. 2. Initial Mock-up UI (Joysticks Controllers and Exit Button)</figcaption></figure>

# 5 AR Interactions and Players' Experience

This section discusses how AR interactions can improve the immersive user experience. The prototype still needs solid testing, which will take place during the summer 2021, so we just discuss the considerations. While Augmented Reality (AR) allows users to interact with things that are not physically present or may not be touched, it always has a component of "fiction". Due to this issue, Augmented Reality always requires the user to be open to such abstractions. While this usually presents no particular problem in a realistic environment, such as a computer simulation of living dinosaurs, this may present a high entry burden in more fictionalised settings, such as Pokémon Go.

As the presented game requires a realistic setting, the requirements in terms of imaging capabilities are low. Consequently, the entry barrier regarding user interaction is lacking. It is much more critical that the player can recognise the interaction through the AR device as a natural interaction similar to the one they already have in their daily
work. Since many do not use sensors for this, it is essential that they can connect the interaction with the operations and system feedback they usually receive. The technical requirements, therefore, depend on the specific setting for the application. However, all AR applications require a mobile device. In this gameplay, one must walk around and interact with different physical environments precisely as they are used to from the real world.

Furthermore, any AR application results from programming that solidifies the requirement for a device capable of computing the inputs and presenting the resulting augmentation of reality. Our project also requires the digital device to possess an integrated camera to scan QR-codes. This eliminates the usage of some tablet computers, but most of the current portable devices have them. There is no need for a high-resolution camera. Unfortunately, there is a requirement regarding the processing power and random-access memory (RAM). The application has to process the user inputs and present the results instantly to allow optimal user interaction.

The mock-up provides immediate insight into the investigation of using Augmented Reality (AR) with the tablet. Using this AR technology would contribute effectively to immersive players' experience to achieve intended learning objectives while playing learning scenarios and could visualise its impact on game design.

# 6 Impact of AR Technology on Game Design

## 6.1 Technical Features

Technical features refer to both software and hardware used to make a game [4]. AR technology can give experience and immersive interaction as required if using appropriate, cost-effective devices [5].

We use a tablet to realise our desire environment, which supports ARCore. ARCore is used as an augmented reality-building interface, which has three features mainly: tracking and movement of entity, environment recognition and light and shadow identification, which is utilised effectively for interactive player participation. ARCore makes use of tablet cameras, processors and motion sensors to offer immersive experiences. Tablet offers a portable and wide display size experience where the player can freely visualise objects and environments efficiently and be immersive.

## 6.2 Interactivity

Interactivity refers to all possible interactions that could be possible in a game, which could give a live and interactive environment to the player [6]. For our initial interactive mock-up, basic AR interactions are implemented to navigate and move forklifts using tablets. Players hold a tablet, and on screen, the player has basic controls with trivial interactions to control forklift. Tablet is portable, and controlling forklift by basic commands with interactive interactions provide players with an immersive interactive experience.

AR technology is close to the real world as it runs in a real environment [7].
## 6.3 Proposed Interactions

Since the initial prototype is in the development phase, the proposed UI contains basic interaction components with tablets: 1) Joystick Controller1 to move forklift backwards and forward. 2) Joystick Controller2 to move forklift right and left. 3) Exit Button to stop the scene shown in Fig.2. It is to observe player immersive experience when interacting with these trivial interactions.

One of the main aims of implementing these basic AR interactions using tablets is to give players an immersive experience. Specifications in tablet allow it to support all AR interactions possible. The spatial relationship between player and gameplays a significant role in providing an immersive environment.

# 7 Conclusion

This paper focused on using AR technology in game design, specifically concerning the technical and interactivity features. The analysis was conducted by setting up an interactive prototype with a tablet selected as a cost-effective device to experience an immersive player experience. Using a tablet to experience AR technology its wide screen space and portability with the ability to support required building required immersive interface made it possible to achieve or provide players with the desired AR immersive experience with basic AR interactions. However, the research and mock-up are at initial phases to observe the overall progress of its implementation at this stage.

## Acknowledgement

This work is funded by the German Federal Ministry of Education and Research (BMBF) through the project DigiLab4U (16DHB2113).

## References

1. Wolf, M., Söbke, H., & Baalsrud Hauge, J.: Designing Augmented Reality Applications as Learning Activity. Augmented Reality in Education, 23–43 (2020) https://doi.org/10.1007/978-3-030-42156-4_2

2. Söbke, H., & Baalsrud Hauge, J.: User-Centered Evaluation of the Learning Effects in the Use of a 3D Gesture Control for a Mobile Location-Based Augmented Reality Solution for Maintenance. (2020) http://ceur-ws.org/Vol-2685/paper5.pdf

3. Ginters, E. et al (2020), "Mapping of Conceptual Framework for Augmented Reality Application in Logistics," 2020 61st International Scientific Conference on Information Technology and Management Science of Riga Technical University (ITMS), 2020, pp. 1-5, doi: 10.1109/ITMS51158.2020.9259302.
4. Plakas, G et al (2016) Augmented Reality in Manufacturing and Logistics: Lessons Learnt from a Real-Life Industrial Application, Procedia Manufacturing, Volume 51, 2020, Pages 1629-1635, ISSN 2351-9789, https://doi.org/10.1016/j.promfg.2020.10.227.

5. Matsumoto, T. et al (2019), "Picking Work using AR Instructions in Warehouses," 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE), 2019, pp. 31-34, doi: 10.1109/GCCE46687.2019.9015334.

6. Baalsrud Hauge, J. et al. (2020) Employing digital twins within production logistics. ICE/ITMC 2020: 1-8

7. Baalsrud Hauge, J. et al. (2020) Digital and Physical Testbed for Production Logistics Operations. APMS (1) 2020: 625-633

8. Sorko, S.R: et al (2019) Potentials of Augmented Reality in Training, Procedia Manufacturing, Volume 31, 2019, Pages 85-90, ISSN 2351-9789, https://doi.org/10.1016/j.promfg.2019.03.014. (https://www.sciencedirect.com/science/article/pii/S2351978919303774)

9. Keepers, M. et al (2020) Gamification of Operational Tasks in Manufacturing - A Literature Review. APMS (1) 2020: 107-114

10. Söbke H. (2019), Using a Location-Based AR Game in Environmental Engineering. ICEC-JCSG 2019: 466-469

11. Brice, D., Rafferty, K., & McLoone, S.: Usability of an Input Modality for AR (L. T. De Paolis & P. Bourdot, Eds.). Springer Link, Springer International Publishing (2020) https://doi.org/10.1007/978-3-030-58465-8_19

12. Wetzel, R., Mccall, R., Braun, A., & Broll, W. : Guidelines for designing augmented reality games. Proceedings of the 2008 Conference on Future Play Re-search, Play, Share - Future Play 08 (2008) doi:10.1145/1496984.1497013

13. Liarokapis, F.: An exploration from virtual to augmented reality gaming. Simulation & Gaming, 37(4), 507-533 2006) doi:10.1177/1046878106293684

14. Huerta, A. G., Rubio, E. H., & Viveros, A. M. : Interaction Modalities for Augmented Reality in Tablets for Older Adults. Communications in Computer and Information Science HCI International 2017 – Posters Extended Abstracts, 427-434. (2017) doi:10.1007/978-3-319-58753-0_61

15. Kirner, C., Zorzal, E. R., & Kirner, T. G.: Case studies on the development of games using augmented reality, volume 2, (2006).