"Abstract:
The Augmented Reality (AR) technology has been around since 1968. It has been used in various applications serving different domains. Especially in the gaming domain, AR made a huge success. Nevertheless, these AR games rarely used Machine Learning (ML) techniques. Instead, they used simple kind of Artificial Intelligence (AI) algorithms. Hence, these games are un-realistic and predictable and so players often got bored too quickly.The use of ML enhances the players' experience dramatically and offers new and creative ways to playing games in AR. The main contribution of this paper is to propose a new approach that adopts Deep Reinforcement Learning (DRL) in AR games. In the proposed approach, a new algorithm is introduced to detect the player's behavior and update its policy. To evaluate the proposed approach, the paper introduces a method that gathers synthetic data in a custom-made environment. To study the impact of the proposed approach, we compare DRL policies of the agents before and after applying the proposed algorithm. The results reveal that the accumulative rewards of the proposed algorithm performs better than the untrained policies.
Published in: 2021 International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC)
Date of Conference: 26-27 May 2021
Date Added to IEEE Xplore: 09 June 2021
ISBN Information:
DOI: 10.1109/MIUCC52538.2021.9447671
Publisher: IEEE
Conference Location: Cairo, Egypt
SECTION I.Introduction
Augmented Reality acts like a magic window; through which the viewer can see holograms and manipulate 3D models [1]. Augmented Reality has been used in different fields to serve certain purposes. Some applications of AR were applied in professions such as medical training, retail, design & modeling, and many others [2]. AR games have been introduced in the 2000s. AR Quake game was the first game to be launched in AR [3]. After that, a lot of other AR games were created. One of the most successful games that was created is Pokemon Go. However, in AR games, the use of Machine Learning techniques were not included as using simple AI algorithms was sufficient.

In the gaming industry, there has been a significant number of successful games. But theses game requires computing resources. These Mobile devices still lack resources compared to conventional information-processing devices such as PCs and laptops. The solution to overcome these challenges is Mobile Cloud Computing (MCC) [4] –​[10].Sweetser et al. [11] talked about different games that used various AI [12] techniques, such as Age of Empires, Enemy Nations, Chess, Connect Four, Half-Life, Doom, and Quake. The techniques that were used in these games are finite state machines, scripting, agents, flocking, fuzzy logic and fuzzy state machines, decision trees, neural networks, genetic algorithms, and extensible AI. Yet, that was never good enough for the players. Eventually, the AI patterns such as movement or attacks become very predictable [13]. The AI becomes more robotic and unrealistic with time, and its diverted from its goal which is creating intelligent scenarios and tasks for the players. Thus, comes the introduction of Machine Learning.

To reach even more realistic behavior, Machine Learning techniques such as Reinforcement Learning (RL) had to be introduced. The main goal of using ML techniques has always been to reach human-like performance and interactions [14]. The ML agent is an autonomous entity that adapts and observers an environment to achieve a certain goal. That agent is given certain tasks and functions to create various situations regarding interactions with the player.

Reinforcement learning is used to solve challenges such as the big amount of data in today’s games [15], and giving the agents proper policies to make decisions in a dynamic/unknown environment. An agent is the type of AI which can learn from its past experiences. It starts to act with basic knowledge and then able to act and adapt automatically through learning. Moreover, RL advanced into another technique which is the Deep Reinforcement Learning; DRL has achieved remarkable results in many games such as Starcraft, Dota2, Chess, and Atari [16].

As shown in the figure 1 below, the idea of DRL is to create an agent that adapts to different environments by collecting synthetic data. This possible is by creating a random AI agent that acts like a human and is not 100% accurate. Firstly, the agent starts learning about that AI through trial and error and through collecting synthetic data from the environment. Secondly, the agent observes the movement of the AI. Thirdly, it performs an action depending on his policy. Then, it checks the algorithms and adapts through the gain/loss of rewards. As the gain of reward indicates that the agent has made a good action and vice versa. After the action is taken, the agent updates his policies.

To this end, the main contributions of this paper is as follows. First, it presents a new DRL approach for games in Augmented Reality. Secondly, the paper introduces an algorithm that creates a better policy for agents in various environments. Thirdly, the paper generates synthetic data from these environments to evaluate the agent’s policy.

Fig. 1. - Reinforcement Learning Diagram
Fig. 1.
Reinforcement Learning Diagram

Show All

The paper is organized into the following sections. Section II overviews the background knowledge on augmented reality, reinforcement learning and deep reinforcement learning. Section III discusses related works. Section IV presents the proposed DRL approach. Finally, section VI concludes the paper.

SECTION II.Background
In this section, we give background knowledge for augment reality and reinforcement learning. Augmented reality is not a simulation of reality - rather, it integrates 3D models in the real life which adds value to the user’s interaction. Reinforcement Learning is learning best actions based on reward or punishment. In reinforcement learning, there are three basic concepts which are state, action, and reward. In the following, we discuss these terms in detail.

A. Augmented Reality
Augmented Reality is the real-time use of virtual information such as text, audio and graphics that are integrated with real-world. It is the rendering of digital images or data onto real-world objects [17]. Some characteristics of AR are overlay of real and digital world, real-time interaction, and registration and alignment in 3d. AR can be applied to many fields such as entertainment, education, medicine, engineering and manufacturing.

B. Reinforcement Learning
Artificial Intelligence is used in making a machine think like a human being and making the machines perform human tasks [18]. Reinforcement learning is a type of AI. Reinforcement Learning is a machine learning method where agents learn to produce the optimal results through a process of trial and error [19]. The two major techniques used in ML are supervised learning and unsupervised learning. In supervised learning, the data is labeled while in unsupervised the data is not labeled. RL has three essential components which are agents, actions and rewards.

The agent chooses an action based on policies at a certain state. A policy defines the learning agent’s way of behaving at a given time [20]. The agent starts learning through trial and error which means that no data is collected in the process. The agent develops its behavior based on the amount of rewards he gets either positive or negative rewards. The more positive the results are, the better the solution and more optimized the agent becomes. The agent has his own policies that changes depending on its results of the rewards. This process is done several times until it reaches the best policies to apply. The idea of how RL works in games is shown in figure 2.

Fig. 2. - The framework diagram of the typical RL for video games
Fig. 2.
The framework diagram of the typical RL for video games

Show All

C. Deep Reinforcement Learning
Deep Reinforcement Learning is a type of RL that works through neural networks. DRL is learning from existing knowledge, and it becomes capable of using any given data, unlike RL. This data is either ready made or collected such as synthetic data. This technique removes the need of having to learn from scratch and it helps in reducing the number of trials and errors [21], which leads to better results..

SECTION III.Related Work
This section is categorized by the research efforts applied in reinforcement learning and deep reinforcement learning.

A. Reinforcement Learning
Here, we discuss papers related to Reinforcement Learning.

The use of RL has succeeded in enchaining and upgrading it’s performance and results by beating old Atari games. Mnih et al. [22] trained agents to adapt to different environments by using DRL. This was done in 6 different games and by giving them rewards to make the agents more human-like. Some of the agents even surpassed the human expectations in 3 of these games. They collected data through each game environment such as shooting, jumping, and running for each agent in all types of games.

In this paper [23], the authors goal was to create a humanlike agent that evokes social effects similar to humans under certain circumstances. By using RL, agents adapt to the game environment by giving the agent rewards when it succeeds and punishments when it fails. They tested the human-like agent created by getting 81 players between the age of 19 to 41 to check whether they can differentiate between real players and agents. They collected the data through the AI agent training and then the agent training with real people to collect more data at the end of every game (episode) which improves the agent skills. The agent was having great results that the real players could barely identify any of the agents. The agents need to have different levels of difficulty. Adding more features is always better, such as training the agent in AR games and VR to have better communication with people.

In this section, researchers achieved results and proved that RL achieves high results. It showed that RL was better than Q-Learning when it comes to teaching agents certain behaviours in any unknown environments.

B. Deep Reinforcement Learning
Now, we discuss papers related to Deep Reinforcement Learning.

This paper [24] talks about the importance of Proximal Policy Optimization (PPO) Algorithm in a multiplayer FPS game in which agents are taught how to play appropriately simulating a human player, they are taught different things such as exploring the environment, increasing their aim skill and decreasing their reaction time. In terms of results, a prototype survival-shooter game was developed in C# exploiting the Unity game engine 3D game engine.

The author [25] tried to achieve the best decisions in all game environments by fixing some components such as learning proper policies. An example of these components is learning proper policies to make decisions in a dynamic unknown environment.. With the large scale of data found in today’s games especially games like MMORPGs and Strategy games. By creating agents to train a single agent or multiagents, the DRL agents adapt and learn fast through a rewarding system and how long they can survive an episode these, In various games, DRL agents achieved human-level or super-human performances from 2D perfect information to 3D imperfect information. The data set is collected from the environment during the episode of training which gives a huge dataset and a quicker response time than a normal data set collection which ends up adding huge varieties.

Another topic that is of importance is the use of simple AI algorithms in the creation of NPCs, which leads to poor path finding and decision making. This shows that no measures were taken towards developing sophisticated AI techniques in games. It is also known that games are fully scripted and still use 30-year-old AI technology [26]. ML techniques such as perception, decision trees, and reinforcement learning have been used in multiple games. They led to more realistic behavior from the NPCs [27]. After the establishment of the game AI field. It is believed that the field is in need of revisiting and enhancement.

Fig. 3. - MMO Results [30]
Fig. 3.
MMO Results [30]

Show All

It is stated that Deep Reinforcement Learning algorithms based on experience replay have achieved unprecedented success in challenging fields. This could have never been done using any other techniques. This article [28] talks about the use of reinforcement learning algorithms and how to fully optimize it as it uses much memory and computation per real interaction. This can be solved by asynchronously executing multiple agents in parallel on multiple instances of the environment which can be done on a single machine with a standard multi-core CPU. The results were much efficient and achieved better results and didn’t consume much memory as the experience replay did.

In a similar paper [29], the author achieved their goal in making multi agent setting trained in different environments to make an MMORPG game with a good human-like agents by using deep reinforcement learning. They started by making normal agents and deep reinforcement agents. Then they made them face each other as shown in the figure !! below. The results were outstanding as almost every time the deep reinforcement agents came out winning by miles against the normal agents, which shows how good the DRL and the Deep Multi-Agent Hierarchical Hybrid Q-Networks of the agent that he created. 7.

The use of a DRL based on a new method called Stateaction-reward-state-action (SARSA) learning was proposed [31], rather than using Q-Learning which has its drawbacks. As it can not directly select policies or deal with decisionmaking problems, resulting in limited output and behavior in the intelligent control field. SARSA Learning aims to achieve even higher results than the RL in terms of imitating human behavior in the face of unknown and complicated environments. The goal of the SARSA learning task is to maximize the future reward when the agent interacts with environment. The results revealed that deep SARSA learning gained higher scores and faster convergence in breakout and seaquest than deep Q learning.

The most used algorithms in reinforcement learning are used in a way that assumes that the agent’s environment does not change. The authors [32] proposed a new framework that involves the evolution of states and actions. They also created a new algorithm Real-Time Actor-Critic (RTAC). After comparing the RTAC to the available algorithm Soft ActorCritic (SAC), it was proved that the RTAC outperform the SAC in both real time and non real time settings.

A new framework that uses gradient descent for the optimization of deep neural network controllers has been proposed to achieve high results in terms of teaching the agents [33]. The authors used four different platforms to assess the framework. They performed the experiments using Arcade Learning environment which simulates the Atari 2600 games. Results proved that the framework provides stable training of neural networks through reinforcement learning is possible with both value-based and policy-based methods.

One of the most used algorithms in ML is the Q-learning algorithm [34] and whether it has any harm on performance or not. The idea of Double Q-learning is to reduce overestimations by decomposing the max operation in the target into action selection and action evaluation. The enhancement of the Double Q-learning algorithm (DQL) is later proposed, which can be generalized to work with large-scale function approximation, which led to great results in terms of performance. [35]

An architecture that tackles 3d environment in first person shooter games was created [36], it involve partially observable states. The authors proposed a method that allows the model to detect nearby enemies or items. The model learns the features while minimizing Q-learning objectives which improves the training speed and performance of the agent [37]. The results showed that the trained agent outperforms ready made AI agents available in games and sometimes out performs humans in a death match game.

In this section, many researchers discussed using various ML algorithms and techniques to achieve high results and accuracy. The majority of them suggested using DRL to achieve human-like performance from the ML agents. They didn’t however suggest the use of DRL in augmented reality and gaming.

SECTION IV.Proposed Approach
In this section, we present the architecture of the proposed approach. We give a brief explanation of the steps which are executed in the training. Then we discuss in detail how they are implemented.

The implementation of the system is shown in the figure 4 below. The system consists of two main components. First, there is the synthetic data which contains different types of data that are gathered in run time. This data is then sent to the agent for the purpose of the training. After that in the second component, the DRL agent begins its training by executing a set of functions consecutively that outputs the agent’s behavior. This process happens in the Unity game Engine and the agents trained are created using Unity’s Machine Learning Library (UML). Now, we demonstrate the two main components of the system in detail.

Fig. 4. - Training Process
Fig. 4.
Training Process

Show All

A. Synthetic Data Component
This component represents our dataset, which is gathered at runtime as synthetic data. It consists of three sub components, which is gathered during runtime (synthetic). Then, it is sent to the agent for purpose of training.

The first and second sub components are the player data and the agent data, which are data gathered during runtime, such as positions, angles, movements, distance, player hit, health, player status, and shooting. The third component is the data collected from the environment, such as obstacles and the number of players, which helps the agent in avoiding these kind of obstacles, and making the best action according to the number of surrounding players.

B. Agent Training Process Component
While this component represents the stages and steps of training process. In the beginning, a set of ready-made functions are called. At the start of the training, the OnEpisode-Begin() function is called, which indicates that the training has begun.

Secondly, the CollectObservations() function is called, which uses Ray Perception Sensors to collect data from the surrounding environment such as the agent’s position and angle.

Thirdly, the OnActionRecvied() function is called, which contains all the actions the agent made in the form of an array of vectors.

These actions are then determined based on the RewardAlgorithm() function shown in figure 5, which is the only function that is overwritten and adjusted to output certain results. This is responsible for achieving a unique behavior for the agent. It is responsible for giving rewards to the agent through the AddReward() ready-made function, which adds a certain reward to the agent. A higher number of rewards ind10thicates that the actions that were made in the training session were proper actions.

The pseudo-code of the proposed algorithm in the RewardAlgorithm() function is presented in figure 5.

The algorithm starts off by placing the players that are close to the agent-monster-in the nearbyPlayers variable. Then, we calculate the difference between the agent’s angle and the players’ angle and we insert each result in the playerAngle variable. After that, we make comparisons between the agent’s angle and the player’s angle, the closer the difference between them reaches 0 (The agent looks at the player directly). The agent then gets rewarded through the AddReward() function, which is responsible for training the agent is called whenever the agent satisfies the condition.

The previous algorithm can be explained by a simple equation assuming that: (X is the player angle, Y is the monster angle, Z is the difference between both angles, and P is the old difference between the angle). The equation is shown in figure 6

Finally, at the end of the training process, the EndEpisode() function is called and the agent’s neural network and policy changes through the use of the PPO (Proximal Policy Optimization) algorithm and based on the rewards it gathered. After which, the process is repeated until the desired goal is achieved.

SECTION V.Experimental Results
The used dataset is synthetic data; In custom environments that are created in the Unity game engine, the dataset are generated and collected at runtime. By running the game through Unity, the data is collected from the scene when the OnEpisodeBegin() is called. The features of the generated data are positions, angles, movements, distance, player hit, health, and player status. In the reward algorithm function, this data is used to update the policy whenever the OnEndEpisode() is called, trying to make the policy as convenient as possible.

After gathering this data, we evaluate our results in terms of Cumulative Rewards, Episode Length, Policy Loss, and Value Loss. They are calculated using TensorBoard toolkit, illustrating them in four line graphs as shown in figure 7.

In these graphs, the red legend shows the results of the DRL agent before changing the RewardAlgroithm() function.

Fig. 5. - Shooting Algorithm for the Agent
Fig. 5.
Shooting Algorithm for the Agent

Show All

While the blue legend shows the results after modifying the algorithm and make it adapt to the custom-made environment.

In figure 7 (a), we talk about the Cumulative Rewards which are the number of rewards the agent which are shown over time. This is made to show the rewards amount with respect to the number of episodes done. Our goal is to gain a certain number of rewards that will develop the agent’s behavior. The results in this graph show that the agent gained a high number of rewards in his training process.

In figure 7 (b), we talk about the Episode Length which is the time taken to finish the episode with respect to the number of episodes. Our goal is to make the agent train for a long duration so that it achieves the required results. These results indicate that the ML-Agent took a lot of time before it ended its training process. The Normal AI however, shows that it failed a lot of times which resulted in ending the training process.

Fig. 6. - Shooting Equation for the Agent
Fig. 6.
Shooting Equation for the Agent

Show All

Fig. 7. - Results
Fig. 7.
Results

Show All

The first result-red legend-for our ML-Agent was calculated according to our Shooting Algorithm, the algorithm adds rewards once the agent’s hits the player, the results were very promising as a start because of the simplicity of the algorithm.

However, that wasn’t enough, we had to add more interactions to the agent. So we gave the agent the ability to rotate and move while shooting. Then, we added angles to the reward algorithm so that the agent could get as close as possible to the enemies.

The rotation algorithm and the attacking algorithm had a lot of issues. The angle which the agent was looking at, was still random. As it used to find the angle that made it acquire the highest amount of rewards. So we had to create a new algorithm-blue legend-to not only to get the rotation and shooting but also to add the agent’s angle and the player’s angle into consideration as well. We had some great results over time.

Putting into consideration that all of that was implemented through the reward algorithm which we created and talked about in detail in section IV.

SECTION VI.Conclusions
Throughout the years, researchers have looked at ways to implement both Machine Learning and Augmented Reality in games, but separately. In this paper, we integrated a DRL approach in AR. The ML Agents were trained through synthetic data gathered during run-time in custom-made environments created in Unity. Each time these agents trained, they developed their behavior and updated their policies. We kept training these agents until they reached human-like behavior and then we deployed them to an AR game. The results obtained through our research were overwhelmingly positive. The results showed that the ML Agents’ behavior became more human-like the more it was trained. The deep reinforcement learning field is very promising with a lot of work to be done in. We combined those two techniques together into one game with human-like AI and monsters to make the game fun and enjoyable. These challenges have not been fully investigated and could be opened for further study in the future."
From the above paper, explain how can deep reinforcement learning can be used in AR gaming in detail. Use paragraph form.