"Abstract
To explore the design of the virtual reality (VR) augmented reality (AR) mobile platform and game decision model based on deep learning (DL), the gesture-based interaction of VR games based on Leap Motion is researched. Based on the interactive features of gestures, a set of general gesture interaction rules in VR games is established. In the meantime, according to the theoretical basis and the characteristics of VR, a set of general models of VR gesture interaction is designed, the factors affecting the efficiency of VR gesture interaction are studied, and reasonable interaction feedback is designed. By using the computer vision and image processing technology, gesture-based interaction can collect natural gestures, extract gesture features, recognize gesture indications, and respond to the user demands. Also, it can extract the basic gestures from gesture-based interaction in VR, analyze the basic features of gestures and gesture-based interaction in VR games, and describes the gesture features by mathematical vectors and sets. The research results show that the application of gesture feature design method in the game can analyze the factors affecting the interaction efficiency. Also, the usability of the gesture-based interaction designed by the gesture design method is verified by tests. Therefore, the AR&DL platform of “AR+DL” establishes a learning platform supported by DL and AR technology. The game decision model is used to describe the process of gesture-based interaction in the game, and the factors affecting the interaction efficiency are reduced, which has certain reference and guidance for VR applications using gesture-based interaction.
Introduction
Virtual reality (VR) games emphasize the sense of immersion, while the sense of immersion comes from the being isolated from the external physical world. Notably, the visual and auditory senses deceive the brain to produce a sense of immersion.1 However, such games bring new problems since users cannot see their bodies. Especially, as the most important part of human interaction perception, the hands of users cannot interact with the virtual world.2 The sense of interaction is the key to obtain game experiences. The users operate through peripheral or wearable devices, and the game sends feedback of user operations, thereby the users will feel pleased and accomplished.3 In the VR environment, users often use the real-world interactive experience to interact with the virtual game world. For example, when a user sees a button in a VR game, the button is pressed subconsciously with the index finger. However, the current interaction mode is often achieved through selecting and clicking the buttons on the handle. In addition, the interactive mode of the graphical user interface era is still utilized. Therefore, there is an urgent need for new methods to guide the interactions in VR games.
Augmented reality (AR) superimposes virtual objects into real-world environments in real-time.4 AR technology belongs to VR (VR) technology. Traditional VR technology allows users to completely immerse themselves in virtual spaces, while AR technology focuses on the real world, using auditory and visual enhancements to enhance the perception of the real world5 Azuma defined AR as a combination of virtuality and reality, a real-time interaction, and three-dimensional registration. AR technology superimposes the virtual world into the real world through multiple methods and devices, which also allows users to interact in multiple styles.6 Most of the traditional AR technologies are based on PC terminals and head-mounted displays, and the current mobile AR technology is rapidly developing. Users can use the existing mobile device camera as a means and attach corresponding virtual conditions to obtain a true reflection of the real situation. However, currently, AR technology is mostly used in the fields of games, navigation, and monument reconstruction. The representation of AR technology needs to be diversified, and the way to combine social network platforms and AR technology perfectly becomes a new problem to be solved.
In this study, the features of gesture-based interaction in VR games are established. A set of common gesture-based interaction rules is established. At the same time, a set of VR gesture interaction general model is designed based on the prototype theory and the features of VR. Also, the factors affecting the interaction efficiency of VR gestures are studied, and reasonable interaction feedback is designed. Then, the theory of research is used to design a VR gesture-based interaction interactive game. The rapid development of AR technology provides more interaction possibilities for humans and computers. The opponent data collection can facilitate the realization of human-computer interaction (HCI) technology in many possible ways. The process of users interacting with virtual scenes with natural gestures is extremely natural and user-friendly, which promotes the rapid development of AR. The customized natural gestures are utilized to interact with virtual scenes, thereby improving the experience of HCI. The overall research has guiding reference value for the future gesture-based interaction design in VR.
Related work
Researches worldwide
Laine (2018) suggested that AR applications (APPs) have become the mainstream in various fields, such as tourism and entertainment, in which the AR games has contributed the most to its propagation.7 Owing to their tremendous screens, high-definition (HD) cameras, and marvelous computing powers, smart personal devices, such as phones and tablet computers, have make AR technology more popular among users worldwide. Specifically, mobile AR technology utilizes smart devices to project the virtual scenes, while users can interact with these scenes through the HD cameras in real-time. The research prospects of educational mobile AR game (EMARG) between 2012 and 2017 were explored through systematic literature reviews and case studies. After comprehensive literature searching and screening, 31 EMARGs were analyzed from the perspective of technology, pedagogy, and games. In addition, an analysis of the 26 AR platforms capable of producing mobile AR APPs was presented. In-depth discussions were continued, during which 13 pieces of guidance were summarized for developing EMARGs in the future. Despite the above results, a comprehensive review of EMARGs researches is needed. Tom and Jung (2018) explored the opportunity to use AR to enhance the user experience. However, in the tourism environment, there was little research on the degree of user acceptance of AR. To explore the acceptance of a technology, the technology acceptance model is often used.8 Previously, scholars have adopted methods in secondary literature to present the external dimensions. However, they have failed to integrate context-specific dimensions. Therefore, this study will propose an acceptance model of tourism AR with 5 targeting groups, including young woman visitors from the UK visiting Berlin, who also experienced the mobile AR application.
Research by SJ and Shankar (2018) showed that regular exercise seems to be effective through range exercise to improve the functional abilities of disables, thereby improving their living quality.9 But the period of recovery was prolonged largely due to the failure of guaranteeing the treating efficiency. Therefore, to aid the rehabilitation and improve the ongoing evaluations of disables, a self-motivated and participatory training program was needed. Consequently, an “AR-NUI-REHAB-MDSS” framework was proposed, in which the natural user interface (NUI)-based physiotherapy rehabilitation was utilized. This framework provided disables with personalized training plans and monitoring systems. In the meantime, it also provided therapists with the AR global remote assistance solution of mobile decision support systems (MDSS). NUI allowed for intuitive HCI through gestures. AR provided an environment with less pleasing adjunctive treatment. MDSS enabled the therapists to individualize the therapeutic plans dynamically. Upper limbs were thought to be challenging due to their functional recovery. The study by Yusoff et al. (2018) showed that many educational games have been used as interactive design tools in the learning environment to enhance student learning outcomes.10 However, the study also claimed that playing video games could harm the behaviors, perceptions, and emotions of students. The purpose of this study is to review related documents in educational games and the functions of games as interactive design tools that affect the interaction of cognitive, emotional, and social skills when students play games.
Researches in China
Wang et al. (2019) indicated that in MEC systems, users of mobile smart devices and service providers can transfer intensive computing tasks to computing resources that were near-most, thereby reducing the latency and saving the energy.11 Once users made an uninstallation decision, they needed to consider the task dependency. In off-loading problems, due to the NP difficulty, most of the current solutions were tentative; thus, it was difficult to adapt to the ever-increasing complexity and dynamicity of APPs and games. To solve this problem, an uninstallation framework with DRL basis was designed, which could effectively learn the unloading strategy distinctively symbolized by a specially-designed S2S neural network (NN). The proposed DRL framework spontaneously discovered the regular patterns behind various APPs to extrapolate the best offload strategy in different situations. Given different conditions, the performance of the proposed framework was evaluated by simulations. The results showed that this framework was superior to two heuristic benchmarks and had near-optimal performance. Studies by Tong (2019) showed that AR can seamlessly create the illusion of virtual elements mixed into the real world, which is one of the most fascinating HCI techniques.12 AR has been used in a variety of real-life situations, including immersive collaborative games, fashion appreciation, interior design, and assistive devices for the visually impaired. This study contributes to the real-time AR application of ARFurniture, which will enable users to imagine interesting furniture with different colors and styles from their smart devices. The core software architecture included deep learning (DL)-based semantic segmentation and fast color conversion. The software architecture allowed the user to prompt the system to color the style of the furniture in the scene on its mobile device and had been successfully deployed on mobile devices. The ARFurniture-related algorithms, system design, and simulation results were presented. In addition, the non-reference image quality metric “Natural Image Quality Estimator (NIQE)” was used to evaluate the immersion and naturalness of ARFuniture. The results showed that ARFurniture has the value of changing the rules of the game and enhances the user experience in interior decoration.
Qiao et al. (2019) claimed that mobile AR is getting more attention. Hardware and APPs were main platforms for mobile AR APPs.13 However, hardware platform for mobile AR was costly and inflexible to implement, while the APP-based mobile AR required extra downloads and in-advance installations, which was not convenient for cross-platform organization. In contrast, Web is a lightweight and cross-platform service provisioning platform. The Web-based AR implementations offered users a wide range of mobile AR experiences. In addition, the emerging 5 G mobile communication networks could improve the communication efficiency of mobile AR intensive computing in a Web-based approach. Wang et al. (2018) proposed the three-stage analysis of VR technology through the system analysis, the implementation technology, the applications, and the future development direction.14 The study found that the VR technology used by ceet had evolved to VR based on building information modeling (bim). These technologies had been applied to various fields, such as of architectural and design visualization. Their research prospects included visualization technologies and the combination of VR technology with emerging educational models. The findings helped researchers and educators effectively integrate VR technology into education, which greatly improved the performance of education.
A summary of foreign and domestic researches
In summary, the application research of DL would become from specific to abstract and from individual to ubiquitous. The development of the “AR+DL” platform is intended to use AR technology to enable learners to achieve the learning goals of DL, and to provide a reference for the application of DL. The use of Leap Motion in VR games has a natural advantage. The VR game environment itself reflects the real world. Users can use the real-life experience to interact naturally in the game environment, thus improving game satisfaction.
Analysis and design of the AR&DL platform
Demand analysis of the AR&DL platform
From the perspective of economic input, the development of the AR&DL platform is more economical. AR&DL platform selects C# object-oriented language, which is installed in Unity3D 5.3.4 personal version development engine; in addition, the platform selects development kits such as Android SDK, ASDDK, and ShareSDK, and these selected development tools are free versions of developers.15 For the requirements of hardware devices, Android application system devices are widely used in other application system devices, the development environment is easy to set up, and debugging devices are easily available. For the composition of project personnel, the author has certain development experience and a certain understanding of the project development process and cycle; in the project development cycle, the input of personnel and capital is not needed. Therefore, the research and development of the AR&DL platform are feasible in economic strategy.
The AR&DL platform uses C# as the development language and features of the object-oriented language. It is equipped with the Unity3D game engine to bring a lot of conveniences to project development. The Vuforia SDK is the most popular, stable, and widely used ARSDK at this stage, which recognizes images, rectangles, cylinders, and 3 D objects, providing developers with more development possibilities.16 EasyAR, as a domestic AR SDK, has been the most popular domestic AR development kit for Chinese developers due to its cloud service system, SLAM large scene recognition, and no watermark processing. Both ARSDKs can be developed on the U3D using the C# language. The development package brings together a variety of AR functions, making the cross-platform and multi-functional project development possible.17 ShareSDK and Sockets are used for the communication of the AR&DL platform. They can also be developed in U3D using C# language, eliminating the need for third-party servers and third-party communication platforms.
At the beginning of the research on the AR&DL platform, according to the general rules of applied research, the development documents required for each stage of the project are prepared, and the AR&DL development process is explained and arranged. A detailed platform test plan is selected, and test points are selected and tested. The selection of objects has been strictly required. During the development cycle, the design and development are carried out in strict accordance with the planned time nodes. Therefore, the project is feasible in terms of organization and management strategy.
As far as the lack of the DL application platform is concerned, the development of the AR&DL platform has theoretical value and practical significance for the promotion and development of DL application research. It is the basis for students to conduct DL in a certain learning environment. In addition to following the laws of system development, the research on the AR&DL platform needs to grasp the factors and features of DL.18 The previous pieces of literature are reviewed. Also, the features and characteristics of the DL platform development are summarized, which lays a decisive foundation for the AR&DL platform to be completed on time and well presented.
The AR&DL platform is the design platform for the game decision model. The platform is an auxiliary tool for the game decision model. Because the audience of the platform is clear, it determines that the function of the platform is to help people get a good game decision-making experience and provide DL for people. The learning environment and learning resources provide learners with innovative and creative learning activities and learning games and promote the learners to achieve the learning results of the DL platforms.7 The following section will introduce the functional distribution of the platform from different learner identities, as shown in Figure 1.

Figure 1. The use case diagram of the platform.
Algorithm analysis of HCI technology
The original image is assumed as s(x, y), the criterion function is assumed as f(x, y), and the threshold value is assumed as t. The segmented image can be expressed as:
 
(1)
Usually, the game decision model is set to 
 and 
, and a binarized image can be obtained.
In the chrominance space HSV analysis, a vector [H, S, P] is used to represent the three-dimensional space, and P represents the probability that the current color is originally a skin color, which can be represented by a two-dimensional function:
 = the probability of color (h, s) being a skin color in a single image
For each point on the image, the probability that it obtains skin color is:
(2)
In the above equation, 
 is the coordinate of the point, 
 and 
 is the h and s of the point.
 
(3)
(4)
In the above equation, 
 is the game decision model.
In a simple environment, the designed method can obtain more comprehensive and complete feature data. However, it is sensitive to complex and varied environments (such as light and moving objects). The simplest background model is to establish the background model by taking the average of a certain number of backgrounds and then subtract the newly acquired images in the same area to obtain the difference image. Next, the correlation filtering is performed on the difference image to obtain the segmented image.
The target image 
 achieved by the background difference can be calculated by the following equation:
  
  
 
(5)
In the above equation, 
 is the differentiated binary image, 
 is the gray value of the i-th pixel of the background acquired image, 
 is the gray value of the i-th pixel of the newly acquired image, and T is the binarization threshold. If the gray difference of the pixel is greater than T, it is determined that the pixel is a segmented image; otherwise, the pixel is considered to be a background.
Design and implementation of interactive game
The design of VR AR mobile platform and game decision model
“The Adventure of Little Bear” is a VR gesture-based interaction game for the exhibition. After wearing the virtual helmet, the user will enter a virtual cartoon world, follow the bear in the game to different scenes, and help the bear collect the ingredients through gesture-based interactions. After collecting all the ingredients, the bears cook the meal and the user completes the entire game experience process.19 The game is mainly divided into three different scenes, in which the user needs to interact with the player in four places. The specific game content is shown in Table 1.
Table 1. The game scenes and interactive tasks.
Scenes	Canton Tower	Bamboo forest	Street
Descriptions	The bear flies above the Canton Tower and the user needs to complete the invitation to accept the action.	The bear asks the user to help stand by and the user helps the bear get the bamboo shoots.	The user comes to the store with the bear.	The user comes to the refrigerator with the bear.
Voice prompts	Follow me to the challenge	Help me take bamboo shoots	Pick up the meat	Open the refrigerator
Interactive tasks	Interactive task	The user needs to complete the gesture interaction that accepts the invitation of the bear.	The user grabs the object through the hand.	The user opens the refrigerator through a gesture.
As shown in Table 1, the game is divided into three game scenes, i.e., “Canton Tower”, “Bamboo Forest”, and “Street”. In the scene of Canton Tower, the user and the bear in the game arrive at the same time from one end of the virtual game scene to the interactive area. Then, the user receives the voice prompts and completes the gesture-based interactions to accept the invitation of the bear to continue the game. In the scene of Bamboo Forest, the user and the bear arrive at the interactive area at the same time. The user receives the voice prompts and helps the bear to get the bamboo shoots needed for the task. In the scene of Street, the user helps the bear pick up the desired item by directly picking up the shop where the bear is located. Then, the user follows the bear to enter the room and helps the bear open the refrigerator door to complete the interactive game experience process.
In a shared network, the different number of frames can be chosen. To verify the selection of the appropriate quantity, experiments are conducted on the game of Breakout after the initial conditions of the various models are consistent. After the same time of training, the experimental results are shown in Figure 2:

Figure 2. Number of frames in the model.
Separate experiments are conducted on the selected game. Given the same saved game experiences, the scores obtained by the two models are shown in Figures 3 to 6:

Figure 3. The highly efficient exploration of the proposed model compared with DQN (1).

Figure 4. The highly efficient exploration of the proposed model compared with DQN (2).

Figure 5. The highly efficient exploration of the proposed model compared with DQN (3).

Figure 6. The highly efficient exploration of the proposed model compared with DQN (4).

Figure 7. Street scenes in the game.
In the above figures, the x-axis of each figure is the number of experiences stored during the training, and the y-axis is the score of the corresponding game. It can be seen from the figures that the model proposed in this study has more efficient exploration ability given the same amount of experience stored in the DQN model in the game. This is due to the shared network architecture used in the proposed model. The architecture uses a total of 10 Bootstrap frames, each of which has a target network for individual training. In addition, their data are shared, once a frame has explored a better strategy, other frames will also know. Then, they verify and continue to explore the strategy so that the exploration of the model can be more efficient.
The scenes in Unity3D are built following the principle of “what you see is what you get”. The scene elements are made and built according to the actual needs of the game. Unity3D itself provides a Scene window for scene construction and a Game window for the presentation of results.20 According to the basic needs of the game, the scene of the game mainly includes static virtual scenes, dynamic virtual elements, and movement trends of dynamic elements in response to user interaction behavior. Unity3D allows importing 3 D MAX modeled fbx files directly, adding features, textures, and lighting to characterize the features.21 For example, according to the pre-art style of the game design, the external modeling of the 3 D city is carried out, and the texture rendering after the Unity3D is imported to generate a more realistic effect, as shown in Figure 7, the scene of the street in the game.
In the scene of Unity3D, the instantiated resources are tied to the presets, and the prepared Prefab is called directly in any scene in the project, such as plants and bamboos in the game. Unity Animation System supports a variety of animation effects to ensure the realization of animation. Also, the script program can be used to achieve animation control. Unity3D allows the development and use of plug-ins, expands the functionality of the system, and improves the efficiency of system writing.22 For example, the iTween numerical interpolation plug-in can be used to design and edit the motion path of dynamic elements, such as the route that can be used to design the free walking of the bears in the game. Rendering particle simulation animations are primarily implemented through the Unity3D particle system. The Partide System in Unity3d is dynamic. The generated particles will change in position and shape over time. When new particles are automatically generated, the old particles will be destroyed and the particle flow will be formed. Then, the material is rendered again, and the effect similar to smoke and flame will appear.23 In addition, the scene building based on the Unity3D game engine is a dynamic modeling process. Some of the modeling properties can be controlled by the program code and changed according to the execution of the system, showing dynamic effects.
Usability test of gesture-based interaction
A usability test is the most basic method in usability engineering. The common method is a task test, which not only helps researchers discover the usability problems encountered by users in using the system but also shows their expectations.24 Such a method can also be used to examine the effectiveness, efficiency, and subjective satisfaction of users when performing interactive tasks in a VR environment, as shown in Figure 8.

Figure 8. The gesture-based interaction of VR.
To evaluate the effects of the gesture model in the VR gesture-based interaction system on the usability and gesture recognition, as well as to confirm the expected effects of gesture interaction model on solving the interactive experience, the gesture interaction model is simulated and tested. First, a total of 16 volunteers aged 20∼26 years old are selected to participate in the experiment (8 girls and 8 boys). Before the game test, the background and the story of the game are told, and the detailed information of the volunteers are collected. The information shows that some of them have contacted some somatosensory devices or played somatosensory games; therefore, they have a certain understanding of gesture-based interaction. The others have no understanding of gesture-based interaction. The information about participated volunteers is shown in Table 2 below.
Table 2. Information on volunteers.
Number of volunteers	Gender	With gesture-based interactions	No gesture-based interactions
8	Male	5	3
8	Female	6	2
During the experiencing process, the volunteers wear the helmet and complete the game. After the game starts, according to the received voice prompts, the volunteers perform their gestures in the effective recognition area of Leap Motion, as shown in Figures 9 and 10.

Figure 9. Test of the VR game.

Figure 10. Experience of the VR game.
Any problems encountered by the volunteers during the game can be described to the moderator in the form of language description. The game host only observes the reaction of the recorded volunteer and does not cause any interference to the volunteers. The game host mainly records whether the gestures of the volunteer meet the expected gesture designs, as well as the smoothness of the game interaction.
Conclusions
In this study, the design of the VR-AR mobile platform and game decision model based on DL is explored, and a VR gesture-based interaction game is developed based on the proposed methods. The rapid development of AR technology provides more possibilities for interaction between humans and computers. The collection of opponent data can facilitate the realization of many possible ways of HCI. The process of users interacting with virtual scenes with natural gestures is extremely natural and humanized, which promotes the rapid development of AR in turn. One of the difficulties of gesture interaction is gesture recognition. Thus, the theoretical results are applied to the gesture interaction system; the customized natural gestures are utilized to interact with virtual scenes, thereby improving the experience of HCI. By using the prototype theory, the gesture design method in the VR environment is proposed, and the method is applied in the VR interactive game. At the same time, the gesture feature is encapsulated to complete the visualization of the gesture definition. The design method can be widely used in other gesture-based interaction systems. In addition, the interactive usability of the game is improved and the final usability test of the game has verified the feasibility of the gesture design. However, deficiencies are inevitable. In this study, only the surface performance is discussed, while the research of the user mental model is not involved. The basic principles of gesture design and the analysis of the game decision model can be more in-depth. In the gesture design method, the gesture extraction statistical methods used for scoring might be not accurate enough, and the data analysis can be performed from a more in-depth perspective."
From the above paper, explain why AI and machine learning are critical to AR gaming. Also explain how AI is used in AR game.