# Interactive and Immersive Learning Using 360° Virtual Reality Contents on Mobile Platforms

Authors: Kanghyun Choi; Yeo-Jin Yoon; Oh-Young Song; Soo-Mi Choi
Journal: Mobile Information Systems
 (2018)

---

## Abstract

Recent advances in mobile virtual reality (VR) devices have paved the way for various VR applications in education. This paper presents a novel authoring framework for mobile VR contents and a play-based learning model for marine biology education. For interactive and immersive mobile VR contents, we develop a multilayer 360° VR representation with image-based interactions such as mesh deformation and water simulation, which enable users to realistically interact with 360° panoramic contents without consuming excessive computational resources. On the basis of this representation, we design and implement play-based learning scenarios to increase the interactivity and immersion of users. Then, we verify the effectiveness of our educational scenarios using a user study in terms of user-created VR contents, interactivity, and immersion. The results show that more experienced elements in VR contents improve the immersion of users and make them more actively involved.


---

## Body

## 1. Introduction

Following the development of consumer virtual reality (VR) devices using smartphones and tablets, the demand for mobile VR contents in various fields such as games, movies, and education has increased. Recently, user-created VR contents have been rapidly increasing with the advancement in low-cost image or video-capturing equipment for mobile VR. However, VR contents that use images or videos suffer from limitations with the lack of interactivity by only providing passive information; thus, the contents cannot draw active involvement of users. In education, we know that VR encourages learners to become more active through immersive experience [1]. Learning with real-time interactions provides learners instant results and helps in the decision making to reach their goals. To achieve a higher level of immersion and maximize educational effects, more engaging interactions between the learners and mobile VR contents are essential.We aim to satisfy the interactive and immersive experience using the following criteria in terms of education.(i)
Environment. We target mobile VR platforms. Compared with the desktop-based VR, the cost of mobile VR hardware such as smartphones and mobile headsets is affordable, making it suitable for production and distribution of educational contents for a large number of users.(ii)
Engagement. We allow users to navigate in the environment and perform appropriate interactions according to the educational purpose within the VR environment, providing natural adaption to the virtual learning environment.(iii)
Immersion. We provide a personalized learning environment to maximize the educational effect by increasing the immersion and participation in virtual space compared with traditional teaching methods.This paper presents a novel authoring framework for mobile VR contents using 360° photographs or videos. To create engaging interactions in the surrounding scenes, we introduce a multilayer representation where each layer can have different interaction types or properties such as deformable objects, moving objects, and water simulation. On the basis of the proposed representation, we develop play-based learning scenarios on a mobile VR platform for education in marine biology and conduct a user study on the interactivity and immersion of the scenarios. The overall pipeline of our play-based learning is shown in Figure1.Figure 1
Overall pipeline of our play-based learning for education in marine biology.The main contributions of this paper are summarized as follows:(i)
For interactive 360° VR contents, we introduced a multilayer representation using image-based interactions that are effective on mobile devices with limited performance.(ii)
We designed a learning model to increase the interactivity and immersion of users, which includes viewer-directed immersive forms, layers for interactivity, achievements, and engaging contents.(iii)
We verified our learning framework where we employed a user study in terms of user-created VR contents, interactivity, and immersion.The remainder of this paper is structured as follows: Section2 provides an overview of related works, and Section 3 describes our multilayer representation using different interactions and immersive experience in mobile VR. We present our active learning model in Section 4 and evaluate the developed learning scenarios through a user study in Section 5. Finally, we discuss our conclusions and future work directions in Section 6.


## 2. Related Work

The 360° VR contents for mobile platforms are usually created by including panoramic photographs or videos. This image-based content generation can reduce the time and cost of building a virtual space compared with the same process using three-dimensional (3D) graphics in traditional VR such as games. In addition, real-time manipulation is possible on mobile devices because it consumes a small amount of memory. The main advantage of using image-based techniques is that anyone can use their own images; thus, learner-driven personalized education is possible with the use of user-created VR contents.


### 2.1. Mobile VR in Education

In recent years, mobile VR education has been shown to provide more interactive learning with fully immersive contents than traditional education approaches [1, 2] A study on motivational active learning methods of several subjects that make learners quickly lose interest, such as Science, Technology, Engineering, and Mathematics [3], measured the level of immersion of learners using the Game Engagement Questionnaire (GEQ) to evaluate the ability of these learning methods. To create VR contents using synthetic 3D graphics, special skills, and extensive efforts are required. Furthermore, delivering user personalized experience and recreating virtual spaces according to the change in the learning context are difficult. The study in [4] provided a conceptual framework for user-generated VR learning across three contexts such as paramedicine, journalism, and new media production. In that study, an active learning environment using student-created 360° videos was proposed where the students could share the videos on social networks to collaborate together. Although 360° contents are more realistic than synthetic 3D VR, limitations on the interactions with the contents have been identified to exist.


### 2.2. Interactivity in 360° Contents

The 360° contents, which are called 360° photographs or videos, are captured by several wide-angle cameras, which are stitched and mapped in a spherical or hemispherical space to observe the surrounding scenes based on the camera [2]. Most research works for 360° content creation focus on accurate stitching for spherical mapping [5, 6], image-based lighting effects [7–9], resolution improvement, and distortion correction of images from wide-angle cameras [10]. Some researchers investigated methods to improve the interactivity in image-based VR contents by adding URLs to 360° images [11], changing scenes using button clicks [12, 13], and adding special effects to reduce the differences between composite boundaries [14]. The study in [8] recently presented an immersive system using powerful PC-tethered VR headsets that provide interactive mixed reality experiences by compositing 3D virtual objects into a live 360° video. However, there are few studies on interactive 360° contents for mobile VR headsets, such as the Samsung Gear VR and Google Daydream View, where all of the processing is done on mobile phones.


### 2.3. Immersive Experience in Mobile VR

Immersion comes from the feeling that a user actually exists in a virtual space reproduced by a computer [15, 16]. Unlike the existing computer-based environment, the mobile VR environment can create more immersive feelings owing to the development of VR devices such as head-mounted displays and tracked controllers [17]. The immersion in VR is not only created by elements such as visual, auditory, and haptic information [15] but also by content resolutions, stereo cues, behavioral fidelity of what is being simulated, and system latency [18]. Mobile VR is especially affected by factors such as resolutions and latency because of low-power constraints. Therefore, to increase the immersive experience in mobile VR, visual quality, sound quality, and intuitive interactions can be considered without consuming excessive computational resources [19]. To improve the immersion in mobile VR in terms of user interaction, a few methods to synchronize the virtual and physical spaces have been presented. The methods enable users to navigate the virtual world by mapping the user movement in the physical space to the virtual space [20–22]. In addition, interactive devices that use hand-gesture recognitions have been developed [23, 24]. However, only a few studies have been conducted on the optimization of visual quality or 3D sounds that consider mobile VR platforms.


### 2.4. Measurement of Immersive Experience

The measurement of immersion in virtual environments has been studied for a long time, mainly in the field of games [25]. GEQ [26] has been widely used as a method for measuring user engagement in games. Recently, a more accurate method to measure immersion in a virtual environment has been proposed in which 10 rating scales and 87 questions are used to show the reliability of the questionnaire [27]. We verify the effectiveness of our framework using the questions corresponding to the flow, immersion, presence, emotion, and judgment among the presented 10 scales.


## 2.1. Mobile VR in Education

In recent years, mobile VR education has been shown to provide more interactive learning with fully immersive contents than traditional education approaches [1, 2] A study on motivational active learning methods of several subjects that make learners quickly lose interest, such as Science, Technology, Engineering, and Mathematics [3], measured the level of immersion of learners using the Game Engagement Questionnaire (GEQ) to evaluate the ability of these learning methods. To create VR contents using synthetic 3D graphics, special skills, and extensive efforts are required. Furthermore, delivering user personalized experience and recreating virtual spaces according to the change in the learning context are difficult. The study in [4] provided a conceptual framework for user-generated VR learning across three contexts such as paramedicine, journalism, and new media production. In that study, an active learning environment using student-created 360° videos was proposed where the students could share the videos on social networks to collaborate together. Although 360° contents are more realistic than synthetic 3D VR, limitations on the interactions with the contents have been identified to exist.


## 2.2. Interactivity in 360° Contents

The 360° contents, which are called 360° photographs or videos, are captured by several wide-angle cameras, which are stitched and mapped in a spherical or hemispherical space to observe the surrounding scenes based on the camera [2]. Most research works for 360° content creation focus on accurate stitching for spherical mapping [5, 6], image-based lighting effects [7–9], resolution improvement, and distortion correction of images from wide-angle cameras [10]. Some researchers investigated methods to improve the interactivity in image-based VR contents by adding URLs to 360° images [11], changing scenes using button clicks [12, 13], and adding special effects to reduce the differences between composite boundaries [14]. The study in [8] recently presented an immersive system using powerful PC-tethered VR headsets that provide interactive mixed reality experiences by compositing 3D virtual objects into a live 360° video. However, there are few studies on interactive 360° contents for mobile VR headsets, such as the Samsung Gear VR and Google Daydream View, where all of the processing is done on mobile phones.


## 2.3. Immersive Experience in Mobile VR

Immersion comes from the feeling that a user actually exists in a virtual space reproduced by a computer [15, 16]. Unlike the existing computer-based environment, the mobile VR environment can create more immersive feelings owing to the development of VR devices such as head-mounted displays and tracked controllers [17]. The immersion in VR is not only created by elements such as visual, auditory, and haptic information [15] but also by content resolutions, stereo cues, behavioral fidelity of what is being simulated, and system latency [18]. Mobile VR is especially affected by factors such as resolutions and latency because of low-power constraints. Therefore, to increase the immersive experience in mobile VR, visual quality, sound quality, and intuitive interactions can be considered without consuming excessive computational resources [19]. To improve the immersion in mobile VR in terms of user interaction, a few methods to synchronize the virtual and physical spaces have been presented. The methods enable users to navigate the virtual world by mapping the user movement in the physical space to the virtual space [20–22]. In addition, interactive devices that use hand-gesture recognitions have been developed [23, 24]. However, only a few studies have been conducted on the optimization of visual quality or 3D sounds that consider mobile VR platforms.


## 2.4. Measurement of Immersive Experience

The measurement of immersion in virtual environments has been studied for a long time, mainly in the field of games [25]. GEQ [26] has been widely used as a method for measuring user engagement in games. Recently, a more accurate method to measure immersion in a virtual environment has been proposed in which 10 rating scales and 87 questions are used to show the reliability of the questionnaire [27]. We verify the effectiveness of our framework using the questions corresponding to the flow, immersion, presence, emotion, and judgment among the presented 10 scales.


## 3. Interactivity and Immersion of 360° VR Contents

In this section, we present methods to improve the interactivity and immersion of mobile VR contents using 360° photographs and videos for education in marine biology.


### 3.1. Interactive 360° VR Contents with Multiple Layers

Watching 360° contents using a VR headset replaces the user real environment with a simulated one and makes the user feel like he or she is actually there. Thus, 360° panoramic contents create a more immersive experience than regular photographs or videos. However, the viewers can usually look around by moving their head without engaging any interactions because the panoramic contents are generally mapped on a spherical surface as a single layer.To create more engaging contents, we propose interactive 360° VR contents with multiple layers, as shown in Figure2. The original input image can be manually segmented by its context such as sky, water, and land by using image tools, and the segmented regions are assigned to different layers. Then, images, animated images, and videos can be added to the original image as separate layers. Images and animated images are usually used for foreground objects, while videos are used for backgrounds. Different interaction properties can be assigned to these multiple layers based on image semantics or user intention. Finally, all the layers are mapped onto each individual sphere. The presented method uses a multilayer representation not only for image composition but also for different types of interactions that enable users to more realistically interact with the 360° panoramic contents.Figure 2
Interactive 360° VR contents with multiple layers.


### 3.2. Creating Deformable Objects

In our education scenarios for marine biology, interactive image deformation is used to create animation for some objects such as seaweeds and starfish based on transformation of the vertices of a 2D mesh on a particular layer. The resulting visual quality depends on the geometric resolution of the mesh, but mobile devices with limited computational power reduce the maximum achievable quality. Moreover, computationally expensive deformation techniques cause increased latency when the user interacts with the objects within a mobile VR environment. Therefore, we adopt a force-based approach to deform a mesh on a spherical surface, as shown in Figure3.Figure 3
Force-based deformation of a mesh on a spherical surface.The force is attenuated using the inverse proportion of the square of the distance from the center of the force, for example, user picking pointp. Attenuated force Fv is defined by dividing original force F by one plus the square of the distance, i.e., 1+d2 , to ensure that the force is at full strength when the distance is zero.(1)Fv=F1+d2.The force can be converted into acceleration viaa=F/m,  where m is the mass. To simplify, we set mass m = 1 for each vertex. Therefore, m is ignored as a=F. Then, the change in velocity Δv is defined by Equation (2), where Δt is the time difference:(2)Δv=FΔt.Vertex velocityVv is computed using Equation (3), where the direction is derived by normalizing the vector that points from the center of the force to the vertex position. The vertices are moved to new positions via Δp=vΔt.(3)Vv=ddΔv.


### 3.3. Realistic Simulation of Water Surface

In our virtual marine contents, realistic simulation of the water surface is an essential method to increase immersion. Real-time performance of the simulation is also required to work with various user interactions in the mobile-based VR environment. To satisfy these requirements, we adopt a physics-based approach to simulate water surface waves by deforming a 2D ocean mesh and implementing it on a graphics processing unit (GPU) hardware. The dynamic behavior of the water surface, such as the ripple effects, is governed by the following 2D wave equation:(4)∂2h∂t2=c2∂2h∂u2+∂2h∂v2.Equation (4) defines height displacement h on the mesh surface depending on wave speed c. t refers to time, and u and v represent the coordinates of the 2D mesh. To numerically solve this second-order partial differential equation, we discretize it by replacing the partial derivatives with the central differences using a regular spatial grid of spacing (k=Δu=Δv) and a constant time step (Δt), yielding the following equation:(5)ht+1i,j−2hti,j+ht−1i,jΔt2=c2hti+1,j−2hti,j+hti−1,jk2+hti,j+1−2hti,j+hti,j−1k2.New height displacementht+1 can be explicitly integrated from old displacements ht and ht−1.(6)ht+1i,j=c2Δt2k2hti+1,j+hti−1,j+hti,j+1+hti,j−1−4hti,j+2hti,j−ht−1i,j.Although this explicit method is fast and easy to implement as a GPU shader, it is only conditionally stable. If the time step becomes very large, the system can become unstable. To increase the stability of the simulation, an artificial attenuation can be added to the new displacement, i.e.,(7)ht+1=αht+1,α<1.In our experiments, we set attenuation coefficientα to 0.985 to generate the ripple effects that naturally disappear. Figure 4 shows the results of water simulation in terms of mouse movement and the attenuation coefficients.Figure 4
Simulation results of water surface waves in terms of mouse movement (a) and the attenuation coefficients (b).
(a)
(b)


### 3.4. Immersive Experience in Mobile VR Environments

The immersion experience in a mobile VR environment is mainly influenced by the viewing devices and content types, as shown in Figure5. Because the VR headsets block out all sights of the outside world, watching 360° images or videos using them makes the users more immersed than watching regular photographs or videos. The wide viewing angle of the headset, such as more than 110°, and the higher quality of the visuals, such as more than 4K resolution, as well as the faster and more stable frame rates would result in deeper immersive experience.Figure 5
Degree of immersion in terms of viewing devices and content types.Furthermore, wearing headphones makes the users even more immersed by also blocking outside sound. Realistic 3D audio effects on a mobile platform will increase their perception of immersion and reduce the side effects, including headaches and motion sickness. Therefore, the users can focus on the learning objectives in a virtual environment and become active learners.


## 3.1. Interactive 360° VR Contents with Multiple Layers

Watching 360° contents using a VR headset replaces the user real environment with a simulated one and makes the user feel like he or she is actually there. Thus, 360° panoramic contents create a more immersive experience than regular photographs or videos. However, the viewers can usually look around by moving their head without engaging any interactions because the panoramic contents are generally mapped on a spherical surface as a single layer.To create more engaging contents, we propose interactive 360° VR contents with multiple layers, as shown in Figure2. The original input image can be manually segmented by its context such as sky, water, and land by using image tools, and the segmented regions are assigned to different layers. Then, images, animated images, and videos can be added to the original image as separate layers. Images and animated images are usually used for foreground objects, while videos are used for backgrounds. Different interaction properties can be assigned to these multiple layers based on image semantics or user intention. Finally, all the layers are mapped onto each individual sphere. The presented method uses a multilayer representation not only for image composition but also for different types of interactions that enable users to more realistically interact with the 360° panoramic contents.Figure 2
Interactive 360° VR contents with multiple layers.


## 3.2. Creating Deformable Objects

In our education scenarios for marine biology, interactive image deformation is used to create animation for some objects such as seaweeds and starfish based on transformation of the vertices of a 2D mesh on a particular layer. The resulting visual quality depends on the geometric resolution of the mesh, but mobile devices with limited computational power reduce the maximum achievable quality. Moreover, computationally expensive deformation techniques cause increased latency when the user interacts with the objects within a mobile VR environment. Therefore, we adopt a force-based approach to deform a mesh on a spherical surface, as shown in Figure3.Figure 3
Force-based deformation of a mesh on a spherical surface.The force is attenuated using the inverse proportion of the square of the distance from the center of the force, for example, user picking pointp. Attenuated force Fv is defined by dividing original force F by one plus the square of the distance, i.e., 1+d2 , to ensure that the force is at full strength when the distance is zero.(1)Fv=F1+d2.The force can be converted into acceleration viaa=F/m,  where m is the mass. To simplify, we set mass m = 1 for each vertex. Therefore, m is ignored as a=F. Then, the change in velocity Δv is defined by Equation (2), where Δt is the time difference:(2)Δv=FΔt.Vertex velocityVv is computed using Equation (3), where the direction is derived by normalizing the vector that points from the center of the force to the vertex position. The vertices are moved to new positions via Δp=vΔt.(3)Vv=ddΔv.


## 3.3. Realistic Simulation of Water Surface

In our virtual marine contents, realistic simulation of the water surface is an essential method to increase immersion. Real-time performance of the simulation is also required to work with various user interactions in the mobile-based VR environment. To satisfy these requirements, we adopt a physics-based approach to simulate water surface waves by deforming a 2D ocean mesh and implementing it on a graphics processing unit (GPU) hardware. The dynamic behavior of the water surface, such as the ripple effects, is governed by the following 2D wave equation:(4)∂2h∂t2=c2∂2h∂u2+∂2h∂v2.Equation (4) defines height displacement h on the mesh surface depending on wave speed c. t refers to time, and u and v represent the coordinates of the 2D mesh. To numerically solve this second-order partial differential equation, we discretize it by replacing the partial derivatives with the central differences using a regular spatial grid of spacing (k=Δu=Δv) and a constant time step (Δt), yielding the following equation:(5)ht+1i,j−2hti,j+ht−1i,jΔt2=c2hti+1,j−2hti,j+hti−1,jk2+hti,j+1−2hti,j+hti,j−1k2.New height displacementht+1 can be explicitly integrated from old displacements ht and ht−1.(6)ht+1i,j=c2Δt2k2hti+1,j+hti−1,j+hti,j+1+hti,j−1−4hti,j+2hti,j−ht−1i,j.Although this explicit method is fast and easy to implement as a GPU shader, it is only conditionally stable. If the time step becomes very large, the system can become unstable. To increase the stability of the simulation, an artificial attenuation can be added to the new displacement, i.e.,(7)ht+1=αht+1,α<1.In our experiments, we set attenuation coefficientα to 0.985 to generate the ripple effects that naturally disappear. Figure 4 shows the results of water simulation in terms of mouse movement and the attenuation coefficients.Figure 4
Simulation results of water surface waves in terms of mouse movement (a) and the attenuation coefficients (b).
(a)
(b)


## 3.4. Immersive Experience in Mobile VR Environments

The immersion experience in a mobile VR environment is mainly influenced by the viewing devices and content types, as shown in Figure5. Because the VR headsets block out all sights of the outside world, watching 360° images or videos using them makes the users more immersed than watching regular photographs or videos. The wide viewing angle of the headset, such as more than 110°, and the higher quality of the visuals, such as more than 4K resolution, as well as the faster and more stable frame rates would result in deeper immersive experience.Figure 5
Degree of immersion in terms of viewing devices and content types.Furthermore, wearing headphones makes the users even more immersed by also blocking outside sound. Realistic 3D audio effects on a mobile platform will increase their perception of immersion and reduce the side effects, including headaches and motion sickness. Therefore, the users can focus on the learning objectives in a virtual environment and become active learners.


## 4. Active Learning Design: Marine Biology Case Study

The learning style of the new generation tends to be more self-directed, engaged, and flexible. Therefore, interactive engagement and active involvement are key elements in digital learning environments for the new generation. The simple forms of mobile VR contents are 360° photographs or videos. When we use the content for educational purpose, lack of interactivity makes the learners become passive in the virtual world. Thus, improvement in the interactivity and immersion in mobile VR contents is crucial to engage learners in the contents and keep them focused and motivated.For active participant learning, we present play-based learning for education in marine biology. The learning scenarios are designed and implemented as mobile VR environment and allow the users to interact with the immersive mobile VR contents. A low-cost approach to user-created VR contents makes the mobile VR environment potentially learner-centered education to enable learners to create active participant VR contents. The main features of our learning scenarios include viewer-directed immersive form, adding layers for interactivity, achieving play-based learning, and engaging contents for active involvement and immersion.


### 4.1. Viewer-Directed Immersive Form

The viewer can decide to look at any part of the 360° scenes. The scene can be typically captured from what is around the camera (an outward view) or around an object at different angles (an inward view), as shown in Figure6. Whereas consumer 360° cameras are mainly focused on outward views, a few apps enable users to capture 360° scenes by having the camera circle around an object. Some cameras simply capture this scene as a hemisphere (a 180° × 360° view), and others can show it as a sphere (a 360° × 360° view). Immersive 360° contents are a valuable tool to create focused learning experience. Figure 7 shows that the user can navigate a 360° marine scene using a VR headset and can control what he or she wishes to view.Figure 6
Capture methods for the outward and inward 360° views and navigation of a 360° content.Figure 7
The 360° navigation of a boat on the water for virtual fishing.
(a)
(b)
(c)


### 4.2. Adding Layers for Interactivity

Layers are a powerful method of creating interactivity and branched stories. For personalized learning, the user can integrate marine plants and animals on additional layers together with animation effects, as shown in Figure8, which are designed to more engage the learners to the content.Figure 8
Creating multiple layers for interactive engagement.


### 4.3. Achievements in Play-Based Learning

Achievements can be a great means of increasing the learner engagement within the learning content and encourage learners to experiment with features that are not commonly used. In addition, achievements can be a fun way for learners to engage in a light-hearted competition. Figure9 shows that the adapted picture from [28] illustrates the three zones of marine life according to the depth of the sea, and Figure 10 shows the user interface of our virtual fishing-game scenario. A high score is given to the player when he or she catches the more difficult fish in the deep sea. Table 1 shows the different scoring depending on the marine life zones.Figure 9
Zones of marine life according to the sea depth.Figure 10
Education on marine life through a virtual fishing-game scenario.
(a)
(b)
(c)Table 1
Different scoring depending on the marine life zones.
Marine organism
Depth
Score
Seaweed, crab, lobster, sea horse, tropical fish, dolphin, jellyfish, and sea lion
Sunlit Ocean (Euphotic) Zone
1
Clam, starfish, shark, whale, tuna, and seal
Twilight Ocean (Disphotic) Zone
3
Coral, eel, lugworm, and anglerfish
Midnight Ocean (Aphotic) Zone
5


### 4.4. Engaging Contents for Active Involvement and Immersion

By adding the deformable objects discussed in Section3.2 on a particular layer or moving objects such as lobsters as shown in Figure 11(a), we can create more interesting contents. During a fishing game, when the player casts a fishing rod, the movement of surface water can be simulated by the method presented in Section 3.3, producing more realistic effects as shown in Figure 11(b). To maximize the immersion, engagement, and learning outcome, we also created marine scenes with head-tracking 3D audio effects and video layers as shown in Figures 11(c) and 11(d).Figure 11
Creating engaging contents. (a) Moving objects, (b) water simulation, (c) 3D audio effects, and (d) video layers.
(a)
(b)
(c)
(d)


## 4.1. Viewer-Directed Immersive Form

The viewer can decide to look at any part of the 360° scenes. The scene can be typically captured from what is around the camera (an outward view) or around an object at different angles (an inward view), as shown in Figure6. Whereas consumer 360° cameras are mainly focused on outward views, a few apps enable users to capture 360° scenes by having the camera circle around an object. Some cameras simply capture this scene as a hemisphere (a 180° × 360° view), and others can show it as a sphere (a 360° × 360° view). Immersive 360° contents are a valuable tool to create focused learning experience. Figure 7 shows that the user can navigate a 360° marine scene using a VR headset and can control what he or she wishes to view.Figure 6
Capture methods for the outward and inward 360° views and navigation of a 360° content.Figure 7
The 360° navigation of a boat on the water for virtual fishing.
(a)
(b)
(c)


## 4.2. Adding Layers for Interactivity

Layers are a powerful method of creating interactivity and branched stories. For personalized learning, the user can integrate marine plants and animals on additional layers together with animation effects, as shown in Figure8, which are designed to more engage the learners to the content.Figure 8
Creating multiple layers for interactive engagement.


## 4.3. Achievements in Play-Based Learning

Achievements can be a great means of increasing the learner engagement within the learning content and encourage learners to experiment with features that are not commonly used. In addition, achievements can be a fun way for learners to engage in a light-hearted competition. Figure9 shows that the adapted picture from [28] illustrates the three zones of marine life according to the depth of the sea, and Figure 10 shows the user interface of our virtual fishing-game scenario. A high score is given to the player when he or she catches the more difficult fish in the deep sea. Table 1 shows the different scoring depending on the marine life zones.Figure 9
Zones of marine life according to the sea depth.Figure 10
Education on marine life through a virtual fishing-game scenario.
(a)
(b)
(c)Table 1
Different scoring depending on the marine life zones.
Marine organism
Depth
Score
Seaweed, crab, lobster, sea horse, tropical fish, dolphin, jellyfish, and sea lion
Sunlit Ocean (Euphotic) Zone
1
Clam, starfish, shark, whale, tuna, and seal
Twilight Ocean (Disphotic) Zone
3
Coral, eel, lugworm, and anglerfish
Midnight Ocean (Aphotic) Zone
5


## 4.4. Engaging Contents for Active Involvement and Immersion

By adding the deformable objects discussed in Section3.2 on a particular layer or moving objects such as lobsters as shown in Figure 11(a), we can create more interesting contents. During a fishing game, when the player casts a fishing rod, the movement of surface water can be simulated by the method presented in Section 3.3, producing more realistic effects as shown in Figure 11(b). To maximize the immersion, engagement, and learning outcome, we also created marine scenes with head-tracking 3D audio effects and video layers as shown in Figures 11(c) and 11(d).Figure 11
Creating engaging contents. (a) Moving objects, (b) water simulation, (c) 3D audio effects, and (d) video layers.
(a)
(b)
(c)
(d)


## 5. Experiments

We developed the proposed learning framework on a desktop computer using Unity3D engine and then exported the educational scenarios to a mobile VR platform using Samsung Galaxy S7, Gear VR, and a dedicated controller as an interaction tool.When we created an interactive 360° VR content, the number of vertices in the 3D spherical model for the foreground image layer was very important for realistic mesh deformation. Figure12 shows that the results using a high-resolution 3D sphere (right) provided smoother interactions than those using a low-resolution 3D sphere (left). Because the realistic interaction methods adopted in this study relatively needed a small amount of computation for mesh processing and used only a simple 3D spherical model compared with the full 3D VR contents, our learning contents were maintained over 60-frame-per-second performance without latency even if we used a 3D spherical model with relatively high resolution.Figure 12
Results of the mesh deformation according to the resolution of spherical models. (a) Deforming objects. (b) Simulation of water surface. (c) VR contents.
(a)
(b)
(c)To verify the effectiveness of the proposed interactive and immersive learning using 360° VR contents, we experimented on three scenarios in marine biology (see scenarios 1–3 in the supplementary materials (availablehere)), which were created based on the active learning design described in Section 4. In scenario 1, users can explore an underwater environment and interact with diverse marine life on the added deformable image layers. In scenario 2, the users can discover marine life while playing virtual fishing on a boat. In scenario 3, they can learn about the types of fish according to the sea depth while playing a waterside fishing game. When they catch a fish, the detailed information of the fish, such as scientific name and species name, is shown on the billboard, and different scores are given based on the sea depth. Each scenario consisted of three different levels of scenes: (1) simple user-created scenes with multiple layers from different image sources, (2) interactive scenes with animations such as realistic water simulation and deformable objects, and (3) immersive scenes with video layers and 3D audio effects. The participants answered questionnaires on two key VR elements, namely, interactivity and immersion, after finishing their experience in each scenario. Then, we analyzed the response results and evaluated whether the immersion of educational contents increased with the addition of more interactive elements.For the user study, we used Samsung Galaxy S7, Gear VR, and a dedicated controller. Prior to the user study, the participants were asked about their previous experience in computer usage and VR devices. For a smooth user-study process, the participants used mobile VR devices and controllers for approximately 5 min to adapt to the interaction style. Then, experiments on interactivity and immersion were conducted by randomly choosing two of the three scenarios to keep their concentration on the experiments by limiting the maximum duration to 20 min. Each scenario required approximately 6 min to perform from Levels 1 to 3, and the average time for the experiments took a total of 17 min, including the warm-up time.In each scenario, Level 1 showed the most basic 360° VR images, whereas Level 2 showed the scene with realistic interactive animation such as moving waves of water and object animation using mesh deformation. In Level 3, the scenes could have a video layer as a moving object or background, such as sky and water, and 3D audio effects could be added to enhance the immersive feelings. Table2 lists the added factors to improve the user-created form, interactivity, and immersion in each learning scenario.Table 2
Techniques for different levels of experience in three scenarios.
Experience
Technique
Scenario 1 (underwater)
Scenario 2 (fishing)
Scenario 3 (coast)
Level 1: user-created scene
Multiple layers
o
o
o
Level 2: interactive scene
Water simulation
o
o
Physics-based deformation
o
o
Level 3: immersive scene
Image layer (objects)
o
o
o
Video layer (backgrounds)
o
3D audio effects
o
oThe participants were encouraged to experience the scenes using one or more of the techniques included in each experience category, whichever was selected among the given three scenarios. To compare the interactivity and immersion according to the level of experience, the participants provided points from 1 to 10 (10 is the best) for each experience category after finishing all level experience in each scenario. Then, two different questionnaires in Table3 were given to measure the interactivity (nine questions) and immersion (17 questions). Twenty-six people who were 14 to 30 years old participated in this user study. We included both, subjects who were familiar with IT and had previous experience in the use of VR and those who were unfamiliar with IT and had no experience of VR, in the user study.Table 3
Questionnaires for interactivity and immersion.
Scales
Interactivity
It was a good time to learn how to control the game
Usability
P
The controller was easy to use
Usability
P
Over time, the controller can be used proficiently
Usability
P
The interaction method was natural
Interactivity
P
The interaction method was clear and easy to understand
Usability
P
It has become possible to interact well over time
Interactivity
P
It was easier to use the controller than traditional input devices (keyboard, mouse, and joystick)
Interactivity
P
Animation (waveform of water, movement of objects) by interaction is reproduced as expected
Interactivity
P
The greater the number of interacting entities, the greater the immersion (depending on the degree of implementation)
Immersion
P
Immersion
I did not know that the time was running while I was playing the game
Flow
P
After finishing the game play, the time apparently passed faster than expected
Flow
P
I could not figure out what was happening around me while playing the game
Immersion
P
I felt realistic in the configured virtual environment
Presence
P
At some point, I forgot that I had a controller
Immersion
P
I was very immersed in the game
Immersion
P
I needed time to immerse myself in the game
Immersion
N
I tried to get more points (a fishing scenario)
Emotion
P
The game was difficult (a fishing scenario)
Emotion
N
I wanted to stop the game in the middle (a fishing scenario)
Emotion
N
I liked the graphics and images
Emotion
P
Overall, the game was fun
Judgment
P
I want to play the game again
Judgment
P
I would like to recommend it to people
Judgment
P
The sound effect felt like it was real (a scenario with a sound)
Immersion
P
Describe what you liked about the virtual reality content
Description
—
Describe improvements to this content when compared to existing virtual reality content
Description
—In our experiment, the participants chose Scenario 1 for 20 people, Scenario 2 for 13 people, and Scenario 3 for 19 people. After experiencing the scenarios, they provided score for the interactivity and immersion in each scenario. Figure13 shows the results of the average scores and standard deviations according to the level of experience in each scenario. In the interactivity case, the number of immersive elements was shown to increase from Levels 1 to 3. The interactivity also increased in all three scenarios. In the immersion case, Scenario 3 showed the highest scores at Level 1. The reason for this result is that the 3D audio effects were added to Scenario 3 to increase the immersion, and it was not reflected in Scenario 1.Figure 13
Average scoring results of the interactivity and immersion for three scenarios according to the different experience levels.
(a)
(b)Next, we conducted a survey on the interactivity and immersion for each scenario. The questionnaire for the interactivity consisted of nine questions: four for usability, four for interactivity, and one for immersion. The questionnaire for immersion consisted of 17 questions according to the measurement of user experience in the immersive virtual environment [27]. Our questionnaire consisted of two questions for flow, five for immersion, one for presence, four for emotion, three for judgment, and two for descriptive evaluation of our framework.We analyzed the response of each participant by calculating the scores. The answers to the positive questions are marked with “P” and the negative questions are marked with “N” in Table3; we calculated scores according to the 5-point Likert scale (strongly agree = 5, agree = 4, neutral = 3, disagree = 2, and strongly disagree = 1). We also set the scores in the opposite order for the case of negative questions (marked with “N” in Table 3). Thus, we could obtain consistent values from negative and positive responses. The total participant response for each question was calculated as an average score. Then, the average score for each scale was calculated according to the number of questions in the scale. Figure 14 shows the evaluation scores and standard deviations of the interactivity and immersion for each scenario according to the different scale, which are marked in Table 3. The reason ‘immersion’ appears in both interactivity and immersion is that interaction methods or scene elements can affect immersion [27].Figure 14
Average survey results of the interactivity and immersion of three scenarios according to scales.
(a)
(b)When we analyzed the overall answers, no negative values (less than 3) in all scenarios were obtained that appeared to be a negative response, but standard deviations existed among the scenarios depending on the different scale properties. In the questionnaire on interactivity, Scenario 1 showed the highest values in the interactivity and usability scales. We analyzed that Scenario 1 contained a small number of immersive experience elements; thus, it provided a simple interaction style such as touching marine lives underwater. From the viewpoint of usability and interactivity, we found that more mixed immersive experience was included, which showed the lower scores in Scenarios 2 and 3, compared to Scenario 1. In the questionnaire about immersion, each scenario was shown to have different characteristics depending on the scale properties. Scenario 2 showed higher judgment and flow, and Scenario 3 exhibited higher scores in the overall scales, except judgment. It shows that as complex immersive experience elements were added, the addition positively influenced the overall scales.In addition, in the descriptive evaluation, the participants responded that it was good to create realistic user-oriented content and generate natural and realistic animation using mesh deformation. The participants also suggested that the rendering quality in some scenes should be improved and optimized.


## 6. Conclusions

We have developed an interactive and immersive 360° content-authoring framework for marine biology education. The developed framework is suitable for many students to study using low-cost mobile VR devices because the developed contents can be installed in smartphones or standalone VR equipment. For active participations, we presented user-created forms using additional layers, realistic image-based interactions utilizing elements in the given images, and more immersive contents using animated objects, videos, and 3D audio effects. We conducted a user study to verify the interactivity and immersion of our education scenarios using different levels of experience. As a result, we found that more experience elements improve the immersion of users and make them become more active.Because the proposed method is processed based on images, creating contents using various low-cost 360° image-capturing devices is easy; thus, it can be utilized not only for educational contents but also for various image-based applications such as marketing and journalism. In the future, the image-based interactions can be improved by applying adaptive mesh deformation to enable more realistic animation depending on the object properties and by automatically segmenting some elements in the images based on the contexts. We will also investigate methods to optimize the presented framework on mobile devices to use more layers and various interactions in an effective manner.



---

*Source: 2306031-2018-10-11.xml*