# A Deep Learning Model with Virtual Reality Technology for Second Language Acquisition

Authors: Yu Zhao; Shuying Liu
Journal: Mobile Information Systems
 (2022)

---

## Abstract

Attention is considered a sufficient condition for transforming input into absorption in the field of second language acquisition and is a major cognitive factor influencing second language learning. The temporal characteristics of attentional shift are a more accurate reflection of second language learners’ thinking processes. Based on this, this study uses deep learning techniques and VR technology to explore the attentional patterns of the second language (English) learners when processing online tasks. The experiments show that the linear attentional control model of young second language learners is closely related to their online task performance, which can visually explain the effect of their linear attentional control on online task completion. The model also has a high regression/prediction accuracy.


---

## Body

## 1. Introduction

Cognitive science provides the mechanisms and processes of intelligent activities such as perception and thinking for the study of deep learning in artificial intelligence [1]. The development of cognitive science has advanced deep learning in artificial intelligence by integrating it with a variety of disciplines to analyse and solve problems from different perspectives [2]. At the same time, the development of deep learning has greatly contributed to language learning, especially second language learning.Language learning based on deep learning is language learning aided by artificial intelligence devices and can intelligently help people to complete communicative tasks and achieve the social function of interacting with people, their knowledge, and their environment [3]. There are many intelligent devices that are closely linked to language learning, and they can be divided into three main categories: robots, specialist software, and integrated platforms. The main types of intelligent language devices are Xunfei translators, lip recognition robots, chariots (Chatterbot), Dasai intelligent educational robots, and Xiaobu English educational companion robots. Professional software includes Google Translator, Kingsoft, and Lingoes Translator, e-learning platforms include Library Genesis and Goodreads, Moodle, Black-board, Sakai, and SuperStar Pan-Asia [4]. These three types of intelligent devices are collectively known as artificial intelligent language learning (AILL), and the difference lies in their different levels of intelligence.“Virtual reality” (VR) refers to “seeing something as reality,” that is, seeing virtual reality visually, generating general assembly drawings, patterns, and then converting into part entities, and inputting CNC machine tools to parts in automatic processing. The real reality (body) is then automatically machined into the CNC machine [5]. VR technology uses computing devices to render and simulate visual and auditory scenes, and these rendering techniques stimulate people’s visual and auditory senses, providing the user with the best possible simulation of visual and auditory organs to make them impressive. The most immediate and stimulating effects are “immersion” and “participation in immersion” [6]. In short, these simulations are generated through computational simulation and are false effects and scenarios.In a sense, computers are not only tools for the study of the mind, but computers with appropriate programming are themselves minds, with cognitive action functions that are equivalent to those of the brain and are extensions of brain cognition, so it is reasonable to believe that the cognitive basis of AI-based language learning is extension cognition. In view of this, this paper, based on an introduction to AI and language learning, explores the cognitive basis of intelligent language learning, i.e., extended cognition, and points out its new implications for second language acquisition [7].


## 2. Related Work

In the early days of VR, Zhao et al. [8] worked on the application of VR technology to the medical field, proposing the use of computers to create a virtual environment for medical practitioners; Schwienhorst [9] studied a televised surgical device in which a doctor observes a three-dimensional image of the operating room to operate an instrument attached to a computer, which transmits the doctor’s movements to a remote control device that performs the surgery and gives the doctor feedback on the force, texture, and sound. Besacier et al. [10] describe the role of VR technology as a way of enabling people to see things that would otherwise be impossible to reach, for example, viewing and analysing the distribution of the heat field generated by the combustion of an internal combustion engine after ignition and allowing pilots to train in a simulated cabin to manoeuvre an aircraft.With the development of education informatization, it has become a proposition to be discussed about how education departments, universities, and teachers can integrate VR technology into teaching Chinese as a foreign language, enhance teaching effectiveness, and promote the quality of Chinese language teaching [11, 12]. At present, there are only a few studies and design based on the application of VR technology in teaching Chinese as a foreign language or in second language acquisition. Lin [13] analysed virtual reality tools for second language acquisition and argued that virtual reality theory is largely based on language learning.Understood from an abstract perspective, cognition is nonchanneled and nonmodal-based cognition. However, from a biological perspective, cognition is embedded in the body and the environment. Research on extended cognition and experiential cognition has been in the mainstream of cognitive science for a long time, but research on the impact of Second Language Acquisition (SLA) [14, 15] has only begun. SLA based on the integration of extension cognition and experiential cognition emphasises the value of the technological environment and follows three main principles: the inseparability principle, i.e., in SLA, mind, body, and environment work together, the adaptability principle, i.e., SLA facilitates people’s survival and prosperity in a complex world, and the synergy principle. The inseparability principle is that, in SLA, the mind, body, and environment work together, the adaptability principle is that SLA facilitates people to survive and prosper in a complex world, and the alignment principle is that the main engine of SLA is interaction and synergy [16].In addition to its role, VR technology currently has limitations, such as limiting the user’s range of motion, the inability to achieve full input for nonverbal communication, and the user’s susceptibility to fatigue AI and VR technology is both objects of study in robotics, a branch of computer science that involves the study and design of intelligent computer systems. Artificial intelligence is relative to human intelligence. The study of techniques enables computers to simulate certain human thought processes and intelligent behaviour (e.g., learning, reasoning, thinking, and planning) [17]. Examples include Google’s Go AI Alphago, Google’s automated driver less technology, and various types of robots (e.g., industrial robots, delivery robots, competition robots, grocery delivery robots, shopping guide robots, and security robots) [18].Strictly speaking, an intelligent robot is an automatic machine that perceives, thinks, and acts. Perception refers to the ability to detect, recognize, and describe the robot’s external environment and its own state. By thinking, we mean that the robot does not simply do what it is told to do in a certain way, but that it has the ability to solve problems on its own or that it can find its own solutions to problems through learning. Action means that the robot also has an operating mechanism with a drive that can perform various tasks [19].


## 3. Deep Learning Models

In this experiment, the parameters and structure of the model were adapted to meet the data processing needs of this experiment using the techniques of [20]. The model used in this paper consists of a six-layer network structure: two convolutional layers, two downsampling layers, a fully connected layer, and an output layer (as shown in Figure 1). Each convolutional layer consists of three parts, convolution, pooling, and nonlinear activation, which are mainly used to extract spatial features, while the sampling layer implements the average pooling [21].Figure 1
Convolutional neural network model.By using CNN for feature recognition and prediction of eye-movement images, it is possible to establish a mapping between attentional transfer patterns and the processing power of an online bilingual task. However, the problem with the black-box technique is that while it is possible to accurately match inputs and outputs, the nonlinear transformations in between make it difficult to explain or account for what input elements which have that effect on the outcome. Therefore, there is also a need to add an interpretable module to the CNN in order to identify key features of the CNN that makes decisions about image recognition [22].In this paper, a heat map visualisation approach is used, whereby a heat map is used to reflect the key features that identify the object. Gradient-weighted class activation mapping (Grad-CAM) [23] is used as a heat map visualisation method to interpret the decision basis of the classification results and to visually represent the key features in eye-movement image recognition.Grad-CAM takes the key feature map after the last layer of convolution operations and then weights each channel in that feature with the gradient of the class associated with that channel.The Grad-CAM algorithm draws a heat map of a single image, which corresponds to the class region of interest in the image. In this paper, we achieve the extraction of group feature patterns by averaging the colour information of the feature heat map over all training samples. According to the Grad-CAM algorithm [24], the weight of all feature maps can be calculated from(1)αk,mc=1z∑i∈w∑j∈h∂yc,m∂Ak,mi,j.Once the weights of the categories on all feature maps are obtained, the heat map is obtained by summing their weights [25]. However, in this paper, a ReLU process is done on the final weighted feature heat map. The reason for adding a ReLU layer is that we only care about those pixels that have a positive effect on category c. If the ReLU layer is not added, some pixels belonging to other categories may end up being brought in, thus affecting the interpretation. Therefore, the heat map of the key features can be derived from(2)LGrad−CAMc,m=ReLU∑kαk,mc∗Ak,m,where LGrad−CAMc,m represents the key feature heat map for the mth graph to discriminate it as class c. Therefore, the classification key feature pattern for all incoming networks classified as c (assuming there are M diagrams) can be represented by(3)LGrad−CAMc=1M∑m∈0,MLGrad−CAMc,m,where LGrad−CAMc represents the average result of the feature map weighted output of all images of category c that are passed into the network. This is then normalised between 0 and 1 to draw a heat map corresponding to the region of interest of the category in the image.Second language learners who have good attentional control, i.e., the ability to follow visual cues to shift their attention immediately while receiving second language stimuli as required by attentional control, have a 64% probability of outperforming the average in immediate long sentence repetition; conversely, they are likely to underperform the average with the same probability [26].


## 4. Data Collection

The data in this paper were collected from 19 students (10 boys and 9 girls, mean age 6.42, and standard deviation, SD = 0.507) in the first grade of a primary school. The TobiiT120 was used to collect the eye-movement data. The participants were asked to watch a video of approximately 4 minutes of English reading aloud. The vocabulary in the video, in which these young second language learners had never been exposed to before, was used to exclude attentional distractions from familiar words in order to reflect the learners’ ability to control their attention to second language stimuli. While the video is being read aloud, the learners are given attention-directing cues using highlighting squares and asking them to listen to the word being read aloud while following the highlighting cues. In addition, to avoid any possible distractions, the videos are presented with a simple white background and black font, as shown in Figure2. The pace of the video was adapted to the attention level of the participants, and the audio speed was 15% slower than the original audio speed, with approximately 56.65 words per minute throughout the video. Nineteen participants rotated through the experiment in order. The experiment was conducted over a period of 2 months, with all participants completing two rounds and some of them proceeding to a third round.Figure 2
Screenshot of the video footage used for eye-movement data collection.Using equation (3) to calculate the average long sentence repetition rate per participant, in equation (1), ALSRRi,j,z is the long sentence repetition rate of participant i on long sentence j in the zth long sentence repetition task, Ri,j,z is the number of words repeated by participant i on long sentence j in the zth long sentence repetition task, and Sj is the word span of long sentence j [27]:(4)ALSRRi,j,z=12∑j=12Ri,j,zSj.The average long sentence repetition rate of all participants in each long sentence repetition task is then calculated from equation (4), and ALLz, in equation (5), represents the average long sentence repetition rate of all participants in the zth task. Two third of the eye-track samples are randomly selected as the training set, and the long sentence repetition rate of their corresponding participants for that attention control ALSRRi,j,z which is marked as positive (+) if ALSRRi,j,z>ALLz; otherwise, it is marked as negative (-):(5)ALLz=119∑i=119ALSRRi,j,z.


## 5. VR-Based Second Language Acquisition Experience

This paper combines VR technology with second language teaching to design “immersion virtual simulation classroom” scenarios that provide second language learners with an intuitive sense of behavioural and psychological realism. The paper is currently exploring this approach to the virtual simulation classroom and will only briefly address the design of the experiential scenarios or interaction scenarios. An experiential scene is essentially a high-frequency scene in which a second language learner can experience key Chinese words and phrases. Its quality is interactivity [25]. Interactivity allows the learner to change from an outsider to a participant, from a third-person perspective to a first-person perspective, allowing the learner to experiment with interaction and dialogue in the experiential scene as they wish and to receive feedback from the AI, thus achieving the purpose of acquiring, practicing, and mastering the phonology, grammar, and pragmatics of certain words and phrases (as shown in Figure 3).Figure 3
Word cloud based on VR display.These interactive scenarios address the lack of “immersion” and “interactivity” in the classroom, the lack of naturally interactive role-playing, the lack of appreciation of the vast and complex euphemisms, and meanings of the Chinese language; they enable “immersive learning” in any context (e.g., through the current Magic Leep support), the challenge of “immersive learning” in any scenario (e.g., through the current Magic Leep support), the challenge of building scenarios and learning languages based on scenarios that are difficult to realise in real or imagined conditions, e.g., disaster drills, space environments, and advanced laboratory environments in different noises, as shown in Figure4.Figure 4
English text recognition in different modes.The following are possible applications for the “immersion virtual simulation classroom”: (1) VR training: where the content to be learned is presented on-screen in 3D as an alternative to complex props, such as the learning of time expressions and the learning of orientation words; (2) VR modelling: allows the learner to visualise how a particular item looks like in different contexts and to interact with the model and receive immediate feedback so that they can learn the real effectiveness of their language output, e.g., teaching word by word, teaching colour words, teaching medical, design, and programming pre-professional Chinese; (3) VR books: when the learner wears the HMD camera and scans the intended content of the book, 3D animations, videos, and sounds. This can bridge the gap between classroom knowledge and real life by bridging the gap between teachers’ language and two-way communication; (4) discovery learning: similar to the task-based teaching method, learners use AR technology to identify the corresponding Chinese characters and explanations in real time and in the field on campus, in monuments, in shopping malls, and other applications, using real-life scenarios or what they have learned, and discover new knowledge alone. New knowledge is learned quickly and effectively in the process of problem solving, as shown in Figure5.Figure 5
Different vocabulary learning effects.The design of the “immersion virtual simulation classroom” is in fact a situational teaching strategy that draws on novelty, using virtual reality technology. On the contrary, it removes the demanding element of the teacher and makes it possible to adjust the content and intensity of the two-way communication according to one’s own wishes in [28]. In the long run, the “immersion virtual classroom” experience optimises the old classroom structure and frees up more time for teachers to implement talent development programmes, accelerating the modernisation of education, as illustrated in Figure 6.Figure 6
Effect of different learning categorization.So far, attentional control patterns of the second language learners presented by convolutional neural networks and Grad-CAM techniques have constructed highly accurate predictions for all marker indications [29]. However, although it is known that learners with poor attentional control do not perform well on long sentence retrieval tasks, we still do not know the extent to which poor attentional control prevents a learner from retelling a sentence of average length.As can be seen from the irregularly marked areas in Figure7, the attentional performance of learners with poor attentional control remained largely consistent with that of learners with good attentional control on the first four clips, but the attentional control performance on clips 5–7 showed control differences, while clips 8-9 showed synchronous attentional control characteristics, and clips 10–18 again showed differences, and the range of differences widens. The attentional control pattern remains consistent in clips 19-20 and is again fragmented in clips 21–24. This provides a more in-depth explanation of the attentional control patterns of second language learners in response to bilingual audiovisual stimuli: learners who completed the immediate online task better tended to have more consistent attentional control, maintaining synchronised audiovisual attention. However, learners who were less able to complete the immediate online task struggled to control their attention, but still experienced asynchronous control of their audiovisual attention. They were largely able to maintain good attentional control over the first 4 clips, and after a break in attention, refocused attention could maintain the control pattern for 2 clips.Figure 7
Character clustering under different rules.


## 6. Conclusions

Extended cognition incorporates the environment into cognitive processes, even at the same level as the cognitive functions of the brain, and offers new perspectives for the study of cognitive science. This view overturns the idea that SLA is only an internalisation process. Intelligent language learning based on extended cognition suggests that SLA should follow the principles of environmental indivisibility, adaptability, and synergy, making full use of the variety of intelligent devices and environmental availability. At the same time, language and information are naturally related, intrinsically unified, and inherently symbiotic, and the technical paradigm of second language acquisition embedded in deep learning models needs to be further explored.



---

*Source: 9686725-2022-03-12.xml*