# Implementation of English “Online and Offline” Hybrid Teaching Recommendation Platform Based on Reinforcement Learning

Authors: Danling Dong; Libo Wu
Journal: Security and Communication Networks
 (2021)

---

## Abstract

At present, there is a serious disconnect between online teaching and offline teaching in English MOOC large-scale hybrid teaching recommendation platform, which is mainly due to the problems of cold start and matrix sparsity in the recommendation algorithm, and it is difficult to fully tap the user's interest characteristics because it only considers the user's rating but neglects the user's personalized evaluation. In order to solve the above problems, this paper proposes to use reinforcement learning thought and user evaluation factors to realize the online and offline hybrid English teaching recommendation platform. First, the idea of value function estimation in reinforcement learning is introduced, and the difference between user state value functions is used to replace the previous similarity calculation method, thus alleviating the matrix sparsity problem. The learning rate is used to control the convergence speed of the weight vector in the user state value function to alleviate the cold start problem. Second, by adding the learning of the user evaluation vector to the value function estimation of the state value function, the state value function of the user can be estimated approximately and the discrimination degree of the target user can be reflected. Experimental results show that the proposed recommendation algorithm can effectively alleviate the cold start and matrix sparsity problems existing in the current collaborative filtering recommendation algorithm and can dig deep into the characteristics of users' interests and further improve the accuracy of scoring prediction.


---

## Body

## 1. Introduction

MOOC is the abbreviation of large-scale online courses, which has attracted widespread attention from academic circles for its advantages of large-scale and openness. Large-scale means that there is no limit to the number of people in the course, and a course can be offered to countless people and can be watched repeatedly. Open means that the course has no access conditions and entry threshold, and anyone can use the learning resources free of charge anywhere [1–4]. As an innovation of traditional classroom, MOOC curriculum breaks the monopoly of high-quality learning resources and provides conditions and possibilities for realizing educational equity.The mixed teaching of online teaching and offline teaching under MOOC mode not only plays an important role in stimulating students' learning interest, improving their learning enthusiasm, cultivating students' learning consciousness, and ability of unity and cooperation but also can effectively solve the problem of time allocation between theoretical explanation and practical practice and at the same time strengthen the connection between teachers and students and improve the teaching efficiency. Online teaching includes online video courses, online homework, and online communication and discussion [4–6]. Classroom teaching is composed of three parts: group report, classroom answering, practice, and classroom assessment. At present, this teaching mode has been applied in ideological and political courses and English courses, and the teaching effect has been well received. MOOC mode unifies students' online and offline and brings them into the credit range, which is more stimulating for students' learning.However, there is a serious disconnect between online teaching and offline teaching in English MOOC large-scale mixed teaching recommendation platform, which is because most of the mainstream English MOOC teaching recommendation platforms use collaborative filtering recommendation algorithm to realize personalized recommendation [7–9]. At present, collaborative filtering recommendation algorithm is the best and most widely used algorithm. The algorithm calculates the similarity between target users or objects and predicts the object preference of target users according to the evaluation of objects by users with similar interests. Collaborative filtering recommendation algorithm can be divided into neighborhood-based collaborative filtering recommendation and model-based collaborative filtering recommendation [10–12]. Neighborhood-based collaborative filtering calculates the similarity between target users and user groups to get users with similar interests and hobbies and then predicts the preference of target users for candidate items according to the evaluation status of these users. However, on the one hand, the collaborative filtering algorithm has problems such as cold start and matrix sparsity. On the other hand, the current recommendation algorithms often only consider the user's rating of items but ignore the user's personalized evaluation of items, so it is difficult to fully tap the user's interest characteristics.Based on the research of recommendation algorithms at home and abroad, aiming at the problems existing in current recommendation algorithms, this study improves the traditional collaborative filtering recommendation algorithm in two aspects. The main contributions of this paper are as follows:(1)
To introduce the idea of reinforcement learning, this paper proposes to measure the similarity between users by comparing the state value functions between users instead of the previous similarity calculation method to alleviate the matrix sparsity problem and solve the cold start problem by controlling the convergence speed of the weights in the state value functions.(2)
User evaluation factors are added on the basis of reinforcement learning recommendation algorithm. By adding the learning of user evaluation vector to the value function estimation of state value function, the state value function of users can be estimated approximately, and the discrimination of target users can be reflected. Adding the learning of the user's internal evaluation vector to the weight update can improve the convergence effect of the user's state value function.(3)
Finally, according to the previously mentioned research content, we design and implement the reinforcement learning recommendation algorithm based on user evaluation and compare it with several popular collaborative filtering recommendation algorithms in the prediction error of the recommended content to prove its personalized recommendation effect.The rest of the paper is organized as follows. In Section2, a literature review is studied in detail, while Section 3 provides the detailed methodology. Section 4 provides detailed results and discussion. Finally, the paper is concluded in Section 5.


## 2. Literature Review

At present, many studies have tried to improve collaborative filtering algorithm. For example, Najafabadi et al. [13] proposed an algorithm model of mixed recommendation by adjusting the weights of collaborative filtering algorithm and content-based algorithm, so that the recommendation system can run different recommendation algorithms by adjusting the weights under different conditions. Wu et al. [14] obtained the user's comments on articles by learning the text data and proposed a hybrid recommendation system based on knowledge. Yang et al. [15] put forward an algorithm of connecting two recommendation modes in series. First, collaborative filtering method is used to get the recommended items of target users, and then, the content-based recommendation algorithm is used to model the target users, and the items in the recommended items set that are inconsistent with the user model are eliminated, thus improving the recommendation effect.Reinforcement learning, together with supervised learning and unsupervised learning, is an important branch of machine learning method. Supervised learning expects to predict unlabeled data outside the training set by learning labeled data in the training set, while unsupervised learning expects to discover hidden information in the data and make predictions through the hidden information. Reinforcement learning is also learning data that is not marked in the training set, but it will get a delayed reward signal. The agent of reinforcement learning can adjust itself according to the reward signal in order to expect more rewards.Polvara et al. [16] research results show that the idea of reinforcement learning is to maximize the sum of long-term rewards as the goal, through the independent exploration of the unknown environment, in a certain state and action under the influence of the environment and get the corresponding reward and then get more reward adjustment strategy, ultimately in the process of continuous interaction with the environment learning to the optimal strategy.


## 3. Methodology

### 3.1. Calculated Value Function Model Based on Reinforcement Learning

The running process of the reinforcement learning algorithm is a state-action-reward-new-state transfer sequence process, that is, the agent according to the action taken from a state to obtain environmental feedback reward and transfer to a new state, and the process is repeated until the end of the state. Since the state of the agent at the next moment only depends on the state of the agent at the current moment and the action to be taken, the process has Markov property. The reinforcement learning median function estimation method is an extremely important method; that is, the optimal strategy is obtained through the state value function or the action value function of the agent.In the application environment of recommendation scenarios, what we need to compare in this paper is the similarity of value functions between users, and we pay more attention to the user's own state. Therefore, we use the user's state value function at each moment to approximately estimate the user's current state. A value function estimation method evaluates a value function of a state by using a sum of linear product of a set of features and corresponding weights [17]:(1)Vws=wϕts=∑i=1Mwiϕis,where ϕts represents the feature function of state s, M represents the number of feature functions, and w represents the weight vector corresponding to each feature.The gradient descent method is used to find the optimal solutionw∗, so that the state value function converges to the local optimum. The gradient descent method is one of the most popular methods for solving practical problems.The state value function of the agent at each timet is approximately estimated using the value function estimation idea in the reinforcement learning method, and the weight of each state is adjusted in real time by using the gradient descent method so that the estimation of the value function is more accurate. The reward function R obtained for each step of the agent is taken as a characteristic function of the state. In the actual application scenario of the recommendation algorithm, the user's rating of a certain item at a certain moment can be regarded as a state of the user, and the user's rating vector of the item can be taken as a reward function R, that is, a characteristic function of the user in the state [18]. Thus, the user’s value function is estimated as follows:(2)Vts=wtϕs=∑i=1nwtiRsi,where Vts represents the state value function vector of the user in the state of s at time t, Rs represents a scoring vector of the user on an article, and wt represents a weight corresponding to each feature vector at time t.According to the gradient descent method, the updated formula ofwt can be converted into(3)wt=wt+αri+1+γVsi+1−Vsi∇wVsi.In this formula,α represents the learning rate (generally between 0 and 1), ri+1 represents the immediate reward value obtained by the user at time i + 1 (in the recommended scenario, it represents the user's scoring value of the article at time i + 1), γ represents the time discount factor, Vsi represents the state value function of the user's state s at time i, and ∇wVsi represents the gradient of the user's state value function to the weight w.In reinforcement learning recommendation algorithm, it is through the adjusted value that the cold start problem of collaborative filtering algorithm is alleviated. The key steps of approximate estimation and updating state value function are shown in Figure1.Figure 1
Key steps of approximate estimation and update of state value function.In order to alleviate the cold start and matrix sparsity problems of collaborative filtering algorithm, the idea of reinforcement learning median function estimation is adopted to replace the similarity calculation method of collaborative filtering algorithm by calculating the user's state value function, and the similarity between users is compared according to the user's state value function to make neighbor recommendation. The similarity of state value function vector can be calculated with all users by calculating the similarity of state value function, thus effectively alleviating the problem that it is difficult to calculate the similarity between the target user and other users when the matrix sparsity is high.Because the state value function is a vector value, which is difficult to compare directly, the user with the highest similarity with the target user's state value function is found by cosine similarity calculation. The calculation formula is as follows:(4)simij=v⟶s↔i⋅v⟶s↔jv⟶s↔i⋅v⟶s↔j,where v⟶s↔i and v⟶s↔j represent the final state value functions of user i and user j, respectively.


### 3.2. Evaluation Criteria

Root mean square error (RMSE) and mean absolute percentage error (MAPE) are used to measure the error between the predicted score and the true score of recommended courses [19]:(5)RMSE=1M∑i=1Mpredictioni−reali2,MAPE=100%M∑i=1Mreali−predictionireali,where m represents the total number of recommended courses for users, prediction (i) represents the prediction score of recommended courses i, and real (i) represents the actual score of courses i by users.


### 3.3. Application of User Evaluation in Reinforcement Learning Method

The current recommendation algorithm's strategy on user evaluation application is to build user-user evaluation matrix and user evaluation and item matrix. The user-item matrix is obtained by multiplying the two, and the target user is selected to recommend the item with the highest item score. The principle of traditional user evaluation recommendation algorithm is shown in Figure2.Figure 2
Principle of traditional user evaluation recommendation algorithm.Firstly, the user-user evaluation matrix is constructed:(6)Lm×s=Lu1e1⋯Lu1en⋮⋱⋮Lume1⋯Lumes,where u represents user (1, 2, …, m) and e represents user evaluation (1, 2, …, s). The value of Lumes in the matrix indicate whether the user m has used the user evaluation s (if yes, the value is 1; if no, it is 0). Second, a user evaluation-item matrix is constructed:(7)Ls×n=Le,i1⋯Le1in⋮⋱⋮Lesi1⋯Lesin,where i represents item (1, 2, …, n). The value of Lesin in the matrix indicate whether the item n has been used for user evaluation s (if yes, the value is 1; if no, it is 0). The user-item matrix can be obtained by multiplying two matrices:(8)Lm×n=Lu1i1⋯Lu1in⋮⋱⋮Lum1⋯Lumin.According to the user-item matrix, the item with the highest score corresponding to the target user in the matrix can be selected and recommended to the corresponding target user.However, the algorithm also failed to fully mine the information of user evaluation itself and simply marked 0 and 1 according to the user evaluation. It failed to make full use of the user's likes and dislikes of articles from the user evaluation and failed to divide the user's interests well, which made it difficult to reflect the distinction between the target user and the rest of the users.Therefore, this study proposes to take the user evaluation vector as a part of the user state value function in reinforcement learning to reflect the user interest, which is represented by the user evaluation vector at every moment. Each dimension in the user evaluation vector represents the user's use of the user evaluation, which is not simply represented by 0 and 1 as the matrix multiplication previously mentioned, but represented by the weight of the evaluation used by the user among all users:(9)φuei=1+eu,i∑u=1m1+eu,i,where eu,i represents the number of times that user u uses the i-th dimension of user evaluation vector, and m represents the total number of users. According to formula (8), the frequency of each user evaluation e used by user u in the whole user group can be obtained, which can reflect the discrimination degree of a single user's user evaluation in the whole user group.φu is regarded as a part of the user state value function to reflect the discrimination degree of user u in the user group. On the other hand, in Section 3.1, the weight vector w is updated by gradient descent method, and the learning of user evaluation is also needed in the method of making the user state value function reach the optimum. Different from the use of user evaluation in (9), in order to learn the weight of w better, it is necessary to learn the weight of each user evaluation within the user.(10)wt1,i=wt0,i+αNi∑j=1nNjrt1+γVt1s−Vt0s,where wt1,i represents the i-th dimension user evaluation weight in the user evaluation vector at t1, rt1 represents the user's scoring value for items at t1, Ni represents the i-th dimension user evaluation times in the user evaluation vector, ∑j=1nNj represents the sum of user evaluation times of all dimensions in the user evaluation vector, and n represents the dimension degree of the user evaluation vector.It can be seen that the weight of each user evaluation within the user is added on the basis of formula (3) of gradient descent method in Section 3.1, which can reflect the discrimination between each user evaluation within the user. This way of updating weight w can make the solution process of user state value function pay more attention to the study of user's internal state. Specifically, by using the weight of a single user's evaluation times within the user, it can be distinguished from other users' evaluations within the user.In order to further improve the performance of recommendation algorithm and personalized recommendation ability, the user evaluation vector is added to the user's state value function to approximate the user's current state more accurately:(11)Vt1s=∑i=1nwt1iRsiφsei,where Vt1s represents the state value function vector of the user at t1 time s, Rsi represents the i-th dimension of the scoring vector, φsei represents the i-th dimension of the user evaluation vector as shown in formula (9), and wt1 represents the weight vector corresponding to the feature vector composed of the scoring vector and the user evaluation vector.By adding user evaluation vector to describe the frequency of user evaluation in the whole user group, so as to reflect the discrimination between users and other users, the user's state value function can be estimated approximately more accurately. Adding the learning of users' internal weights can adjust the gradient and make the state value function converge to the optimal value more easily. Finally, the user closest to the target user's state value function is calculated by cosine similarity formula (4), and the favorite items are selected for neighbor recommendation. The specific recommendation process is shown in Figure 3.Figure 3
Flow of reinforcement learning recommendation algorithm based on user evaluation.


### 3.4. Implementation of Hybrid Teaching Recommendation Platform

In MOOC mode, the evaluation of students' learning effect includes the following parts: 20% video learning progress and online quiz, 30% classroom performance (classroom language skills training performance and group report performance), 10% homework submission and quality, and 40% final exam. In addition, in the specific teaching practice, teachers can adjust the proportion of tests according to different class types, such as oral English class, writing class, reading class, and listening class, guide the emphasis of teaching while combining with teaching practice, change the single test method in traditional teaching, give students more opportunities to show their comprehensive ability, explore their potential, and realize new evaluation in online education environment. The operation mode of the hybrid teaching recommendation platform is shown in Figure4.Figure 4
Operation mode of hybrid teaching recommendation platform.


## 3.1. Calculated Value Function Model Based on Reinforcement Learning

The running process of the reinforcement learning algorithm is a state-action-reward-new-state transfer sequence process, that is, the agent according to the action taken from a state to obtain environmental feedback reward and transfer to a new state, and the process is repeated until the end of the state. Since the state of the agent at the next moment only depends on the state of the agent at the current moment and the action to be taken, the process has Markov property. The reinforcement learning median function estimation method is an extremely important method; that is, the optimal strategy is obtained through the state value function or the action value function of the agent.In the application environment of recommendation scenarios, what we need to compare in this paper is the similarity of value functions between users, and we pay more attention to the user's own state. Therefore, we use the user's state value function at each moment to approximately estimate the user's current state. A value function estimation method evaluates a value function of a state by using a sum of linear product of a set of features and corresponding weights [17]:(1)Vws=wϕts=∑i=1Mwiϕis,where ϕts represents the feature function of state s, M represents the number of feature functions, and w represents the weight vector corresponding to each feature.The gradient descent method is used to find the optimal solutionw∗, so that the state value function converges to the local optimum. The gradient descent method is one of the most popular methods for solving practical problems.The state value function of the agent at each timet is approximately estimated using the value function estimation idea in the reinforcement learning method, and the weight of each state is adjusted in real time by using the gradient descent method so that the estimation of the value function is more accurate. The reward function R obtained for each step of the agent is taken as a characteristic function of the state. In the actual application scenario of the recommendation algorithm, the user's rating of a certain item at a certain moment can be regarded as a state of the user, and the user's rating vector of the item can be taken as a reward function R, that is, a characteristic function of the user in the state [18]. Thus, the user’s value function is estimated as follows:(2)Vts=wtϕs=∑i=1nwtiRsi,where Vts represents the state value function vector of the user in the state of s at time t, Rs represents a scoring vector of the user on an article, and wt represents a weight corresponding to each feature vector at time t.According to the gradient descent method, the updated formula ofwt can be converted into(3)wt=wt+αri+1+γVsi+1−Vsi∇wVsi.In this formula,α represents the learning rate (generally between 0 and 1), ri+1 represents the immediate reward value obtained by the user at time i + 1 (in the recommended scenario, it represents the user's scoring value of the article at time i + 1), γ represents the time discount factor, Vsi represents the state value function of the user's state s at time i, and ∇wVsi represents the gradient of the user's state value function to the weight w.In reinforcement learning recommendation algorithm, it is through the adjusted value that the cold start problem of collaborative filtering algorithm is alleviated. The key steps of approximate estimation and updating state value function are shown in Figure1.Figure 1
Key steps of approximate estimation and update of state value function.In order to alleviate the cold start and matrix sparsity problems of collaborative filtering algorithm, the idea of reinforcement learning median function estimation is adopted to replace the similarity calculation method of collaborative filtering algorithm by calculating the user's state value function, and the similarity between users is compared according to the user's state value function to make neighbor recommendation. The similarity of state value function vector can be calculated with all users by calculating the similarity of state value function, thus effectively alleviating the problem that it is difficult to calculate the similarity between the target user and other users when the matrix sparsity is high.Because the state value function is a vector value, which is difficult to compare directly, the user with the highest similarity with the target user's state value function is found by cosine similarity calculation. The calculation formula is as follows:(4)simij=v⟶s↔i⋅v⟶s↔jv⟶s↔i⋅v⟶s↔j,where v⟶s↔i and v⟶s↔j represent the final state value functions of user i and user j, respectively.


## 3.2. Evaluation Criteria

Root mean square error (RMSE) and mean absolute percentage error (MAPE) are used to measure the error between the predicted score and the true score of recommended courses [19]:(5)RMSE=1M∑i=1Mpredictioni−reali2,MAPE=100%M∑i=1Mreali−predictionireali,where m represents the total number of recommended courses for users, prediction (i) represents the prediction score of recommended courses i, and real (i) represents the actual score of courses i by users.


## 3.3. Application of User Evaluation in Reinforcement Learning Method

The current recommendation algorithm's strategy on user evaluation application is to build user-user evaluation matrix and user evaluation and item matrix. The user-item matrix is obtained by multiplying the two, and the target user is selected to recommend the item with the highest item score. The principle of traditional user evaluation recommendation algorithm is shown in Figure2.Figure 2
Principle of traditional user evaluation recommendation algorithm.Firstly, the user-user evaluation matrix is constructed:(6)Lm×s=Lu1e1⋯Lu1en⋮⋱⋮Lume1⋯Lumes,where u represents user (1, 2, …, m) and e represents user evaluation (1, 2, …, s). The value of Lumes in the matrix indicate whether the user m has used the user evaluation s (if yes, the value is 1; if no, it is 0). Second, a user evaluation-item matrix is constructed:(7)Ls×n=Le,i1⋯Le1in⋮⋱⋮Lesi1⋯Lesin,where i represents item (1, 2, …, n). The value of Lesin in the matrix indicate whether the item n has been used for user evaluation s (if yes, the value is 1; if no, it is 0). The user-item matrix can be obtained by multiplying two matrices:(8)Lm×n=Lu1i1⋯Lu1in⋮⋱⋮Lum1⋯Lumin.According to the user-item matrix, the item with the highest score corresponding to the target user in the matrix can be selected and recommended to the corresponding target user.However, the algorithm also failed to fully mine the information of user evaluation itself and simply marked 0 and 1 according to the user evaluation. It failed to make full use of the user's likes and dislikes of articles from the user evaluation and failed to divide the user's interests well, which made it difficult to reflect the distinction between the target user and the rest of the users.Therefore, this study proposes to take the user evaluation vector as a part of the user state value function in reinforcement learning to reflect the user interest, which is represented by the user evaluation vector at every moment. Each dimension in the user evaluation vector represents the user's use of the user evaluation, which is not simply represented by 0 and 1 as the matrix multiplication previously mentioned, but represented by the weight of the evaluation used by the user among all users:(9)φuei=1+eu,i∑u=1m1+eu,i,where eu,i represents the number of times that user u uses the i-th dimension of user evaluation vector, and m represents the total number of users. According to formula (8), the frequency of each user evaluation e used by user u in the whole user group can be obtained, which can reflect the discrimination degree of a single user's user evaluation in the whole user group.φu is regarded as a part of the user state value function to reflect the discrimination degree of user u in the user group. On the other hand, in Section 3.1, the weight vector w is updated by gradient descent method, and the learning of user evaluation is also needed in the method of making the user state value function reach the optimum. Different from the use of user evaluation in (9), in order to learn the weight of w better, it is necessary to learn the weight of each user evaluation within the user.(10)wt1,i=wt0,i+αNi∑j=1nNjrt1+γVt1s−Vt0s,where wt1,i represents the i-th dimension user evaluation weight in the user evaluation vector at t1, rt1 represents the user's scoring value for items at t1, Ni represents the i-th dimension user evaluation times in the user evaluation vector, ∑j=1nNj represents the sum of user evaluation times of all dimensions in the user evaluation vector, and n represents the dimension degree of the user evaluation vector.It can be seen that the weight of each user evaluation within the user is added on the basis of formula (3) of gradient descent method in Section 3.1, which can reflect the discrimination between each user evaluation within the user. This way of updating weight w can make the solution process of user state value function pay more attention to the study of user's internal state. Specifically, by using the weight of a single user's evaluation times within the user, it can be distinguished from other users' evaluations within the user.In order to further improve the performance of recommendation algorithm and personalized recommendation ability, the user evaluation vector is added to the user's state value function to approximate the user's current state more accurately:(11)Vt1s=∑i=1nwt1iRsiφsei,where Vt1s represents the state value function vector of the user at t1 time s, Rsi represents the i-th dimension of the scoring vector, φsei represents the i-th dimension of the user evaluation vector as shown in formula (9), and wt1 represents the weight vector corresponding to the feature vector composed of the scoring vector and the user evaluation vector.By adding user evaluation vector to describe the frequency of user evaluation in the whole user group, so as to reflect the discrimination between users and other users, the user's state value function can be estimated approximately more accurately. Adding the learning of users' internal weights can adjust the gradient and make the state value function converge to the optimal value more easily. Finally, the user closest to the target user's state value function is calculated by cosine similarity formula (4), and the favorite items are selected for neighbor recommendation. The specific recommendation process is shown in Figure 3.Figure 3
Flow of reinforcement learning recommendation algorithm based on user evaluation.


## 3.4. Implementation of Hybrid Teaching Recommendation Platform

In MOOC mode, the evaluation of students' learning effect includes the following parts: 20% video learning progress and online quiz, 30% classroom performance (classroom language skills training performance and group report performance), 10% homework submission and quality, and 40% final exam. In addition, in the specific teaching practice, teachers can adjust the proportion of tests according to different class types, such as oral English class, writing class, reading class, and listening class, guide the emphasis of teaching while combining with teaching practice, change the single test method in traditional teaching, give students more opportunities to show their comprehensive ability, explore their potential, and realize new evaluation in online education environment. The operation mode of the hybrid teaching recommendation platform is shown in Figure4.Figure 4
Operation mode of hybrid teaching recommendation platform.


## 4. Experimental Results and Analysis

### 4.1. Experimental Data and Pretreatment

The experiment is carried out on the MOOC learning platform of Chinese universities, and the simulation tool is MATLAB7.8. The English curriculum data files of 10 schools are randomly selected on the learning platform and preprocessed. After data preprocessing, it includes five fields: user id, course id, user rating (0 to 5 points, increasing according to half points), user evaluation (filtered from tags), and time when users submit rating evaluation. The characteristics of 10 datasets after pretreatment in this experiment are shown in Table1.Table 1
Experimental dataset.
Dataset nameNumber of usersNumber of coursesAverage number of user ratingsAverage number of user evaluationsNumber of dataSparsity (%)A316623.7124.685251097.74B467420.2122.388526098.70C354321.0625.324506096.36D398622.8424.365571097.06E539426.1424.026128096.02F610624.0225.2811506097.82G496723.6823.354537097.11H503822.1022.065031097.92I469525.7125.344958096.35J385423.1624.619113097.24It can be seen from Table1 that the number of user evaluations is slightly more than the number of user ratings, because users often only score courses once, but the user evaluations made may contain many words that reflect the likes and dislikes of users, and these words can be used as one dimension of the user evaluation vector.


### 4.2. Comparison and Analysis of Experimental Results

Root mean square error (RMSE) and mean absolute percentage error (MAPE) are used to jointly evaluate the deviation between the predicted score and the true score of recommended courses. First, according to RMSE, the specific values of time discount factorγ and learning rate α in each dataset are determined. After determining the optimal values, the performance of the proposed algorithm is compared with several popular collaborative filtering recommendation algorithms to fully verify its effectiveness in improving the accuracy of personalized recommendation of users.Because time discount factorγ and learning rate α directly affect the convergence effect of weight w, these two parameters need to be slightly adjusted on different scale datasets to make w achieve better convergence effect. In order to make the estimated value of the next state value function better approximate to the true value of the current state value function, the value of γ is between 0.5 and 1.0, and the value of α is between 0.1 and 1.0. The best combination value of the two is determined by the recommended performance.For dataset A, the influence of combination value ofα and γ on the recommended algorithm is shown in Table 2.Table 2
Influence of combination value ofα and γ on recommendation algorithm in dataset A.
Γ0.50.60.70.80.91.0α0.10.91730.91030.90720.90490.90510.90890.20.91480.90870.90270.90030.90130.90430.30.91160.90440.89830.89710.89770.90180.40.91080.90190.89610.89420.8950.90040.50.90840.89830.89340.89150.89290.89710.60.90450.89520.89030.88730.88970.89330.70.90270.89140.88710.88570.88630.89160.80.90120.88960.88470.88380.88510.88730.90.90410.89030.88640.88470.8870.891For dataset B, the influence of combination value ofα and γ on the recommended algorithm is shown in Table 3.Table 3
Influence of combination value ofα and γ on recommendation algorithm in dataset B.
γ0.50.60.70.80.91.0α0.10.89340.8920.89120.89130.89230.89730.20.89170.89030.88840.89060.8910.89240.30.88940.88740.88580.8880.88930.89010.40.88720.88590.88210.88470.88640.88790.50.88690.88150.88030.88260.88420.88540.60.8850.87930.87770.87990.88310.88340.70.88310.87720.87540.87820.87930.88270.80.88420.87860.8760.87930.88140.88390.90.88780.88230.88090.88160.8840.885Taking datasets A and B as examples, the influence surfaces ofα and γ combination values on the recommendation algorithm are shown in Figure 5.Figure 5
Influence surface of combination value ofα and γ on recommendation algorithm. (a) Dataset A. (b) Dataset B.
(a)(b)It can be seen from Figure5 that, for dataset A, when α = 0.8 and γ = 0.8, the accuracy of the reinforcement learning recommendation algorithm based on user evaluation reaches the best. For dataset B, the accuracy is optimal when α = 0.7 and γ = 0.7. According to the results, when the dataset is small, α should be increased to converge to the optimum faster. When the dataset is large, α should be appropriately reduced to better control the law of convergence effect.After determining the optimal combination ofα and γ on datasets A and B, the proposed algorithm is compared with user-based collaborative filtering algorithm [20], item-based collaborative filtering algorithm [21], user-based K-nearest neighbor recommendation algorithm [22], and trust-based matrix decomposition recommendation algorithm [23, 24], respectively. The comparison results of RMSE are shown in Figure 6.Figure 6
RMSE comparison of each recommended algorithm. (a) Dataset A. (b) Dataset B.
(a)(b)It can be seen from Figure6 that the RMSE of the proposed recommendation algorithm between the predicted score and the true score of the recommended course on datasets A and B is the smallest, indicating that the absolute error between the predicted score and the true value of the recommended course is the smallest in the reinforcement learning recommendation algorithm based on user evaluation.Then, we compare MAPE [25–28] with the previously mentioned four popular collaborative filtering algorithms on two datasets A and B, so as to investigate the accuracy of each algorithm in prediction score more comprehensively. The experimental results are shown in Figure 7.Figure 7
MAPE comparison of recommended algorithms. (a) Dataset A. (b) Dataset B.
(a)(b)It can be seen from Figure7 that the MAPE of the reinforcement learning recommendation algorithm model based on user evaluation is the smallest between the predicted score and the true score of the recommended course through the divided data of different scales on the two datasets A and B. Compared with other algorithms, the proposed algorithm not only performs well in each sparse matrix but also has great advantages (no matter absolute error or relative error) in datasets with small data scale.


## 4.1. Experimental Data and Pretreatment

The experiment is carried out on the MOOC learning platform of Chinese universities, and the simulation tool is MATLAB7.8. The English curriculum data files of 10 schools are randomly selected on the learning platform and preprocessed. After data preprocessing, it includes five fields: user id, course id, user rating (0 to 5 points, increasing according to half points), user evaluation (filtered from tags), and time when users submit rating evaluation. The characteristics of 10 datasets after pretreatment in this experiment are shown in Table1.Table 1
Experimental dataset.
Dataset nameNumber of usersNumber of coursesAverage number of user ratingsAverage number of user evaluationsNumber of dataSparsity (%)A316623.7124.685251097.74B467420.2122.388526098.70C354321.0625.324506096.36D398622.8424.365571097.06E539426.1424.026128096.02F610624.0225.2811506097.82G496723.6823.354537097.11H503822.1022.065031097.92I469525.7125.344958096.35J385423.1624.619113097.24It can be seen from Table1 that the number of user evaluations is slightly more than the number of user ratings, because users often only score courses once, but the user evaluations made may contain many words that reflect the likes and dislikes of users, and these words can be used as one dimension of the user evaluation vector.


## 4.2. Comparison and Analysis of Experimental Results

Root mean square error (RMSE) and mean absolute percentage error (MAPE) are used to jointly evaluate the deviation between the predicted score and the true score of recommended courses. First, according to RMSE, the specific values of time discount factorγ and learning rate α in each dataset are determined. After determining the optimal values, the performance of the proposed algorithm is compared with several popular collaborative filtering recommendation algorithms to fully verify its effectiveness in improving the accuracy of personalized recommendation of users.Because time discount factorγ and learning rate α directly affect the convergence effect of weight w, these two parameters need to be slightly adjusted on different scale datasets to make w achieve better convergence effect. In order to make the estimated value of the next state value function better approximate to the true value of the current state value function, the value of γ is between 0.5 and 1.0, and the value of α is between 0.1 and 1.0. The best combination value of the two is determined by the recommended performance.For dataset A, the influence of combination value ofα and γ on the recommended algorithm is shown in Table 2.Table 2
Influence of combination value ofα and γ on recommendation algorithm in dataset A.
Γ0.50.60.70.80.91.0α0.10.91730.91030.90720.90490.90510.90890.20.91480.90870.90270.90030.90130.90430.30.91160.90440.89830.89710.89770.90180.40.91080.90190.89610.89420.8950.90040.50.90840.89830.89340.89150.89290.89710.60.90450.89520.89030.88730.88970.89330.70.90270.89140.88710.88570.88630.89160.80.90120.88960.88470.88380.88510.88730.90.90410.89030.88640.88470.8870.891For dataset B, the influence of combination value ofα and γ on the recommended algorithm is shown in Table 3.Table 3
Influence of combination value ofα and γ on recommendation algorithm in dataset B.
γ0.50.60.70.80.91.0α0.10.89340.8920.89120.89130.89230.89730.20.89170.89030.88840.89060.8910.89240.30.88940.88740.88580.8880.88930.89010.40.88720.88590.88210.88470.88640.88790.50.88690.88150.88030.88260.88420.88540.60.8850.87930.87770.87990.88310.88340.70.88310.87720.87540.87820.87930.88270.80.88420.87860.8760.87930.88140.88390.90.88780.88230.88090.88160.8840.885Taking datasets A and B as examples, the influence surfaces ofα and γ combination values on the recommendation algorithm are shown in Figure 5.Figure 5
Influence surface of combination value ofα and γ on recommendation algorithm. (a) Dataset A. (b) Dataset B.
(a)(b)It can be seen from Figure5 that, for dataset A, when α = 0.8 and γ = 0.8, the accuracy of the reinforcement learning recommendation algorithm based on user evaluation reaches the best. For dataset B, the accuracy is optimal when α = 0.7 and γ = 0.7. According to the results, when the dataset is small, α should be increased to converge to the optimum faster. When the dataset is large, α should be appropriately reduced to better control the law of convergence effect.After determining the optimal combination ofα and γ on datasets A and B, the proposed algorithm is compared with user-based collaborative filtering algorithm [20], item-based collaborative filtering algorithm [21], user-based K-nearest neighbor recommendation algorithm [22], and trust-based matrix decomposition recommendation algorithm [23, 24], respectively. The comparison results of RMSE are shown in Figure 6.Figure 6
RMSE comparison of each recommended algorithm. (a) Dataset A. (b) Dataset B.
(a)(b)It can be seen from Figure6 that the RMSE of the proposed recommendation algorithm between the predicted score and the true score of the recommended course on datasets A and B is the smallest, indicating that the absolute error between the predicted score and the true value of the recommended course is the smallest in the reinforcement learning recommendation algorithm based on user evaluation.Then, we compare MAPE [25–28] with the previously mentioned four popular collaborative filtering algorithms on two datasets A and B, so as to investigate the accuracy of each algorithm in prediction score more comprehensively. The experimental results are shown in Figure 7.Figure 7
MAPE comparison of recommended algorithms. (a) Dataset A. (b) Dataset B.
(a)(b)It can be seen from Figure7 that the MAPE of the reinforcement learning recommendation algorithm model based on user evaluation is the smallest between the predicted score and the true score of the recommended course through the divided data of different scales on the two datasets A and B. Compared with other algorithms, the proposed algorithm not only performs well in each sparse matrix but also has great advantages (no matter absolute error or relative error) in datasets with small data scale.


## 5. Conclusion

In this paper, the idea of value function estimation in reinforcement learning is introduced, and the similarity between users is evaluated through the state value function between users (which is different from the traditional similarity calculation method), thus effectively alleviating the matrix sparsity problem and alleviating the cold start problem by controlling the convergence speed of the control value. At the same time, the user evaluation vector is introduced, and the user's interests and hobbies are mined by the proportion of user evaluation in the user group and within the user, so as to improve the personalized recommendation effect. Experiments on the English MOOC hybrid teaching recommendation platform show that the error of the reinforcement learning recommendation algorithm based on user evaluation in predicting the score of courses is smaller than other existing collaborative filtering recommendation algorithms in all datasets, thus achieving better recommendation performance.



---

*Source: 4875330-2021-09-30.xml*