## 3 Methodology
### Teaching Path Planning with DRL  
In this section, we use DRL to deal with the teaching path planning problem. First of all, based on the commonality between the student's learning process and the student model, we define the entire class teaching process as a POMDP. We formulate this process through the four following points. Combined with the specific model, the parameters are defined as follows:  
* State Space: State Space \(S\) is all the possible states of all students in the class, that is, the possibilities for mastering a knowledge point. The difference is that for the EFC model, there are knowledge point difficulties, the times of learning, and the time after the last learning of each knowledge point. Contrary to the EFC model, the HLR model has no knowledge point difficulties and has a feature matrix. The feature matrix contains the times of the learned knowledge points, the times that knowledge points can be memorized, and the times that knowledge points cannot be recalled. * Observation Space: Because the teaching process is a POMDP, the real state cannot be obtained, and there is only one observation result, that is, the partial state. All possible states of this part constitute the Observation Space \(O\), which is specifically all current real-time feedback, that is, whether the knowledge point is mastered. In actual teaching, real students obtain results by answering questions, etc. The model tests the samples through the following formula: \[O\sim Bernoulli(S)\] (20)
* Action Space: The action space \(A\) is a collection of actions that can be acted upon, that is, knowledge points that teachers can choose to teach. * Transfer Function: The transfer function refers to the process of transforming state \(S^{t+1}\) into state \(S^{t}\) through action \(A\). Specifically, after the teacher teaches, the class as whole changes state. In the EFC model, it changes the times of learning the knowledge point and the interval time between two times of learning. In the HLR model, it is similar to EFC, except that the feature matrix \(\theta\) is additionally changed. The BKT model completes the state transition through Formula (9). Both Long Short-Term Memory (LSTM) and Gate Recurrent Unit (GRU) are part of the most widespread recurrent neural networks today, and are both capable of learning long-distance dependencies. Through comparative experiments, the performance of the two is similar, but the LSTM training process is more stable in this environment. Therefore, as a continuous process, we use the PPO algorithm based on the LSTM neural network structure to solve the POMDP problem. Specifically, we use the ready-made implementation of the open source framework garage. We identify the two following points combining the above exact model and four constraints:  
* Reward Function: We define the player's reward as the expected value of the test. The formula is as follows: \[grade=\sum_{i=0}^{K}P_{i}\] (21) In constraint 3, it can be seen that the overall reward is composed of three parts, as shown in the following formula: \[Reward=a_{0}R_{p}+a_{1}R_{e}+a_{2}R_{a}\] (22) \[R_{p}=min(m\cdot p,num(pass))\] (23) \[R_{p}=min(m\cdot e,num(excellent))\] (24) \[R_{a}=\sum_{i=0}^{m}grade_{i}/m\] (25)where \(R_{p}\) represents the pass rate reward, \(R_{e}\) represents the excellent rate reward, \(R_{a}\) represents the average score reward, \(num(pass)\) represents the number of students who passed the test, and \(num(excellent)\) represents the number of students with excellent test scores. We consider the logarithmic reward, for which the formula is as follows: \[Reward=log(a_{0}R_{p}+a_{1}R_{e}+a_{2}R_{a})\] (26)
* Discount factor: The discount factor determines whether to pay more attention to current returns or long-term returns. In the teaching process, we usually pay more attention to the teaching income at any time, so a large discount factor is better. The observation space, the sequence of knowledge points learned, and the current time together constitute the input of the LSTM.