# Memory-Enhanced Knowledge Reasoning with Reinforcement Learning
## 3 Reinforcement Learning Framework Algorithm Based on ConvR Embedding
### Slstm  
The core idea of the LSTM model is to use the \"cell state\" \(\mathsf{C}_{\mathsf{f}}\). The \"cell state\" acts like a conveyor belt that runs directly throughout the entire chain, with only a small amount of linear interaction. Information flowing through it can easily be kept unchanged. LSTM removes or adds information to the cell state using a carefully designed structure called \"gates\". A gate is a way of selectively allowing information to pass through, and it consists of a sigmoid neural network layer and a pointwise multiplication operation. Three gate functions are introduced in LSTM: the input gate, forget gate, and output gate, which are used to control input values, memory values, and output values, respectively. However, the LSTM model may face the problem of long-term dependencies when processing long sequence data, where earlier inputs have less impact on subsequent predictions. This is because the memory unit of the LSTM model may gradually decay as the time step increases, causing it to gradually forget earlier input information. This may lead to a decrease in the performance of the LSTM model in long sequence data. Moreover, as the amount of data increases and the network deepens, there is a risk of gradient vanishing. Therefore, in our work, we propose an improved LSTM algorithm--the SELU-activated  
Figure 6: ConvR-based Reinforcement Learning Knowledge Graph Reasoning with Strong Memory framework. The blue of \(e_{s}\) represents the entity input at each step, the yellow of \(r_{0}\) represents the query relationship input at each step, the green of \(h_{0}\) represents the historical memory recorded in the path of each step, the pink of self-attention represents the self-attention module, and in the strategy \(\pi\), green and purple represent the \(\theta\) updates of the parameters. LSTM (SLSTM) algorithm--to address the problems of long-term dependencies, gradient vanishing, and exploding. The model is shown in Figure 5.  
Based on the latest research results on activation functions, the improved LSTM model makes the state \(h_{t}\) and \(c_{t}\) enter an activation function during the gate computation process, and the output end also needs to go through an activation function. The activation function selected is the SELU function [35], which ensures that the gradient of each layer is transmitted in a stable manner during the optimization process, and the SELU function has the function of automatically converging the activation values of neurons to zero mean and unit variance, achieving the purpose of self-normalization. In addition, the normalization of SELU restricts the range of variance and also ensures robustness in multi-level training, effectively avoiding the problem of gradient vanishing and exploding during the training process. The function is presented in Eq. (1) as follows:  
\[selu(x)=\lambda \begin{cases}x\ if\ x>0\\ \alpha(e^{x}-1)\ if\ x<0\end{cases}  \tag{1}\]  
where \(x\) refers to the value entered into the activation function; both \(\lambda\) and \(\alpha\) are hyper-parameters, and their values are obtained by proof rather than training. \(\lambda\) in formula 1 is the scaling coefficient, \(\alpha\) is the scaling parameter, and the values of \(\lambda\) and \(\alpha\) are derived mathematically, where \(\lambda\) is usually chosen as 1.0507 and \(\alpha\) is usually chosen as 1.6733. Then, the input gate is removed and combined with the forget gate, and the addition of new information and the retention of the old state are set as complementary values (with a sum of 1). That is, we only forget when new information needs to be added, and we only add new information when forgetting is necessary. For knowledge reasoning, in order to ensure the accuracy of reasoning, long-term effective memory is required. Therefore, combined with the \"peephole connections\" proposed by [PERSON] and [PERSON], 2000 [36], the hidden gate is connected to the state of the previous unit, and the output gate is connected to the current unit. Status connection is established.