# Memory-Enhanced Knowledge Reasoning with Reinforcement Learning
## 3 Reinforcement Learning Framework Algorithm Based on ConvR Embedding
### ConvR Embedding Model  
The ConvE model is insufficient to comprehensively capture the interactions between input entities and relations, as it models interactions in the region of matrix adjacency of input entities and relations. In order to maximize the interaction between entities and relations, [PERSON] [37] et al. proposed the ConvR model, which continues the framework of ConvE. The difference is that the relation is used as the convolution kernel. The essence lies  
Figure 11: Output gate process.  
Figure 10: Memory merge process.  
in adaptively building the relation embedding into a convolution kernel and interacting with the entity embedding so that each convolution is 100% interactive. The overall model is shown in Figure 12.  
Firstly, after being given a triplet, the relational representation should be split and reshaped into a set of filters. Next, the relational representation of the head entity should be reshaped and used as the input for the convolutional layer. Then, the filter should be applied to the input to capture interactions between filters (part of the relational representation) and different regions of the input (entity representation). Finally, the convolutional features are projected and matched to the representation of the tail entity. In comparison to ConvE, which utilizes a global filter, ConvR employs an adaptive filter that is constructed from a relational representation and uses the relation as a convolution kernel.  
In order to use the relation as the convolution kernel, the relation embedding is first divided into several small blocks \(r^{1}\ldots r^{n}\) of the same size. This process is called \"Split\". At the same time, because 2D convolution is used, each small block \(r^{\prime}\in\mathbb{R}^{d_{r}/c}\), is reshaped into a height \(h\), width \(w\), and convolution kernel \(R^{\prime}\in\mathbb{R}^{h\times w}\), \(d_{r}=clw\).  
During convolution, the relational specialized convolution kernel \(R^{\prime}\) is used for operations, and the mathematical expression of the convolution process is as follows:  
\[c^{\prime}_{m,n}=f\left(\sum\limits_{i,j}s_{m+i-1,n+j-1}\times r^{\prime}_{i, j}\right) \tag{10}\]  
where \(c^{\prime}_{m,n}\) represents the response of the l-th filter at position \((m,n)\) on the convolved feature map. The function \(f\) is an activation function, such as ReLU, etc. \(s_{m+i-1,n+j-1}\) is a part of the input entity embedding, and \(r^{\prime}_{i,j}\) is the value at position \((i,j)\) in the l-th filter. The entire summation process is the convolution operation, which calculates the dot product of the filter with the entity embedding at each position.  
For instance, when training on the FB15K-237 dataset, the data that participate in the convolution primarily include entity embeddings and relation embeddings. Entity embeddings are representations of entities obtained through pre-training of the model, while relation embeddings are a series of convolutional filters obtained through splitting and reshaping the relational representations. These embeddings are updated along with the model parameters during the training process to ultimately capture the complex interactions between entities.