# Recommendation of Knowledge Graph Convolutional Networks Based on Multilayer BiLSTM and Self-Attention

Authors: Yao Qiu; Yajie Liu; Ying Tong; Xuyu Xiang
Journal: Mobile Information Systems
 (2022)

---

## Abstract

To solve the problems of cold start, sparse data, and poor recommendation performance in collaborative filtering recommendation, an end-to-end framework algorithm based on BiLSTM and BAGCN was proposed. In order to discover the higher-order structural information in the knowledge graph, stacked BiLSTM is used to extract the features of embedded entities and relationships, respectively, and the depth dependence features of user-item interaction matrix are mined. The neighborhood representation of each entity is then calculated by sampling adjacent entities of a fixed size. Then, the self-attention mechanism is used to learn the semantic association between entities and neighboring entities to obtain the final neighborhood information. Aggregators are used to combine neighborhood information and bias information when computing node representations. By extending the sampling of adjacent entities to multihop simulation of higher-order adjacent information, users’ potential long-distance interests can be captured. Compared with the baseline model, the superiority of this method is verified.


---

## Body

## 1. Introduction

With the explosive growth of Internet information, users are faced with the problem of information overload [1], and traditional search engines have been unable to meet users’ retrieval needs, so the recommendation system have emerged as the times require.Traditional methods, such as collaborative filtering (CF) [2], exploit the entire user-item interaction matrix to mine users’ interest, which suffer from cold start and data sparsity problems. The matrix factorization (MF) model [3] added the concept of latent vector to the existing matrix, strengthening the model’s ability to deal with sparse data. Koren [4] put forward SVD, which transforms both items and users into the same hidden factor space. The space tries to explain the rating by describing the items and users by the factors that are automatically inferred from user feedback. He et al. [5] proposed the matrix factorization model NCF based on the neural network structure, which takes into account both the user’s explicit rating and implicit feedback on the item. Followed by this, Guo et al. [6] proposed DeepFM to explicitly describe user preferences for different factors. The above methods model each pair of user-item interaction data as an independent data instance and do not consider the association between them, so attribute-based collaboration information cannot be extracted from user behavior.The graph structure can well describe the degree of association between data, and knowledge graph (KG) is a natural graph data that contains a lot of heterogeneous information [7]. The essence of KG is a large-scale semantic network, which contains rich semantic features among items. It can help to discover users’ potential interest as auxiliary information for recommender systems. At the same time, the data with semantic correlation can make the recommendation results interpretable while learning.The current mainstream knowledge graph-based recommendation can roughly be divided into two categories: path-based methods and embedding-based and joint-based methods. Embedding-based methods all use the knowledge graph embedding (KGE) method [8] to map entity vectors or relation vectors to a low-dimensional vector space, such as CKE proposed by Zhang et al. [9] and DKN proposed by Wang et al. [10]. However, this method ignores the connectivity of information in KG and lacks interpretability.Path-based methods use the user-item graph to calculate the path similarity of users or items by predefining meta-paths and use semantic connectivity in KG for recommendations. The Hete-MF, proposed by Yu et al. [11], utilized L different metapaths to find the similarity between items on each path. Luo et al. [12] proposed Hete-CF, which uses the similarity between users and items together as a regularization to find items of interest for users. The PER algorithm treats KG as a heterogeneous information network and extracts latent features from metapaths to represent the connectivity in different relational paths between users and items. However, the type and number of meta-paths of such methods need to be manually defined, and the performance is easily affected [13].The joint method combines the above two methods, which not only makes full use of the semantic information in KG but also inherits the interpretability of the path-based method. For example, RippleNet [14] used the embedding-based propagation idea to model users by analogizing the propagation process of user preferences to ripple diffusion. Subsequently, Wang et al. [15] proposed the KGCN model, which characterized nodes by mining the association attributes between entities on the KG, capturing the correlation between items and aggregating the information of neighbor nodes. However, this scheme is greatly affected by sparse data, and the representation of nodes is not accurate enough, so the performance of model prediction needs to be further improved.Therefore, in order to solve the above problems, this paper proposes a recommendation algorithm for knowledge graph convolutional networks based on multilayer BiLSTM and self-attention mechanism. Long and short-term neural network is a kind of recurrent neural network, while bidirectional short and long-term memory network is divided into two independent long and short-term neural network. First, we apply the stacked multilayer BiLSTM to perform feature extraction on the initial entity and relationship vectors and then combine the learned entity feature vector with the initial vector to obtain the entity’s own vector, that is, the 0-order representation. Finally, we realize high-order information representation through information transfer. The influence of data sparseness on the model is slowed down to a certain extent by adding its own information. Meanwhile, the self-attention mechanism can further learn the relationship between entities and adjacent entities, making the representation of nodes more accurate and effectively improving the performance of the recommendation system.


## 2. The Recommendation Algorithm Based on Multilayer BiLSTM and Self-Attention Mechanism

The overall framework of our method is shown in Figure1, which is divided into a knowledge graph embedding layer, a neighborhood information calculation layer, and a prediction layer. L indicates that the neighborhood information is extended to the first, second, third, or higher order.Figure 1
Overall framework of BAGCN.


### 2.1. Problem Description

In a recommendation scenario, a user set and an item set are given, which is denoted asU=u1,u2,…,um and I=i1,i2,…,in. The interaction matrix of users and items is Y∈Rm×n, interaction data is defined as u,yui,i, and m and n are the number of users and items, respectively. yui=1 indicates that there is interaction between the user and the item, which is regarded as a positive example of user-item interaction; otherwise, it indicates that no interaction has occurred, which is regarded as a negative example. Given a knowledge graph G and a user-item interaction matrix Y, we need to predict the probability that a user u will interact with an item i without interacting before. The learning objective function can be expressed as y˜ui=Fu,i;θ, where y˜ui represents the probability that the user will interact with the item and θ represents all model parameters of the function.


### 2.2. BAGCN Layer

The BAGCN algorithm is widely used in accurately capturing the higher-order structural proximity between entities in knowledge graphs and the semantic association of relations between entities and adjacent entities. The self-attention mechanism is actually a kind of network attention mechanism that is actually a kind of network configuration [16, 17].


#### 2.2.1. Knowledge Graph Embedding

Knowledge graph embedding is an effective way to convert entities and relations into vector representations, which can transfer high-dimensional sparse features into low-dimensional feature vectors, to obtain a more convenient form for model input. The user,U=u1,u2,…,um, obtains the corresponding vector representation by querying the user embedding matrix I∈Rm×T, where T is the dimension of the embedding vector. Given a candidate pair of user u and item (entity) v, the entity vector and the relation vector get the corresponding initial embeddings v^0 and r^ in the entity embedding matrix and the relation embedding matrix through embedding lookup.In this paper, we apply the multilayer BiLSTM to learn features for entity and relation embeddings. The single-layer BiLSTM has achieved good results in simple prediction tasks, but the potential interest of users are often deeply hidden in the user-item interaction matrix. Therefore, this paper proposes to use the stacked multilayer time series network BiLSTM to extract the deep dependency features in the user and item interaction matrix to mine the potential interest of users. A single-layer BiLSTM [18] is composed of a forward LSTM and a backward LSTM, the former calculates the hidden layer state h←h←1,h←2,…,h←t and the latter calculates the hidden layer state hh1,h2,…,ht, which can be spliced to obtain the final hidden layer state. The multilayer BiLSTM network is composed of the forward multilayer LSTM and backward multilayer LSTM, and the input of the Nth layer is the output of the N−1th layer. The model structure can be seen in Figure 2.Figure 2
BiLSTM structure with multiple layers stacked.First, takev^0 and r^ as the input of the multilayer BiLSTM model for feature extraction and then splice the extracted entity embedding vector with the initial embedding vector v^0 to obtain the 0th-order information v0 of the entity, that is, the representation vector of the entity itself. Its process can be formulated as(1)Xn=BXn−1,Xn−1←∪<Xn−1c1,Xn−1c2,…,Xn−1cm>,Rn=BXn−1∪BXn−2,Xn−1⟵∪<Xn−1c1,Xn−1c2,…,Xn−1cm>,v0=v^0+Xn,where Xn and Rn are the embedded representations of entities and relations obtained after feature extraction, respectively. C1,C2,…Cm represents hidden layer nodes, and the number of internal hidden layer nodes is determined by the dimensions of entities and relationships. ⨄ indicates the feature stacking symbol.Since the input of BiLSTM contains the output of forward and backward LSTM, we sum and average the forward and backward features to effectively utilize the forward and backward feature information, which can be formulated as(2)Xn−1c1=Xn−1Forward1+Xn−1Backed−Output11≤c≤NH,where H is the number of hidden layer nodes and N is the total number of cells. Xn−1Forward1 represents the forward LSTM output vector and Xn−1Backed−Output1 represents the backward LSTM output vector.


#### 2.2.2. Neighborhood Information Calculation

The aggregation calculation of neighborhood information needs to integrate the information of neighboring nodes. Specifically,Nv represents the set of neighbor nodes directly connected to entity v, rx,y indicates the relationship between ex and ey, and the neighborhood of entity v is represented as vP(v)u. When calculating the information representation of the neighborhood, the inner product function z:Rd×Rd⟶R is used to calculate the weight matrix between the user vector and the adjacency vector.(3)A=zu,r,where d represents the dimension and A represents the impact of different relationships on users by adding user information. In order to characterize the neighborhood structure of entity v, first calculate the linear combination of neighborhood as(4)vNvu=∑eεPVA˜e,where A˜ is the normalized matrix of A.(5)A˜=expA∑eεPvexpA,where e is the vector representation of the neighbor entity.


#### 2.2.3. Self-Attention

Attention mechanism [19, 20] was initially applied in the field of machine translation, and now, it has been widely applied in image processing, recommendation system, and other aspects. This method draws on the mechanism of human visual selective attention, and its core purpose is to screen out the important content of the current task from a large amount of information. The self-attention mechanism proposed in this paper consists of a gated recurrent unit (GRU) module and a self-attention module. The calculated neighborhood representation v^P(v)u is taken as the input of self-attention, and the aggregation features of lower-dimensional neighborhood information are extracted through the GRU module. Input the obtained feature vector into the self-attention module to further learn the semantic association between entities and adjacent entities and more accurately calculate the representation of neighborhood information. The specific implementation process is as follows:Firstly,vPvu is sent to the GRU module as the input feature to extract the low-dimensional feature, and the output feature I is obtained through the learning of function h.(6)I=hvPvu|u,r.Then, the function bodyAtten is created, which is consistent with the implementation process of ordinary self-attention mechanism. The feature vector I processed by the GRU module is used as the input of the self-attention module, and the specific operation is shown in the following formula:(7)v=tanhI•Wa+b,v′=v•u,a=softmaxy′,O=I∗a,where • is the tensor-dot operation, wa is the trainable transformation matrix, b and u are vectors for training, α is the normalized weight, and O is the final output feature of self-attention.Finally, a full connection layer is used to obtain the final neighborhood information representationv^Pvu, and the specific calculation process is as follows:(8)v^Pvu=FCO,where FC is the operation function of the full connection layer.


#### 2.2.4. Information Aggregation

The last step of the BAGCN layer is information aggregation. The model aggregates the node’s own informationv0 and all its neighborhood information v^J(v)u as the final representation of the node through the aggregator, takes the sum of the two vectors, and then performs nonlinear transformation on them. Here, Jv=K, K is a configurable constant, representing the number of neighbors sampled. Technically, if Jv<K, this paper samples with a put back; otherwise, it randomly samples fixed K neighbor nodes. In the real knowledge graph, there are great differences in the number of neighbor nodes of different entities. In this paper, the complete set of neighbor nodes is not included in the calculation, but a fixed size neighborhood set is sampled for each entity to ensure the same calculation mode and calculation efficiency of each batch. Its calculation is as follows:(9)agg=σW⋅v0+v^Jvu+b,where W and b are transformation weights and bias terms, respectively, and σ is a nonlinear function.The initial embedding representation of an entity is zero-order, and it is propagated to its neighbors to obtain first-order representation. The process is repeated from first-order representation to higher-order representation. The high-order representation can tap deeper potential interest of users.


### 2.3. Prediction Layer

The prediction layer predicts the probability of interaction itemi of user u by inputting a function f:Rd×Rd⟶R, as shown in the following formula:(10)y^u,i=fu,i.


### 2.4. Loss Function

The cross-entropy loss function is used to train the model in this paper, and the specific formula is as follows:(11)loss=∑uεU∑i:yui=1Jyui,y^ui−∑n=1TuEin∼QvnJyuin,y^uin+λF22,where J is the cross-entropy loss, Q is the negative sampling distribution, and Tu is the negative sample number of user u. In this paper, Tu=v:yuv=1, Q follows uniform distribution and the last term is L2 regularization.


## 2.1. Problem Description

In a recommendation scenario, a user set and an item set are given, which is denoted asU=u1,u2,…,um and I=i1,i2,…,in. The interaction matrix of users and items is Y∈Rm×n, interaction data is defined as u,yui,i, and m and n are the number of users and items, respectively. yui=1 indicates that there is interaction between the user and the item, which is regarded as a positive example of user-item interaction; otherwise, it indicates that no interaction has occurred, which is regarded as a negative example. Given a knowledge graph G and a user-item interaction matrix Y, we need to predict the probability that a user u will interact with an item i without interacting before. The learning objective function can be expressed as y˜ui=Fu,i;θ, where y˜ui represents the probability that the user will interact with the item and θ represents all model parameters of the function.


## 2.2. BAGCN Layer

The BAGCN algorithm is widely used in accurately capturing the higher-order structural proximity between entities in knowledge graphs and the semantic association of relations between entities and adjacent entities. The self-attention mechanism is actually a kind of network attention mechanism that is actually a kind of network configuration [16, 17].


### 2.2.1. Knowledge Graph Embedding

Knowledge graph embedding is an effective way to convert entities and relations into vector representations, which can transfer high-dimensional sparse features into low-dimensional feature vectors, to obtain a more convenient form for model input. The user,U=u1,u2,…,um, obtains the corresponding vector representation by querying the user embedding matrix I∈Rm×T, where T is the dimension of the embedding vector. Given a candidate pair of user u and item (entity) v, the entity vector and the relation vector get the corresponding initial embeddings v^0 and r^ in the entity embedding matrix and the relation embedding matrix through embedding lookup.In this paper, we apply the multilayer BiLSTM to learn features for entity and relation embeddings. The single-layer BiLSTM has achieved good results in simple prediction tasks, but the potential interest of users are often deeply hidden in the user-item interaction matrix. Therefore, this paper proposes to use the stacked multilayer time series network BiLSTM to extract the deep dependency features in the user and item interaction matrix to mine the potential interest of users. A single-layer BiLSTM [18] is composed of a forward LSTM and a backward LSTM, the former calculates the hidden layer state h←h←1,h←2,…,h←t and the latter calculates the hidden layer state hh1,h2,…,ht, which can be spliced to obtain the final hidden layer state. The multilayer BiLSTM network is composed of the forward multilayer LSTM and backward multilayer LSTM, and the input of the Nth layer is the output of the N−1th layer. The model structure can be seen in Figure 2.Figure 2
BiLSTM structure with multiple layers stacked.First, takev^0 and r^ as the input of the multilayer BiLSTM model for feature extraction and then splice the extracted entity embedding vector with the initial embedding vector v^0 to obtain the 0th-order information v0 of the entity, that is, the representation vector of the entity itself. Its process can be formulated as(1)Xn=BXn−1,Xn−1←∪<Xn−1c1,Xn−1c2,…,Xn−1cm>,Rn=BXn−1∪BXn−2,Xn−1⟵∪<Xn−1c1,Xn−1c2,…,Xn−1cm>,v0=v^0+Xn,where Xn and Rn are the embedded representations of entities and relations obtained after feature extraction, respectively. C1,C2,…Cm represents hidden layer nodes, and the number of internal hidden layer nodes is determined by the dimensions of entities and relationships. ⨄ indicates the feature stacking symbol.Since the input of BiLSTM contains the output of forward and backward LSTM, we sum and average the forward and backward features to effectively utilize the forward and backward feature information, which can be formulated as(2)Xn−1c1=Xn−1Forward1+Xn−1Backed−Output11≤c≤NH,where H is the number of hidden layer nodes and N is the total number of cells. Xn−1Forward1 represents the forward LSTM output vector and Xn−1Backed−Output1 represents the backward LSTM output vector.


### 2.2.2. Neighborhood Information Calculation

The aggregation calculation of neighborhood information needs to integrate the information of neighboring nodes. Specifically,Nv represents the set of neighbor nodes directly connected to entity v, rx,y indicates the relationship between ex and ey, and the neighborhood of entity v is represented as vP(v)u. When calculating the information representation of the neighborhood, the inner product function z:Rd×Rd⟶R is used to calculate the weight matrix between the user vector and the adjacency vector.(3)A=zu,r,where d represents the dimension and A represents the impact of different relationships on users by adding user information. In order to characterize the neighborhood structure of entity v, first calculate the linear combination of neighborhood as(4)vNvu=∑eεPVA˜e,where A˜ is the normalized matrix of A.(5)A˜=expA∑eεPvexpA,where e is the vector representation of the neighbor entity.


### 2.2.3. Self-Attention

Attention mechanism [19, 20] was initially applied in the field of machine translation, and now, it has been widely applied in image processing, recommendation system, and other aspects. This method draws on the mechanism of human visual selective attention, and its core purpose is to screen out the important content of the current task from a large amount of information. The self-attention mechanism proposed in this paper consists of a gated recurrent unit (GRU) module and a self-attention module. The calculated neighborhood representation v^P(v)u is taken as the input of self-attention, and the aggregation features of lower-dimensional neighborhood information are extracted through the GRU module. Input the obtained feature vector into the self-attention module to further learn the semantic association between entities and adjacent entities and more accurately calculate the representation of neighborhood information. The specific implementation process is as follows:Firstly,vPvu is sent to the GRU module as the input feature to extract the low-dimensional feature, and the output feature I is obtained through the learning of function h.(6)I=hvPvu|u,r.Then, the function bodyAtten is created, which is consistent with the implementation process of ordinary self-attention mechanism. The feature vector I processed by the GRU module is used as the input of the self-attention module, and the specific operation is shown in the following formula:(7)v=tanhI•Wa+b,v′=v•u,a=softmaxy′,O=I∗a,where • is the tensor-dot operation, wa is the trainable transformation matrix, b and u are vectors for training, α is the normalized weight, and O is the final output feature of self-attention.Finally, a full connection layer is used to obtain the final neighborhood information representationv^Pvu, and the specific calculation process is as follows:(8)v^Pvu=FCO,where FC is the operation function of the full connection layer.


### 2.2.4. Information Aggregation

The last step of the BAGCN layer is information aggregation. The model aggregates the node’s own informationv0 and all its neighborhood information v^J(v)u as the final representation of the node through the aggregator, takes the sum of the two vectors, and then performs nonlinear transformation on them. Here, Jv=K, K is a configurable constant, representing the number of neighbors sampled. Technically, if Jv<K, this paper samples with a put back; otherwise, it randomly samples fixed K neighbor nodes. In the real knowledge graph, there are great differences in the number of neighbor nodes of different entities. In this paper, the complete set of neighbor nodes is not included in the calculation, but a fixed size neighborhood set is sampled for each entity to ensure the same calculation mode and calculation efficiency of each batch. Its calculation is as follows:(9)agg=σW⋅v0+v^Jvu+b,where W and b are transformation weights and bias terms, respectively, and σ is a nonlinear function.The initial embedding representation of an entity is zero-order, and it is propagated to its neighbors to obtain first-order representation. The process is repeated from first-order representation to higher-order representation. The high-order representation can tap deeper potential interest of users.


## 2.2.1. Knowledge Graph Embedding

Knowledge graph embedding is an effective way to convert entities and relations into vector representations, which can transfer high-dimensional sparse features into low-dimensional feature vectors, to obtain a more convenient form for model input. The user,U=u1,u2,…,um, obtains the corresponding vector representation by querying the user embedding matrix I∈Rm×T, where T is the dimension of the embedding vector. Given a candidate pair of user u and item (entity) v, the entity vector and the relation vector get the corresponding initial embeddings v^0 and r^ in the entity embedding matrix and the relation embedding matrix through embedding lookup.In this paper, we apply the multilayer BiLSTM to learn features for entity and relation embeddings. The single-layer BiLSTM has achieved good results in simple prediction tasks, but the potential interest of users are often deeply hidden in the user-item interaction matrix. Therefore, this paper proposes to use the stacked multilayer time series network BiLSTM to extract the deep dependency features in the user and item interaction matrix to mine the potential interest of users. A single-layer BiLSTM [18] is composed of a forward LSTM and a backward LSTM, the former calculates the hidden layer state h←h←1,h←2,…,h←t and the latter calculates the hidden layer state hh1,h2,…,ht, which can be spliced to obtain the final hidden layer state. The multilayer BiLSTM network is composed of the forward multilayer LSTM and backward multilayer LSTM, and the input of the Nth layer is the output of the N−1th layer. The model structure can be seen in Figure 2.Figure 2
BiLSTM structure with multiple layers stacked.First, takev^0 and r^ as the input of the multilayer BiLSTM model for feature extraction and then splice the extracted entity embedding vector with the initial embedding vector v^0 to obtain the 0th-order information v0 of the entity, that is, the representation vector of the entity itself. Its process can be formulated as(1)Xn=BXn−1,Xn−1←∪<Xn−1c1,Xn−1c2,…,Xn−1cm>,Rn=BXn−1∪BXn−2,Xn−1⟵∪<Xn−1c1,Xn−1c2,…,Xn−1cm>,v0=v^0+Xn,where Xn and Rn are the embedded representations of entities and relations obtained after feature extraction, respectively. C1,C2,…Cm represents hidden layer nodes, and the number of internal hidden layer nodes is determined by the dimensions of entities and relationships. ⨄ indicates the feature stacking symbol.Since the input of BiLSTM contains the output of forward and backward LSTM, we sum and average the forward and backward features to effectively utilize the forward and backward feature information, which can be formulated as(2)Xn−1c1=Xn−1Forward1+Xn−1Backed−Output11≤c≤NH,where H is the number of hidden layer nodes and N is the total number of cells. Xn−1Forward1 represents the forward LSTM output vector and Xn−1Backed−Output1 represents the backward LSTM output vector.


## 2.2.2. Neighborhood Information Calculation

The aggregation calculation of neighborhood information needs to integrate the information of neighboring nodes. Specifically,Nv represents the set of neighbor nodes directly connected to entity v, rx,y indicates the relationship between ex and ey, and the neighborhood of entity v is represented as vP(v)u. When calculating the information representation of the neighborhood, the inner product function z:Rd×Rd⟶R is used to calculate the weight matrix between the user vector and the adjacency vector.(3)A=zu,r,where d represents the dimension and A represents the impact of different relationships on users by adding user information. In order to characterize the neighborhood structure of entity v, first calculate the linear combination of neighborhood as(4)vNvu=∑eεPVA˜e,where A˜ is the normalized matrix of A.(5)A˜=expA∑eεPvexpA,where e is the vector representation of the neighbor entity.


## 2.2.3. Self-Attention

Attention mechanism [19, 20] was initially applied in the field of machine translation, and now, it has been widely applied in image processing, recommendation system, and other aspects. This method draws on the mechanism of human visual selective attention, and its core purpose is to screen out the important content of the current task from a large amount of information. The self-attention mechanism proposed in this paper consists of a gated recurrent unit (GRU) module and a self-attention module. The calculated neighborhood representation v^P(v)u is taken as the input of self-attention, and the aggregation features of lower-dimensional neighborhood information are extracted through the GRU module. Input the obtained feature vector into the self-attention module to further learn the semantic association between entities and adjacent entities and more accurately calculate the representation of neighborhood information. The specific implementation process is as follows:Firstly,vPvu is sent to the GRU module as the input feature to extract the low-dimensional feature, and the output feature I is obtained through the learning of function h.(6)I=hvPvu|u,r.Then, the function bodyAtten is created, which is consistent with the implementation process of ordinary self-attention mechanism. The feature vector I processed by the GRU module is used as the input of the self-attention module, and the specific operation is shown in the following formula:(7)v=tanhI•Wa+b,v′=v•u,a=softmaxy′,O=I∗a,where • is the tensor-dot operation, wa is the trainable transformation matrix, b and u are vectors for training, α is the normalized weight, and O is the final output feature of self-attention.Finally, a full connection layer is used to obtain the final neighborhood information representationv^Pvu, and the specific calculation process is as follows:(8)v^Pvu=FCO,where FC is the operation function of the full connection layer.


## 2.2.4. Information Aggregation

The last step of the BAGCN layer is information aggregation. The model aggregates the node’s own informationv0 and all its neighborhood information v^J(v)u as the final representation of the node through the aggregator, takes the sum of the two vectors, and then performs nonlinear transformation on them. Here, Jv=K, K is a configurable constant, representing the number of neighbors sampled. Technically, if Jv<K, this paper samples with a put back; otherwise, it randomly samples fixed K neighbor nodes. In the real knowledge graph, there are great differences in the number of neighbor nodes of different entities. In this paper, the complete set of neighbor nodes is not included in the calculation, but a fixed size neighborhood set is sampled for each entity to ensure the same calculation mode and calculation efficiency of each batch. Its calculation is as follows:(9)agg=σW⋅v0+v^Jvu+b,where W and b are transformation weights and bias terms, respectively, and σ is a nonlinear function.The initial embedding representation of an entity is zero-order, and it is propagated to its neighbors to obtain first-order representation. The process is repeated from first-order representation to higher-order representation. The high-order representation can tap deeper potential interest of users.


## 2.3. Prediction Layer

The prediction layer predicts the probability of interaction itemi of user u by inputting a function f:Rd×Rd⟶R, as shown in the following formula:(10)y^u,i=fu,i.


## 2.4. Loss Function

The cross-entropy loss function is used to train the model in this paper, and the specific formula is as follows:(11)loss=∑uεU∑i:yui=1Jyui,y^ui−∑n=1TuEin∼QvnJyuin,y^uin+λF22,where J is the cross-entropy loss, Q is the negative sampling distribution, and Tu is the negative sample number of user u. In this paper, Tu=v:yuv=1, Q follows uniform distribution and the last term is L2 regularization.


## 3. Experimental Results and Analysis

### 3.1. Datasets

We evaluate the proposed BAGCN model by recommending movies, books, and music on three datasets, i.e., MovieLens-20M, Book-Crossing, and Last.FM, respectively. The triple information of constructing the knowledge graph of each dataset comes from Microsoft Satori, and a subset of triples is selected in the entire knowledge graph, whose confidence is greater than 0.9. Table1 records the basic information of the three datasets.Table 1
Basic data set information.
Movielens-20MBook-crossingLast.FMUser-item’ interactive informationUsers138159196761872Items16954200033846Interactions1350162217257642346KG informationEntities102569257879366Relations321860KG triples4994746078715516These three datasets belong to explicit feedback. In order to mine user preference information, we convert explicit feedback into implicit feedback to learn model recommendation in this paper. In implicit feedback, the user with a positive interaction with an item can be counted as 1, and a sampled negative sample set for each user is counted as 0, which means the sample set has not been observed. The rating range of the MovieLens-20M dataset is 1∼5, and the rating range of Book-Crossing is 1∼10. Meanwhile, the rating threshold of the MovieLens-20M dataset is set to 4, which means a rating greater than 4 is regarded as a positive interaction. The two datasets, Book-Crossing and Last.FM, are sparser than MovieLens-20M, so we do not set a threshold.


### 3.2. Comparative Models Introduction

SVD [4]: a classic CF-based model. It analyzes the user’s liking of each factor and the degree to which the item contains each factor based on the existing data.CKE [9]: integrate various auxiliary information such as structured, textual, and visual knowledge with CF in a unified recommendation framework for joint training.PER [13]: viewing the knowledge graph as a heterogeneous information network, it represents the connectivity between users and items by extracting latent features of meta-paths.KGCN [15]: a representative of a hybrid recommendation training model. It implements recommendation by integrating the characteristics of knowledge graphs and graph convolutional neural networks and aggregating neighbor information and bias to obtain item representations [21–24].


### 3.3. Experimental Setup

In BAGCN, we set the functionsz and f that are the inner products, σ is the activation function ReLU of the nonlast layer aggregator, and tanh is the activation function of the last layer aggregator. Table 2 provides other hyperparameter settings of this model. K denotes the number of neighbors of each sampled entity, d denotes the dimension of the embedding, H denotes the number of iterations, η denotes the learning rate, and λ denotes the regularization term coefficient. The hyperparameters are sized by optimizing AUC on the validation set, and the ratio of training set, evaluation set, and test set is set to 6 : 2:2. We evaluate the BAGCN algorithm in two experimental scenarios: (1) in CTR prediction, the trained model is used to predict each interaction in the test set, and AUC, F1, and ACC are used to evaluate CTR prediction. (2) In top-N recommendation, the trained model is designed to select N items with the highest predicted click probability for each user in the test set for recommendation. By default, N = 1. All trainable parameters are optimized using the Adam algorithm.Table 2
Hyperparameter settings.
HyperparameterMovielens-20MBook-crossingLast.FMK4816d8168λ10–72 × 10−510–4η2 × 10−22 × 10−45 × 10−4H111Batch_size65536256128PER uses artificially designed paths as features, SVD sets MovieLens-20M, and Book-Crossingd = 8, η = 0.5. For Last.FM, d = 8, η = 0.1. The dimension d of the three datasets of the CKE model is set to 64, 128, and 64, respectively, the training weight for the KG part is 0.1, and the learning rate is set the same as SVD. The KGCN model is set to K = 4, d = 32, η = 0.02, H = 2, and λ = 10−7 in the MovieLens-20M dataset, and the parameters of Book-Crossing are set to K = 8, d = 64, η = 0.0002, H = 1, and λ = 2 × 10−5, respectively. In Last.FM, K = 8, d = 16, η = 0.0005, H = 1, λ = 10−4, and H represents the number of aggregation iterations. The number of training rounds is set to 60, and the number of training rounds for MovieLens-20M is set to 30 for the Last.FM and Book-Crossing datasets.


### 3.4. Experimental Results Analysis

The CTR prediction and top-N recommendation results in different datasets are displayed in Tables3–5, Figures 3, and 4. Overall, the BAGCN model outperforms the baselines on all three datasets tested, including ACC, AUC, F1, precision, and recall. Comprehensively comparing the performance of various indicators in the three datasets, our method has improved the accuracy rate (ACC), while AUC has increased. The multilayer bidirectional recurrent neural network BiLSTM can effectively utilize the forward and backward feature information of entities and relationships. At the same time, adding an attention mechanism layer can perform biased aggregation of neighborhood information to make the representation of nodes more accurate. By observing the experimental results, the following conclusions can be drawn:(1)
The performance of BAGCN is better than that of all the compared baseline models. Compared with the KGCN model in the three datasets of movies, books, and music, ACC is improved by 1.00%, 2.58%, and 1.03%, and the pre is improved by 0.63%, 2.28%, and 1.38%.(2)
PER performs the worst due to its heavy reliance on manually designed meta-paths, and the optimal path is difficult to define in reality.(3)
In the three data sets, Book-Crossing and Last.FM are more sparse than MovieLens-20M data, but the improvement of each index is more obvious, indicating that the BAGCN model proposed in this paper can alleviate data sparsity to a certain extent sexual problems.Table 3
Model performance on MovieLens-20M.
ModelACCAUCF1Pre@1Recall@1SVD [4]—0.96300.9190——CKE [9]—0.92400.8710——PER [13]—0.83200.7880——KGCN [15]0.92350.97200.92420.91650.9321BAGCN0.93350.97800.93430.92280.9461Table 4
Model performance on Book-Crossing.
ModelACCAUCF1Pre@1Recall@1SVD [4]—0.67200.6350——CKE [9]—0.67700.6110——PER [13]—0.61700.5620——KGCN [15]0.60100.64710.61590.59750.6403BAGCN0.62680.67910.64250.62030.6682Table 5
Model performance on Last.FM
ModelACCAUCF1Pre@1Recall@1SVD [4]—0.76900.6960——CKE [9]—0.74400.6730——PER [13]—0.63300.5960——KGCN [15]0.70930.78480.71070.70890.7154BAGCN0.71960.79130.71880.72270.7175Figure 3
(a) The performance of pre on MovieLens-20M. (b) The performance of pre on Book-Crossing. (c) The performance of pre on Last.FM.
(a)(b)(c)Figure 4
(a) The performance of recall on MovieLens-20M. (b) The performance of recall on Last.FM. (c) The performance of recall on Book-Crossing.
(a)(b)(c)


### 3.5. Hyperparameter Optimization

Different hyperparameters will affect the effect of the model. In order to find the best value of the model on the validation dataset, it is necessary to optimize the super parameters. This section optimizes the number of sampling neighborsK and the embedding dimension d.


#### 3.5.1. The Influence of DifferentK on the Model

The use efficiency of the knowledge graph is studied by changing the number of sampling neighborsK. It can be observed from Table 6 that, for the datasets MovieLens-20M, Last.FM, and Book-Crossing, the model has the best performance when K = 4, 16, and 8, respectively. Because the dataset of MovieLens-20M is denser than the other two, only fewer neighbors need to be sampled. If K is too small, the model cannot include enough neighbor information to represent nodes; but if K is too large, noise will be introduced, which will affect the recommendation effect of the model.Table 6
AUC result of BAGCN with differentK
K2481632MovieLens-20M0.97750.97800.97770.97780.9777Last.FM0.79000.79070.78900.79130.7905Book-Crossing0.67660.67450.67910.67540.6743


#### 3.5.2. The Influence of Differentd on the Model

In addition, the influence of different embedding dimensionsd, which is also the number of hidden nodes of BiLSTM, on the model is studied. It can be observed from Table 7 that increasing the size of d at first can improve the recommendation effect of the model, but as the value of d continues to increase, the performance of the model becomes worse instead. This is because a larger d will cause the model to be overfitted, which will affect the effect of the model.Table 7
AUC result of BAGCN with differentd.
d48163264MovieLens-20M0.97620.97800.97680.97260.9689Last.FM0.78760.79130.78110.76900.7546Book-Crossing0.67390.67080.67910.66910.6643


## 3.1. Datasets

We evaluate the proposed BAGCN model by recommending movies, books, and music on three datasets, i.e., MovieLens-20M, Book-Crossing, and Last.FM, respectively. The triple information of constructing the knowledge graph of each dataset comes from Microsoft Satori, and a subset of triples is selected in the entire knowledge graph, whose confidence is greater than 0.9. Table1 records the basic information of the three datasets.Table 1
Basic data set information.
Movielens-20MBook-crossingLast.FMUser-item’ interactive informationUsers138159196761872Items16954200033846Interactions1350162217257642346KG informationEntities102569257879366Relations321860KG triples4994746078715516These three datasets belong to explicit feedback. In order to mine user preference information, we convert explicit feedback into implicit feedback to learn model recommendation in this paper. In implicit feedback, the user with a positive interaction with an item can be counted as 1, and a sampled negative sample set for each user is counted as 0, which means the sample set has not been observed. The rating range of the MovieLens-20M dataset is 1∼5, and the rating range of Book-Crossing is 1∼10. Meanwhile, the rating threshold of the MovieLens-20M dataset is set to 4, which means a rating greater than 4 is regarded as a positive interaction. The two datasets, Book-Crossing and Last.FM, are sparser than MovieLens-20M, so we do not set a threshold.


## 3.2. Comparative Models Introduction

SVD [4]: a classic CF-based model. It analyzes the user’s liking of each factor and the degree to which the item contains each factor based on the existing data.CKE [9]: integrate various auxiliary information such as structured, textual, and visual knowledge with CF in a unified recommendation framework for joint training.PER [13]: viewing the knowledge graph as a heterogeneous information network, it represents the connectivity between users and items by extracting latent features of meta-paths.KGCN [15]: a representative of a hybrid recommendation training model. It implements recommendation by integrating the characteristics of knowledge graphs and graph convolutional neural networks and aggregating neighbor information and bias to obtain item representations [21–24].


## 3.3. Experimental Setup

In BAGCN, we set the functionsz and f that are the inner products, σ is the activation function ReLU of the nonlast layer aggregator, and tanh is the activation function of the last layer aggregator. Table 2 provides other hyperparameter settings of this model. K denotes the number of neighbors of each sampled entity, d denotes the dimension of the embedding, H denotes the number of iterations, η denotes the learning rate, and λ denotes the regularization term coefficient. The hyperparameters are sized by optimizing AUC on the validation set, and the ratio of training set, evaluation set, and test set is set to 6 : 2:2. We evaluate the BAGCN algorithm in two experimental scenarios: (1) in CTR prediction, the trained model is used to predict each interaction in the test set, and AUC, F1, and ACC are used to evaluate CTR prediction. (2) In top-N recommendation, the trained model is designed to select N items with the highest predicted click probability for each user in the test set for recommendation. By default, N = 1. All trainable parameters are optimized using the Adam algorithm.Table 2
Hyperparameter settings.
HyperparameterMovielens-20MBook-crossingLast.FMK4816d8168λ10–72 × 10−510–4η2 × 10−22 × 10−45 × 10−4H111Batch_size65536256128PER uses artificially designed paths as features, SVD sets MovieLens-20M, and Book-Crossingd = 8, η = 0.5. For Last.FM, d = 8, η = 0.1. The dimension d of the three datasets of the CKE model is set to 64, 128, and 64, respectively, the training weight for the KG part is 0.1, and the learning rate is set the same as SVD. The KGCN model is set to K = 4, d = 32, η = 0.02, H = 2, and λ = 10−7 in the MovieLens-20M dataset, and the parameters of Book-Crossing are set to K = 8, d = 64, η = 0.0002, H = 1, and λ = 2 × 10−5, respectively. In Last.FM, K = 8, d = 16, η = 0.0005, H = 1, λ = 10−4, and H represents the number of aggregation iterations. The number of training rounds is set to 60, and the number of training rounds for MovieLens-20M is set to 30 for the Last.FM and Book-Crossing datasets.


## 3.4. Experimental Results Analysis

The CTR prediction and top-N recommendation results in different datasets are displayed in Tables3–5, Figures 3, and 4. Overall, the BAGCN model outperforms the baselines on all three datasets tested, including ACC, AUC, F1, precision, and recall. Comprehensively comparing the performance of various indicators in the three datasets, our method has improved the accuracy rate (ACC), while AUC has increased. The multilayer bidirectional recurrent neural network BiLSTM can effectively utilize the forward and backward feature information of entities and relationships. At the same time, adding an attention mechanism layer can perform biased aggregation of neighborhood information to make the representation of nodes more accurate. By observing the experimental results, the following conclusions can be drawn:(1)
The performance of BAGCN is better than that of all the compared baseline models. Compared with the KGCN model in the three datasets of movies, books, and music, ACC is improved by 1.00%, 2.58%, and 1.03%, and the pre is improved by 0.63%, 2.28%, and 1.38%.(2)
PER performs the worst due to its heavy reliance on manually designed meta-paths, and the optimal path is difficult to define in reality.(3)
In the three data sets, Book-Crossing and Last.FM are more sparse than MovieLens-20M data, but the improvement of each index is more obvious, indicating that the BAGCN model proposed in this paper can alleviate data sparsity to a certain extent sexual problems.Table 3
Model performance on MovieLens-20M.
ModelACCAUCF1Pre@1Recall@1SVD [4]—0.96300.9190——CKE [9]—0.92400.8710——PER [13]—0.83200.7880——KGCN [15]0.92350.97200.92420.91650.9321BAGCN0.93350.97800.93430.92280.9461Table 4
Model performance on Book-Crossing.
ModelACCAUCF1Pre@1Recall@1SVD [4]—0.67200.6350——CKE [9]—0.67700.6110——PER [13]—0.61700.5620——KGCN [15]0.60100.64710.61590.59750.6403BAGCN0.62680.67910.64250.62030.6682Table 5
Model performance on Last.FM
ModelACCAUCF1Pre@1Recall@1SVD [4]—0.76900.6960——CKE [9]—0.74400.6730——PER [13]—0.63300.5960——KGCN [15]0.70930.78480.71070.70890.7154BAGCN0.71960.79130.71880.72270.7175Figure 3
(a) The performance of pre on MovieLens-20M. (b) The performance of pre on Book-Crossing. (c) The performance of pre on Last.FM.
(a)(b)(c)Figure 4
(a) The performance of recall on MovieLens-20M. (b) The performance of recall on Last.FM. (c) The performance of recall on Book-Crossing.
(a)(b)(c)


## 3.5. Hyperparameter Optimization

Different hyperparameters will affect the effect of the model. In order to find the best value of the model on the validation dataset, it is necessary to optimize the super parameters. This section optimizes the number of sampling neighborsK and the embedding dimension d.


### 3.5.1. The Influence of DifferentK on the Model

The use efficiency of the knowledge graph is studied by changing the number of sampling neighborsK. It can be observed from Table 6 that, for the datasets MovieLens-20M, Last.FM, and Book-Crossing, the model has the best performance when K = 4, 16, and 8, respectively. Because the dataset of MovieLens-20M is denser than the other two, only fewer neighbors need to be sampled. If K is too small, the model cannot include enough neighbor information to represent nodes; but if K is too large, noise will be introduced, which will affect the recommendation effect of the model.Table 6
AUC result of BAGCN with differentK
K2481632MovieLens-20M0.97750.97800.97770.97780.9777Last.FM0.79000.79070.78900.79130.7905Book-Crossing0.67660.67450.67910.67540.6743


### 3.5.2. The Influence of Differentd on the Model

In addition, the influence of different embedding dimensionsd, which is also the number of hidden nodes of BiLSTM, on the model is studied. It can be observed from Table 7 that increasing the size of d at first can improve the recommendation effect of the model, but as the value of d continues to increase, the performance of the model becomes worse instead. This is because a larger d will cause the model to be overfitted, which will affect the effect of the model.Table 7
AUC result of BAGCN with differentd.
d48163264MovieLens-20M0.97620.97800.97680.97260.9689Last.FM0.78760.79130.78110.76900.7546Book-Crossing0.67390.67080.67910.66910.6643


## 3.5.1. The Influence of DifferentK on the Model

The use efficiency of the knowledge graph is studied by changing the number of sampling neighborsK. It can be observed from Table 6 that, for the datasets MovieLens-20M, Last.FM, and Book-Crossing, the model has the best performance when K = 4, 16, and 8, respectively. Because the dataset of MovieLens-20M is denser than the other two, only fewer neighbors need to be sampled. If K is too small, the model cannot include enough neighbor information to represent nodes; but if K is too large, noise will be introduced, which will affect the recommendation effect of the model.Table 6
AUC result of BAGCN with differentK
K2481632MovieLens-20M0.97750.97800.97770.97780.9777Last.FM0.79000.79070.78900.79130.7905Book-Crossing0.67660.67450.67910.67540.6743


## 3.5.2. The Influence of Differentd on the Model

In addition, the influence of different embedding dimensionsd, which is also the number of hidden nodes of BiLSTM, on the model is studied. It can be observed from Table 7 that increasing the size of d at first can improve the recommendation effect of the model, but as the value of d continues to increase, the performance of the model becomes worse instead. This is because a larger d will cause the model to be overfitted, which will affect the effect of the model.Table 7
AUC result of BAGCN with differentd.
d48163264MovieLens-20M0.97620.97800.97680.97260.9689Last.FM0.78760.79130.78110.76900.7546Book-Crossing0.67390.67080.67910.66910.6643


## 4. Conclusion

Aiming at the problems of cold start, data sparsity, and poor performance of existing recommendation algorithms, this paper proposed the BAGCN algorithm. The algorithm extracts the features of the embedding of entities and relationships in the knowledge map, learns the semantic information in the knowledge map through the self-attention mechanism, aggregates the neighborhood information in a biased way, and then extends to multihop simulation of high-order structure information to tap the potential interest of users. Through experiments on several datasets, the BAGCN model is proved to be superior to the baseline model in various performance evaluation indexes of film, book, and music recommendation. In this paper, negative samples are sampled in a uniform way, and high-quality negative samples are as important for model learning as positive samples. In addition, the user’s demographic information and the recommendation model will be considered as auxiliary information to improve the performance of user integration.



---

*Source: 8247846-2022-09-07.xml*