# TRAL: A Tag-Aware Recommendation Algorithm Based on Attention Learning
## 4 Method
### The Deep Component
#### 4.1.2 Attention Layer  
In a real-life scenario, there are obvious differences in users' behavior habits and cultural backgrounds. Therefore, the number and the content of tags for the same item  
Figure 2: The overall structure of the proposed model. The upper right corner illustrates the difference between average pooling and attention pooling. The weights of different feature vectors are set to the same value in average pooling, while these weights will be automatically learned according to their importance in attention pooling.  
marked by different users will be significantly different. Note that the input dimensions of full connected neural networks are required to be consistent. Additional operators should be taken to convert the variable list of embedding vectors to a fixed-length vector. A common approach is to construct a pooling layer, where the list of embedding vectors is compressed by a weighted sum operator. Suppose that the list of embedding vectors for a given user is \(h_{t}^{ll}=\{e_{1},e_{2},\cdots,e_{L}\}\), the compressed feature vector is obtained by the following weighted sum operation:  
\[v_{t}^{ll}=\sum_{j=1}^{L}a_{j}e_{j} \tag{3}\]  
where \(a_{j}\) is the weight indicating the importance of the corresponding tag-based embedding vector \(e_{j}\). Two widely adopted strategies are sum pooling and average pooling, which treat each embedding vector equally and set all weights to the same value.  
Clearly, user tagging behaviors play critical role in modeling user preferences in TRS. For a certain user, the compressed fix-length feature vector by sum or average pooling remains constant no matter what the predicted item is. It requires that the tag-based feature vector is capable to express multiple interests of users. To this end, the embedding size should be expanded to large enough, resulting in the increase in computation burden. Moreover, we argue that not all user tagging information are equally important and effective when predicting a target item. Those tag-based features that are less useful should naturally be given a lower weight, which is the main limitation of sum or average pooling.  
Based on the above considerations, we resort to the attention mechanism, which allows the weight to be calculated automatically from data. The rationale is that the contributions of different embedding vectors should be taken into consideration when compressing them into a single representation vector. In this work, we construct a two-layer neural network to realize the attention mechanism. Specifically, the weight of each tagging feature vector is first calculated by:  
\[\hat{a_{j}}=q^{T}ReLU(W_{a}e_{j}+b) \tag{4}\]  
where \(W_{a}\in\mathbb{R}^{m\times d}\), \(b\in\mathbb{R}^{m}\), \(q\in\mathbb{R}^{m}\) are learnable parameters, and \(m\) represents the number of hidden layer neurons in the attention network, which is called attention factor. The final attention weights are then normalized by a softmax function:  
\[a_{j}=\frac{\exp(\hat{a_{j}})}{\sum_{j=1}^{L}\exp(\hat{a_{j}})} \tag{5}\]  
In this way, a fixed-length user representation \(v_{t}^{ll}\) can be adaptively derived by discriminating the importance of different embedding feature vectors. An attention network with similar structures is created so as to generate a fixed-length item representation \(v_{t}^{l}\).