# Efficient Tree Policy with Attention-Based State Representation for Interactive Recommendation
## 3 Method
### Attention-Based State Representation Model  
In this section, we describe the proposed attention-based state representation model in detail.  
The proposed state representation model consists of an input layer, a feature embedding layer, an attention layer, and an output layer, as shown in Figure 2. The input layer receives the feature vectors from items and the users' feedback, and then preprocessed them by a fully connected layer. The attention layer calculates the weights of each corresponding item. Finally, the output layer concatenates the user feature and the output context embedding vector of the attention layer. In the input layer, the historical \(N\) interactions, consisting of the recommended items and their corresponding rewards along with the user ID are collected. Assuming that the recommender agent is performing \(t\)-th recommendation, then \(N\) previous item-reward pairs (from \(t-N-1\) to \(t-1\)) are used to encode the state. The item ID and user ID are mapped to a latent feature vector via matrix factorization. The feature vectors are denoted as the item feature and user feature, respectively, which are fixed during learning. The user feedback on the corresponding recommended items, such as ratings, is also added to the item feature. In the feature embedding layer, the input user features and item features are encoded by a fully connected layer to obtain the feature embeddings.  
Figure 2: Attention-based state representation model.  
In order to capture the dynamics for different items in long and noisy sequential data, we adopt an attention layer to discriminatively capture the contributions of each item together with the user features. The attention layer aims at learning the integration weights for both item and user features. Denoting \(h_{i}\) as the \(i\)th preprocessed item feature and \(h_{user}\) as the preprocessed user feature, the attentive context embedding is calculated as follows:  
\[\begin{split}& e=\sum_{i=1}^{N}(\alpha_{ti}h_{i})+\alpha_{user}h_{user},\\ & s.t.\sum_{i=1}^{N}(\alpha_{ti})+\alpha_{user}=1\end{split} \tag{1}\]  
where \(\alpha_{ti}\) is the integration weight of the \(i\)th item embedding vector with respect to the \(t\)th target item and \(\alpha_{user}\) is the integration weight for a specific user.  
The attention weights are calculated through a Softmax layer as below:  
\[\begin{split}&\alpha_{ti}=\frac{\exp(f(h_{i}))}{\sum_{i=1}^{N}\exp(f(h_ {i}^{F}))+\exp(f(h_{user}))}\\ &\alpha_{user}=\frac{\exp(f(h_{user}))}{\sum_{i=1}^{N}\exp(f(h_ {i}^{F}))+\exp(f(h_{user}))}\\ & f(h_{i})=ReLU(W^{\alpha}h_{i}^{T}+b^{\alpha})\;\forall h_{i} \in\{h_{i},h_{user}\}\end{split} \tag{2}\]  
where \(W^{\alpha}\) and \(b^{\alpha}\) are the weight and bias parameters for the attention layer, respectively.  
Finally, in the output layer, we concatenate the input user feature and the attentive context embedding as the state embedding vector. The concatenate symbol is represented as \(\oplus\) in Figure 2.  
Although the state representation model DRR-att in [4] looks similar to our proposed model, compared to DRR-att, our proposed state representation model involves users' feedback as the input of items. In addition, we also preprocess the input features by a fully connected layer and simplify the integration of user feature and item features. Our experimental results show that our proposed method outperforms DRR-att in our benchmark datasets.