SlowGuess/NAACL_2022_OCR huggingface dataset:
to represent  $(u,r)$ ,  $\mathbf{f}^t = [\mathbf{u}_f^t;\mathbf{r}_c^t]$ , where  $\mathbf{u}_f^t = \text{Aggregate}([\mathbf{P}^t|\mathbf{N}^t])$ .  $\mathbf{P}^t, \mathbf{N}^t$  and  $\mathbf{r}_c^t$  are the title representations of the news in user positive and negative feedback sequences and the candidate news extracted by the title encoder. [] denotes the row-wise concatenation for matrices.
Then, this module leverages four different fully connected layers to calculate the weights for four representations extracted by the semantic interest encoder (i.e.,  $\mathbf{p}_s^t$ ,  $\mathbf{n}_s^t$ ,  $\mathbf{p}_h^t$  and  $\mathbf{n}_h^t$ ). For example, the weight of  $\mathbf{p}_s^t$  is calculated by
$$
\beta_ {s} ^ {p} = \tanh  \left(\mathbf {f} ^ {t} \mathbf {W} _ {1} ^ {p, s} + \mathbf {b} _ {1} ^ {p, s}\right) \mathbf {W} _ {2} ^ {p, s} + b _ {2} ^ {p, s} \tag {9}
$$
where  $\mathbf{W}_1^{p,s} \in \mathbb{R}^{2d \times d}$ ,  $\mathbf{W}_2^{p,s} \in \mathbb{R}^{d \times 1}$ ,  $\mathbf{b}_1^{p,s} \in \mathbb{R}^d$ ,  $b_2^{p,s} \in \mathbb{R}$  are learnable parameters. The weights,  $\beta_s^n$ ,  $\beta_h^n$  and  $\beta_h^n$ , of the representations  $\mathbf{n}_s^t$ ,  $\mathbf{p}_h^t$  and  $\mathbf{n}_h^t$  can be calculated by the same way in Eq.(9).
Finally, the user content-view representation is calculated by
$$
\mathbf {u} ^ {t} = \beta_ {s} ^ {p} * \mathbf {p} _ {s} ^ {t} + \beta_ {s} ^ {n} * \mathbf {n} _ {s} ^ {t} + \beta_ {h} ^ {p} * \mathbf {p} _ {h} ^ {t} + \beta_ {h} ^ {n} * \mathbf {n} _ {h} ^ {t} \tag {10}
$$
Another fusion net is used to fuse four interest representations extracted by the collaborative interest encoder and has a similar structure with the above one. The only difference is that it uses the outputs of the news ID embedding layer to represent the  $(u,r)$  pair,  $\mathbf{f}^o = [\mathbf{u}_f^o;\mathbf{r}_c^o]$ , where  $\mathbf{u}_f^o = \text{Aggregate}([\mathbf{P}^o|\mathbf{N}^o])$ . Then, it uses  $\mathbf{f}^o$  to calculate the respective weights  $\theta_s^p$ ,  $\theta_s^n$ ,  $\theta_h^p$  and  $\theta_h^n$  for four interest representations  $\mathbf{p}_s^o$ ,  $\mathbf{n}_s^o$ ,  $\mathbf{p}_h^o$  and  $\mathbf{n}_h^o$  by the same way in Eq.(9). The final user graph-view representation is calculated by
$$
\mathbf {u} ^ {o} = \theta_ {s} ^ {p} * \mathbf {p} _ {s} ^ {o} + \theta_ {s} ^ {n} * \mathbf {n} _ {s} ^ {o} + \theta_ {h} ^ {p} * \mathbf {p} _ {h} ^ {o} + \theta_ {h} ^ {n} * \mathbf {n} _ {h} ^ {o} \tag {11}
$$
4.7 Prediction
Following (Wu et al., 2019c), the clicking probability score  $\hat{y}$  is computed by the inner product of the user representation and the candidate news representation:  $\hat{y} = \mathbf{u}^{t\top}\mathbf{r}_c^t +\mathbf{u}^{o\top}\mathbf{r}_c^o$  where  $\mathbf{u}^{t\top}\mathbf{r}_{c}^{t}$  stands for the score calculated from title information and  $\mathbf{u}^{o\top}\mathbf{r}_{c}^{o}$  stands for the score calculated from collaborative information.
4.8 Training
Following (Wu et al., 2019c), for each positive sample, we randomly select  $l_{k}$  negative samples from the same user to construct a  $l_{k} + 1$  classification task. Each output of the DRPN for a classification sample is like  $[\hat{y}^{+},\hat{y}_{1}^{-},\dots,\hat{y}_{l_{k}}^{-}]$ , where  $\hat{y}^{+}$  denotes
# users	654870	# avg. titles words	12.66
# news	104153	# positive samples	1048414
# words	54292	# negative samples	25145229
# avg. positive feedback sequence length	4.14
# avg. negative feedback sequence length	96.80
Table 1: Statistics of the dataset.
the clicking probability score of the positive sample and the rest denote the scores of the  $l_{k}$  negative samples. We define the training loss (to be minimized) as follows.
$$
\mathcal {L} = - \sum_ {i \in \mathcal {P}} \log \left(\frac {\exp \left(\hat {y} _ {i} ^ {+}\right)}{\exp \left(\hat {y} _ {i} ^ {+}\right) + \sum_ {j = 1} ^ {j = l _ {k}} \exp \left(\hat {y} _ {i , j} ^ {-}\right)}\right) \tag {12}
$$
where  $\mathcal{P}$  denotes the set of positive samples.
4.9 Computation Complexity
The time complexity of the title encoder is  $\mathcal{O}(L^2 d + Ld^2)$ , where  $L$  is the title length and  $d$  is the embedding size. The time complexity of each interest encoder is  $\mathcal{O}((l_p + l_n)d^2 +(l_p^2 +l_n^2 +(l_p + l_n)^2)d)$  where  $l_{p}$  and  $l_{n}$  are the lengths of positive and negative feedback sequences. The time complexity of GNN is  $\mathcal{O}(|\mathcal{G}|d)$ , where  $|\mathcal{G}|$  denotes the number of edges that existed in collaborative graph. Hence, The overall time cost is  $\mathcal{O}((l_p + l_n)(Ld^2 +L^2 d) + (l_p^2 +l_n^2 +(l_p + l_n)^2 +|\mathcal{G}|)d)$ .
During the inference phase, we can compute the news representations in advance and the computation complexity will be  $\mathcal{O}((l_p^2 + l_n^2 + (l_p + l_n)^2)d)$ .
There is no off-the-shelf dataset in which the user profile includes both positive and negative historical feedback sequences. Therefore, we use MIND  ${}^{1}$  dataset (its original user profile only contains positive feedback) to re-build one to conduct the experiments. The original MIND dataset contains the user impression logs. An impression log records the news displayed to a user when visiting the news website homepage at a specific time, and the click behaviors on the news list. We re-build the dataset based on the MIND's impression logs as follows: (1) Select the impression logs of the first 5 days of the original training set. Then we add the news that a user has seen but did not click to his negative feedback sequence, and add the news he clicked
1 https://msnews.github.io/
to his positive feedback sequence. In this manner, the user profile includes both positive and negative historical feedback sequences; (2) Training set: the impression logs of 6-th day of the original training set; (3) Validation set: the first  $10\%$  chronological impression logs of the original validation set; (4) Testing set: the last  $90\%$  chronological impression logs of the original validation set.
The training, validation, and testing sets use the same user profile built in Step (1). Since the user profiles are only built in Step (1) which is ahead of Step (2)-(4), there is no label leakage to validation and testing sets. Moreover, same as the original MIND dataset, the re-built dataset also has  $44.6\%$  users of validation set and  $48.7\%$  users of test set that are not shown in the re-built training set. Table 1 shows some statistics of the re-built dataset.
5.2 Baseline Approaches and Metrics
We evaluate the performance of DRPN by comparing it with several baseline methods, including: (1) LibFM (Rendle, 2012), factorization machine (FM); (2) DeepFM (Guo et al., 2017), which combines the FM and neural networks; (3) DKN (Wang et al., 2018a), which uses the CNN to fuse the entity and word embeddings to learn news representations; (4) LSTUR (An et al., 2019), which uses the GRU to model short- and long-term interests from the click history; (5) NPA (Wu et al., 2019b), which introduces the attention mechanism to select important words and news; (6) DEERS (Zhao et al., 2018), which uses GRU to encode positive and negative feedback sequences; (7) DFN (Xie et al., 2020), a factorization-machine based network which uses transformers to encode both positive and negative feedback sequences to enhance performance; (8) GERL (Ge et al., 2020), which constructs user-news graph to enhance the performance; (9) NAML (Wu et al., 2019a), which uses multi-view learning to aggregate different kinds of information to represent news; (10) NRMS (Wu et al., 2019c), which uses multi-head self-attention to learn news and user representations; (11) NAML + TCE, which incorporates the denoising training strategy TCE (Wang et al., 2021) into NAML; (12) NRMS + TCE, which improves NRMS by using TCE.
Following the previous news recommendation work (Wu et al., 2020b, 2019c; Ge et al., 2020), we use AUC, MRR, nDCG@5, and nDCG@10 scores as our evaluation metrics.
For DRPN, the representation dimension  $d$  is set to 300. We use the GloVe.840B.300d (Pennington et al., 2014) as the pre-trained word embeddings. The maximum title length is set to 15. The lengths of feedback sequences  $l_{p}$  and  $l_{n}$  are set to 30 and 60. Padding and truncation are used to keep sequence and word numbers the same. The head number  $l_{h}$  in multi-head self-attention is set to 6. The hidden size  $d^{\prime}$  in the gated aggregation layer is set to 200. The head number in graph neural network  $l_{h}^{\prime}$  is set to 2. The negative sampling ratio  $l_{k}$  is set to 4. When preparing data for graph neural network, we only input sub-graph that contains nodes in the user feedback sequences. Moreover, we pick the maximum 5 neighbor nodes for each node  $r$  in user feedback sequences, which are most frequently co-clicked with  $r$ . We have also released the source code at https://github.com/chungdz/DRPN.
For NRMS, DKN, LSTUR, NPA, and NAML, we use the official code and settings  $^{2}$ . For others, we reimplement them and set their parameters based on the experimental setting strategies reported by their papers.
For fair comparisons, all methods only use the news ID, title, category and subcategory as features. The validation set was used for tuning hyperparameters and the final performance comparison was conducted on the test set.
Models	AUC	MRR	nDCG@5	nDCG@10
LibFM	60.48	26.38	27.75	34.63
DeepFM	62.18	27.26	29.08	35.68
DKN	64.00	28.98	31.49	38.22
LSTUR	65.31	30.31	33.34	39.86
NPA	64.35	29.61	32.88	39.23
DEERS	65.29	30.78	33.78	40.09
DFN	63.11	29.14	31.88	38.33
GERL	64.08	29.34	32.50	38.96
NAML	65.84	30.60	33.89	40.23
NRMS	65.46	30.73	33.78	40.13
NAML + TCE	65.95	30.66	33.93	40.52
NRMS + TCE	65.84	31.58	34.93	41.26
DRPN	67.30	32.68	36.27	42.33
Table 2: Performance comparison of all methods. Best results are highlighted in bold.
The experimental results of all models are summarized in Table 2. We make the following observations from the results. First, our proposed
$^{2}$ https://github.com/microsoft/recommenders
Positive implicit feedback	Negative implicit feedback
Weights	Categories	Titles	Weights	Categories	Titles
■	Sports	The Geno Smith ... thing in Seahawks' win over 49ers.	■	Finance	Confidence in the US economy accelerates.
■	Sports	Russell Wilson has MVP ... beat 49ers in OT classic.	■	Music	Top 100 country songs of all time.
■	Weather	... farmers endure major crop and profit losses as climate changes.	■	Lifestyle	200 shocking home photos you have to see.
■	Sports	Russell Wilson, Richard Sherman swap jackets despite ...	■	Politics	New poll shows Buttigieg in the lead in Iowa. Can he win it all?
■	Movies	Actress accuses Roman Polanski of raping her in 1975.	■	Sports	Watch: John Harbaugh shared awesome moment with Lamar Jackson.
■	Finance	Dean Foods files for bankruptcy.	■	Sports	Angels hire Tony La Russa as baseball operations special assistant.
■	Music	Broadway actress Laurel Griggs dies at Age 13.	■	Sports	Fred Taylor on Pro Football Hall of Fame: 'I think I belong'.
■	Lifestyle	A master suite ... is asking for $1,200/month in rent.	■	Sports	What this week's 5 biggest upsets mean for the NFL playoff picture.
Top 2 recommended news	Last 2 recommended news
Feedbacks	Categories	Titles	Feedbacks	Categories	Titles
Clicked	Sports	Colin Kaepermick is about to get what he deserves: a chance.	Ignored	Moives	... Disney for casting 'the White Guy' in The Little Mermaid.
Clicked	Sports	... after Redskins' player Montae Nicholson took her to hospital.	Ignored	Music	Taylor Swift ... be owed $7.9 million in unpaid royalties.
Figure 4: Visualization of the attention weights for an example user's feedback in the denoising aggregator and the recommendation results for him in validation dataset. A darker color indicates a larger attention weight.
Models	AUC	MRR	nDCG@5	nDCG@10
DRPN	67.30	32.68	36.27	42.33
DRPN-D	66.51	31.09	35.40	41.63
DRPN-G	66.82	31.90	35.34	41.62
DRPN-DG	66.14	31.17	34.57	40.97
DRPN-N	66.38	31.39	34.79	41.13
DRPN-P	65.89	31.11	34.38	40.59
Table 3: Performance comparison of all variants of DRPN. Best results are highlighted in bold.
model, DRPN, outperforms all baselines on the news recommendation datasets. Second, among all baselines, the methods which use the deep neural networks to model the news (i.e., DKN, NPA, LSTUR, DFN, DEERS, NAML, GERL, NRMS) perform better than the feature-based methods (e.g., LibFM and DeepFM). This performance improvement should be attributed to better news representation methods. Among the deep neural methods, NRMS+TCE achieves the best performance by using two level multi-head self-attention to learn user representations and using TCE to denoise the negative samples. Third, among two baselines that use both positive and negative feedback, DFN performs worse than DEERS. The reason may be that original DFN depends on the explicit feedback but the experimental dataset only contains implicit feedback. Compared with NAML, DEERS has a competitive performance even if its news encoder is a simple pooling layer. This also proves the effectiveness of the negative implicit feedback.
To highlight the individual contribution of each module, we use the following variants of DRPN to run an ablation study: (1)  $DRPN-D$ , which removes the denoising aggregator; (2)  $DRPN-G$ , which re
moves the knowledge graph part; (3)  $DRPN-DG$ , which removes the knowledge graph part and the denoising aggregator; (4)  $DRPN-N$ , which only uses the positive feedback; (5)  $DRPN-P$ , which only uses the negative feedback.
The results are shown in Table 3. First, DRPN-D and DRPN-G perform worse than DRPN, proving the effectiveness of the designed denoising module and the collaborative graph. Second, the results of DRPN-N and DRPN-P indicate the effectiveness of negative and positive feedback, respectively. Third, even without deliberately designing, by using both positive and negative implicit feedback, DRPN-DG can achieve competitive performance compared with the strongest baseline NRMS+TCE. This further proves the effectiveness of the negative feedback.
To intuitively illustrate the effectiveness of the denoising aggregator, we sample a user and visualize his historical feedback attention weights in the denoising aggregator of the semantic interest encoder. The upper part of the Figure 4 shows the attention weights and ranks the news in descending order of the attention weight. We can find in positive feedback sequence, the top 4 news are about sports and weather and the last 4 news are about music, movie, finance, and lifestyle. Meanwhile, in negative feedback sequence, the top 4 news are about finance, music, politics, and lifestyle, and the last 4 news are all about sports. This indicates that the denoising aggregator believes that the user likes sports, and dislikes the topics such as finance, music, movies, politics, and lifestyle. As shown in the lower part of Figure 4, based on the predicted user preferences, we can see DRPN prefers to recommend the sports news for this user. Moreover, in the validation data,
we can observe that this user clicks the top 2 recommended news and ignores the last 2 news. It suggests the user preference extracted by the denoising aggregator is consistent with the user's real behaviors. In summary, the visualization results indicate the denoising module can better capture the user's real preferences by conducting the inter- and intra- comparisons between the positive and negative implicit feedback sequences.
In this paper, we propose a novel deep neural news recommendation model DRPN. In DRPN, we design two aggregators to extract user interests from both positive and negative implicit feedback. The content-based aggregator focuses on the contents in the news representations and the denoising aggregator aims to mitigate the noise impact commonly existing in the implicit feedback. Besides, apart from the title information, DRPN also exploits the collaborative information by the graph neural network to further improve the recommendation performance. Experimental results on a large-scale public dataset demonstrate the state-of-the-art performance of DRPN. The further study results also show the effectiveness of the denoising module.
This work was supported by National Key R&D Program of China, No. 2018YFC0117000.
Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019. Neural news recommendation with long- and short-term user representations. In ACL (1), pages 336-345. Association for Computational Linguistics.
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR, abs/1607.06450.
Zhi Bian, Shaojun Zhou, Hao Fu, Qihong Yang, Zhenqi Sun, Junjie Tang, Guiquan Liu, Kaikui Liu, and Xiaolong Li. 2021. Denoising user-aware memory network for recommendation. In RecSys, pages 400-410. ACM.
Abhinandan S Das, Mayur Datar, Ashutosh Garg, and Shyam Rajaram. 2007. Google news personalization: scalable online collaborative filtering. In Proceedings of the 16th international conference on World Wide Web, pages 271-280.
Gianmarco De Francisci Morales, Aristides Gionis, and Claudio Lucchese. 2012. From chatter to headlines: harnessing the real-time web for personalized news recommendation. In Proceedings of the fifth ACM international conference on Web search and data mining, pages 153-162.
Suyu Ge, Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2020. Graph enhanced representation learning for news recommendation. In Proceedings of The Web Conference 2020, pages 2863-2869.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. Deepfm: A factorization-machine based neural network for CTR prediction. In IJCAI, pages 1725-1731. ijcai.org.
Linmei Hu, Chen Li, Chuan Shi, Cheng Yang, and Chao Shao. 2020a. Graph neural news recommendation with long-term and short-term interest modeling. Information Processing & Management, 57(2):102142.
Linmei Hu, Siyong Xu, Chen Li, Cheng Yang, Chuan Shi, Nan Duan, Xing Xie, and Ming Zhou. 2020b. Graph neural news recommendation with unsupervised preference disentanglement. In ACL, pages 4255-4264. Association for Computational Linguistics.
Wouter IJntema, Frank Goossen, Flavius Frasincar, and Frederik Hogenboom. 2010. Ontology-based news recommendation. In Proceedings of the 2010 EDBT/ICDT Workshops, pages 1-6.
Miaomiao Li and Licheng Wang. 2019. A survey on personalized news recommendation technology. IEEE Access, 7:145861-145879.
Rui Liu, Huilin Peng, Yong Chen, and Dell Zhang. 2020. Hypernews: Simultaneous news recommendation and active-time prediction via a double-task deep neural network. In Proceedings of the 29th International Joint Conference on Artificial Intelligence and the 17th Pacific Rim International Conference on Artificial Intelligence (IJCAI-PRICAI), Yokohama, Japan.
Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017. Embedding-based news recommendation for millions of users. In KDD, pages 1933-1942. ACM.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543. ACL.
Zhaopeng Qiu, Yunfan Hu, and Xian Wu. 2022. Graph neural news recommendation with user existing and potential interest modeling. ACM Trans. Knowl. Discov. Data, 16(5).
Zhaopeng Qiu, Xian Wu, and Wei Fan. 2020. Automatic distractor generation for multiple choice questions in standard tests. In COLING, pages 2096-2106. International Committee on Computational Linguistics.
Steffen Rendle. 2012. Factorization machines with libfm. ACM Trans. Intell. Syst. Technol., 3(3):57:1-57:22.
Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjing Wang, and Yu Sun. 2021. Masked label prediction: Unified message passing model for semi-supervised classification. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 1548-1554. International Joint Conferences on Artificial Intelligence Organization.
Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018a. DKN: deep knowledge-aware network for news recommendation. In WWW, pages 1835-1844. ACM.
Menghan Wang, Mingming Gong, Xiaolin Zheng, and Kun Zhang. 2018b. Modeling dynamic missingness of implicit feedback for recommendation. Advances in neural information processing systems, 31:6669.
Wenjie Wang, Fuli Feng, Xiangnan He, Liqiang Nie, and Tat-Seng Chua. 2021. Denoising implicit feedback for recommendation. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 373-381.
Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019a. Neural news recommendation with attentive multi-view learning. In IJCAI, pages 3863-3869. ijcai.org.
Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019b. NPA: neural news recommendation with personalized attention. In KDD, pages 2576-2584. ACM.
Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019c. Neural news recommendation with multi-head self-attention. In EMNLP/IJCNLP (1), pages 6388–6393. Association for Computational Linguistics.
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2020a. User modeling with click preference and reading satisfaction for news recommendation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3023-3029. International Joint Conferences on Artificial Intelligence Organization. Main track.
Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. 2020b. MIND: A large-scale dataset for news recommendation. In ACL, pages 3597-3606. Association for Computational Linguistics.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. 2020c. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems.
Ruobing Xie, Cheng Ling, Yalong Wang, Rui Wang, Feng Xia, and Leyu Lin. 2020. Deep feedback network for recommendation. Proceedings of IJCAI-PRICAI.
Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin. 2018. Recommendations with negative feedback via pairwise deep reinforcement learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1040-1048.
Qiannan Zhu, Xiaofei Zhou, Zeliang Song, Jianlong Tan, and Li Guo. 2019. Dan: Deep attention neural network for news recommendation. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):5973-5980.
In this paper, to better learn the representations, our method refines the historical behaviors of the user by the denoising manner. There are still some potential directions to further improve our approach. First, since the user profile in the experimental dataset only contains the historical behaviors and has no basic information (e.g., gender and age), our current approach doesn't support these features but they are widely used in practice. After these features are ready, we can convert them to embeddings and fuse them with the semantic interest representations obtained by two interest encoders to better represent the user. Second, the news generally contains many forms of features except for the title (such as the cover image and author information) and our approach will explore how to involve more features to better represent the news.
A.2 Potential Risks
Our approach is based on the collaborative filtering, which may lead to that all of recommended news are similar to what the user has seen. This is a common problem faced by the majority of recommender systems. The concentration of a large number of similar information may narrow users' perspective and result in an imbalance on the personal information structure (Li and Wang, 2019). Our method can combine with some rule/human-based strategies (such as popularity based recommendation) to improve the recommendation diversity to alleviate this problem.