SlowGuess/ACL_2020_OCR huggingface dataset:
mance of news recommendation methods on unseen users is slightly lower than that on overlap users which are included in training data. However, the performance on both kinds of users in terms of MRR and nDCG metrics has no significant difference. This result indicates that by inferring user interest from the content of their previously clicked news, the news recommendation models trained on part of users can be effectively applied to the remaining users and new users coming in future.
5.3 News Content Understanding
Next, we explore how to learn accurate news representations from textual content. Since the MIND dataset is quite large-scale, we randomly sampled 100k samples from both training and test sets for the experiments in this and the following sections.
5.3.1 News Representation Model
First, we compare different text representation methods for learning news representation. We select three news recommendation methods which have strong performance, i.e., NAML, LSTUR and NRMS, and replace their original news representation module with different text representation methods, such as LDA, TF-IDF, average of word embed
Figure 3: BERT for news representation.
ding (Avg-Emb), CNN, LSTM and multi-head self-attention (Self-Att). Since attention mechanism is an important technique in NLP (Yang et al., 2016), we also apply it to the aforementioned neural text representation methods. The results are in Table 4.
We have several findings from the results. First, neural text representation methods such as CNN, Self-Att and LSTM can outperform traditional text representation methods like TF-IDF and LDA. This is because the neural text representation models can be learned with the news recommendation task, and they can capture the contexts of texts to generate better news representations. Second, Self-Att and LSTM outperform CNN in news representation. This is because multi-head self-attention and LSTM can capture long-range contexts of words, while CNN can only model local contexts. Third, the attention mechanism can effectively improve the performance of different neural text representation methods such as CNN and LSTM for news recommendation. It shows that selecting important words in news texts using attention can help learn more informative news representations. Another interesting finding is that the combination of LSTM and attention can achieve the best performance. However, to our best knowledge, it is not used in existing news recommendation methods.
5.3.2 Pre-trained Language Models
Next, we explore whether the quality of news representation can be further improved by the pretrained language models such as BERT (Devlin et al., 2019), which have achieved huge success in different NLP tasks. We applied BERT to the news representation module of three state-of-the-art news recommendation methods, i.e., NAML, LSTUR and NRMS. The results are summarized in Fig. 3. We find that by replacing the origi
AUC	MRR	nDCG@5	nDCG@10
Title	66.22	31.92	34.53	40.23
Abs.	64.17	30.49	32.81	38.57
Body	66.32	31.88	34.42	40.22
Title + Abs. + Body (Con)	67.07	32.34	34.98	40.74
Title + Abs. + Body + Cat. (Con)	67.09	32.40	35.03	40.80
Title + Abs. + Body + Cat. + Ent. (Con)	67.23	32.41	35.04	40.83
Title + Abs. + Body (AMV)	67.38	32.37	35.12	40.79
Title + Abs. + Body + Cat. (AMV)	67.50	32.43	35.21	40.96
Title + Abs. + Body + Cat. + Ent. (AMV)	67.60	32.51	35.24	41.03
Table 5: News representation with different news information. "Abs.", "Cat." and "Ent." mean abstract, category and entity, respectively.
nal word embedding module with the pre-trained BERT model, the performance of different news recommendation methods can be improved. It shows the BERT model pre-trained on large-scale corpus like Wikipedia can provide useful semantic information for news representation. We also find that fine-tuning the pre-trained BERT model with the news recommendation task can further improve the performance. These results validate that the pre-trained language models are very helpful for understanding news articles.
5.3.3 Different News Information
Next, we explore whether we can learn better news representation by incorporating more news information such as abstract and body. We try two methods for news text combination. The first one is direct concatenation (denoted as  $Con$ ), where we combine different news texts into a long document. The second one is attentive multi-view learning (denoted as  $AMV$ ) (Wu et al., 2019a) which models each kind of news text independently and combines them with an attention network. The results are shown in Table 5. We find that news bodies are more effective than news titles and abstracts in news representation. This is because news bodies are much longer and contain richer information of news content. Incorporating different kinds of news texts such as title, body and abstract can effectively improve the performance of news recommendation, indicating different news texts contain complementary information for news representation. Incorporating the category label and the entities in news texts can further improve the performance. This is because category labels can provide general topic information, and the entities are keywords to understand the content of news. Another finding is that the attentive multi-view learning method is better than direct text combination in incorporating different news texts. This is because different news texts usually have different characteristics, and it is better
AUC	MRR	nDCG@5	nDCG@10
Average	65.22	31.22	33.66	39.39
Attention	66.17	31.94	34.52	40.24
Candidate-Att	66.01	31.62	34.20	39.87
GRU	66.37	31.99	34.59	40.33
LSTUR	66.44	32.00	34.57	40.31
Self-Att	66.91	32.48	35.12	40.85
Table 6: Different user modeling methods.
to learn their representations using different neural networks and model their different contributions using attention mechanisms.
5.4 User Interest Modeling
Most of the state-of-the-art news recommendation methods infer users' interest in news from their previously clicked news articles (Wu et al., 2019c; An et al., 2019). In this section we study the effectiveness of different user interest modeling methods. We compare 6 methods, including simple average of the representations of previously clicked news (Average), attention mechanism used in (Wu et al., 2019a) (Attention), candidate-aware attention used in (Wang et al., 2018) (Candidate-Att), gated recurrent unit used in (Okura et al., 2017) (GRU), long- and short-term user representation used in (An et al., 2019) (LSTUR) and multi-head self-attention used in (Wu et al., 2019c) (Self-Att). For fair comparison, the news representations in these methods are all generated using LSTM. The results are shown in Table 6.
We find that Attention, Candidate-Att, and GRU all perform better than Average. This is because Attention can select informative behaviors to form user representations, Candidate-Att can incorporate the candidate news information to select informative behaviors, and GRU can capture the sequential information of behaviors. LSTUR performs better than all above methods, because it can model both long-term and short-term user interest using behaviors in different time ranges. Self-Att can also achieve strong performance, because it can model the long-range relatedness between the historical behaviors of users for better user modeling.
We also study the influence of click history length on user interest modeling. In Fig. 4 we show the performance of three news recommendation methods, i.e., LSTUR, NAML and NRMS, on the users with different lengths of news click history. We find that their performance in general improves on the users with more news click records. This
Figure 4: Users with different numbers of clicked news.
result is intuitive, because more news click records can provide more clues for user interest modeling. The results also show it is quite challenging to infer the interest of users whose behaviors on the news platform are scarce, i.e., the cold-start users.
In this paper we present the MIND dataset for news recommendation research, which is constructed from user behavior logs of Microsoft News. It contains 1 million users and more than 160k English news articles with rich textual content such as title, abstract and body. We conduct extensive experiments on this dataset. The results show the importance of accurate news content understanding and user interest modeling for news recommendation. Many natural language processing and machine learning techniques such as text modeling, attention mechanism and pre-trained language models can contribute to the performance improvement of news recommendation.
In the future, we plan to extend the MIND dataset by incorporating image and video information in news as well as news in different languages, which can support the research of multi-modal and multi-lingual news recommendation. In addition, besides the click behaviors, we plan to incorporate other user behaviors such as read and engagement to support more accurate user modeling and performance evaluation. Many interesting researches can be conducted on the MIND dataset, such as designing better news and user modeling methods, improving the diversity, fairness and explainability of news recommendation results, and exploring privacy-preserving news recommendation. Besides news recommendation, the MIND dataset can also be used in other natural language processing tasks such as topic classification, text summarization and news headline generation.
We would like to thank Xi Chen, Virl Hill, Jesse Pannoni, Sally Salas and Ting Cai in the Microsoft News team for their great support in releasing this dataset and for their great help in preparing the data. We also want to thank Le Zhang, Miguel Gonzalez-Fierro and Tao Wu at Microsoft Azure for their support in Microsoft Recommenders repository and Azure resource. Finally, we thank Jingwei Yi and Ling Luo for their help on experiments.
Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019. Neural news recommendation with long-and short-term user representations. In ACL, pages 336-345.
Trapit Bansal, Mrinal Das, and Chiranjib Bhattacharyya. 2015. Content driven user profiling for comment-worthy recommendations of news and blog articles. In RecSys., pages 195-202. ACM.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In NIPS, pages 2787-2795.
Cheng Chen, Xiangwu Meng, Zhenghua Xu, and Thomas Lukasiewicz. 2017. Location-aware personalized news recommendation with deep semantic analysis. IEEE Access, 5:1624-1638.
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016. Wide & deep learning for recommender systems. In DLRS, pages 7-10. ACM.
Abhinandan S Das, Mayur Datar, Ashutosh Garg, and Shyam Rajaram. 2007. Google news personalization: scalable online collaborative filtering. In WWW, pages 271-280. ACM.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*, pages 4171–4186.
Florent Garcin, Christos Dimitrakakis, and Boi Faltings. 2013. Personalized news recommendation with context trees. In RecSys., pages 105-112.
Suyu Ge, Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2020. Graph enhanced representation learning for news recommendation. In WWW, pages 2863-2869.
Jon Atle Gulla, Lemei Zhang, Peng Liu, Ozlem Ozgöbek, and Xiaomeng Su. 2017. The adressa dataset for news recommendation. In Proceedings of the international conference on web intelligence, pages 1042-1048. ACM.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. Deepfm: a factorization-machine based neural network for ctr prediction. In AAAI, pages 1725-1731. AAAI Press.
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In CIKM, pages 2333-2338.
Ilija Ilievski and Sujoy Roy. 2013. Personalized news recommendation based on implicit feedback. In Proceedings of the 2013 international news recommender systems workshop and challenge, pages 10-15. ACM.
Margarita Karkali, Dimitris Pontikis, and Michalis Vazirgiannis. 2013. Match the news: Afirefox extension for real-time news recommendation. In SI-GIR, pages 1117-1118. ACM.
Benjamin Kille, Frank Hopfgartner, Torben Brodt, and Tobias Heintz. 2013. The plista dataset. In Proceedings of the 2013 international news recommender systems workshop and challenge, pages 16-23. ACM.
Michal Kompan and Mária Bieliková. 2010. Content-based news recommendation. In International conference on electronic commerce and web technologies, pages 61-72. Springer.
Yehuda Koren. 2008. Factorization meets the neighborhood: a multifaceted collaborative filtering model. In KDD, pages 426-434.
Lei Li, Ding-Ding Wang, Shun-Zhi Zhu, and Tao Li. 2011. Personalized news recommendation: a review and an experimental investigation. Journal of computer science and technology, 26(5):754.
Lihong Li, Wei Chu, John Langford, and Robert E Schapire. 2010. A contextual-bandit approach to personalized news article recommendation. In WWW, pages 661-670. ACM.
Jianxun Lian, Fuzheng Zhang, Xing Xie, and Guangzhong Sun. 2018. Towards better representation learning for personalized news recommendation: a multi-channel deep fusion approach. In *IJ-CAI*, pages 3805–3811.
Jiahui Liu, Peter Dolan, and Elin Rønby Pedersen. 2010. Personalized news recommendation based on click behavior. In IUI, pages 31-40. ACM.
Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017. Embedding-based news recommendation for millions of users. In KDD, pages 1933-1942. ACM.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad:  $100,000+$  questions for machine comprehension of text. In EMNLP, pages 2383-2392.
Steffen Rendle. 2012. Factorization machines with libfm. TIST, 3(3):57:1-57:22.
Jeong-Woo Son, A Kim, Seong-Bae Park, et al. 2013. A location-based news article recommendation with explicit localized semantic analysis. In SIGIR, pages 293-302. ACM.
Gabriel de Souza Pereira Moreira, Felipe Ferreira, and Adilson Marques da Cunha. 2018. News session-based recommendations using deep neural networks. In DLRS, pages 15-23. ACM.
Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018. Dkn: Deep knowledge-aware network
for news recommendation. In WWW, pages 1835-1844.
Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019a. Neural news recommendation with attentive multiview learning. In *IJCAI-19*, pages 3863–3869.
Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019b. Npa: Neural news recommendation with personalized attention. In KDD, pages 2576-2584. ACM.
Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019c. Neural news recommendation with multi-head self-attention. In EMNLP-IJCNLP, pages 6390-6395.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In NAACL, pages 1480-1489.
3606