SlowGuess/ACL_2020_OCR huggingface dataset:
Neural Recommendation Methods: Neural networks specially designed for news recommendation, including (1)  $DFM$  (Lian et al., 2018), a deep fusion model combining dense layers with different depths and using attention mechanism to select important features; (2)  $DKN$  (Wang et al., 2018), incorporating entity information in knowledge graphs with Kim CNN (Kim, 2014) to learn news representations and using news-level attention network to learn user representations; (3)  $GRU$  (Okura et al., 2017), using auto-encoders to represent news and a GRU network to represent users; (4)  $NRMS$  (Wu et al., 2019e), leveraging multi-head self-attention for news and user representation learning; (5) Hi-Fi Ark (Liu et al., 2019), summarizing user history into highly compact and complementary vectors as archives, and learning candidate-dependent user
Table 1: Statistics of the dataset.
Methods	AUC	MRR	NDCG@5	NDCG@10
LibFM	0.5661	0.2414	0.2689	0.3552
DSSM	0.5949	0.2675	0.2881	0.3800
Wide&Deep	0.5812	0.2546	0.2765	0.3674
DeepFM	0.5830	0.2570	0.2802	0.3707
DFM	0.5861	0.2609	0.2844	0.3742
DKN	0.6032	0.2744	0.2967	0.3873
GRU	0.6102	0.2811	0.3035	0.3952
NRMS	0.6275	0.2985	0.3217	0.4139
Hi-Fi Ark	0.6027	0.3162	0.3335	0.4204
NPA	0.6243	0.3321	0.3535	0.4380
FIM	0.6359*	0.3354*	0.3582*	0.4436*
FIMfirst	0.6258	0.3266	0.3484	0.4348
FIMlast	0.6319	0.3323	0.3549	0.4407
Table 2: The performance of different methods on news recommendation. The best and second best results are highlighted in boldface and underlined respectively. \*The improvement over all baseline methods is significant at  $p$  -value  $<  0.05$
representation via attentive aggregation of such archives; (6) NPA (Wu et al., 2019b), using personalized attention with user ID's embedding as the query vector to select important words and news.
Ablation Variants: To verify the effects of multi-grained representation and sequential matching, we further setup two comparing ablation models, i.e., (1)  $FIM_{first}$ : a variant in which we use feature maps of the first news representation layer for matching and recommendation. In this scenario, the HDC module degenerates into a one-layer standard CNN encoder. (2)  $FIM_{last}$ : a variant using the outputs of the last layer in HDC (namely, the  $L$ -th embedding type) to represent each news for matching. Due to the hierarchical representation architecture, higher-level features synthesize information from lower-level features, and can model more complex lexical and semantic clues.
Table 2 shows the results of our model and all comparative methods. Several observations can be made. First, neural news recommendation methods (e.g., GRU, NRMS, Hi-Fi Ark, NPA) are generally better than traditional methods (e.g., LibFM, DeepFM) that are based on manual feature engineering. The reason might be that handcrafted features are usually not optimal, and deep neural networks take the advantages of extracting implicit semantic features and modeling latent relationships between user and news representations.
Second, our model FIM consistently outperforms other baselines in terms of all metrics, including the state-of-the-art deep learning based mod
(a) Negative sampling ratio  $K$
(b) 3D CNN hierarchy for image  $\mathbf{Q}$
(c) Hierarchy of HDC news encoder
(d) Incorporating two-level category annotations
Figure 4: Performances w.r.t. different hyper-parameters and input information.
els. This validates the advantage of the pair-wise multi-level matching architecture in synthetically detecting fine-grained matching information from news segment pairs to predict the probability of a user clicking a candidate news.
Third, both  $\mathrm{FIM}_{\text {first }}$  and  $\mathrm{FIM}_{\text {last }}$  show a decrease of performance compared to FIM. The latter is better than the former, indicating the effectiveness of constructing higher-level representations on the basis of low levels via the hierarchical mechanism of HDC. Besides, compared with DKN that utilizes knowledge-enhanced CNNs to learn news representations,  $\mathrm{FIM}_{\text {first }}$  has a better performance, illustrating the advantage of pair-wise matching fashion. Another notable thing is that while  $\mathrm{FIM}_{\text {last }}$  underperforms FIM, it can outperform all other competitors on all metrics. However, the benefit of interacting news pairs at multigrained semantic levels is still significant.
In this section, we further investigate the impacts of different parameters and inputs on the model performance, and discuss the contribution of multi-grained representation and matching architecture.
5.1 Quantity & Input Analysis
We first study how FIM performs with different negative sampling ratio  $K$ . Figure 4(a) shows the experimental results. We can find that the performance consistently improves when  $K$  is lower than 5, then begins to decline. The possible reason is that with a too small  $K$ , the useful information exploited from negative samples is limited. However, when too many negative samples are incorporated, they may become dominant and the imbalance of training data will be increased. Thus it is more difficult for the model to precisely recognize the positive samples, which will also affect the recommendation performance. Overall, the optimal setting of  $K$  is moderate (e.g.,  $K = 4$ ).
We then explore the influence of the 3D convolution & max-pooling neural network for processing the matching image  $\mathbf{Q}$ . Comparing results are illustrated in Figure 4(b), where the CNN hierarchy  $a\_ b$  means that the number of filters for the first layer and the second layer are set to  $a$  and  $b$ , separately. As shown, given the filter number  $a$  for the first layer, the performance first increases with a larger filter number  $b$  for the second layer, since more high-order information can be extracted. Then the performance begins to decrease, possibly because
(a)  $\mathbf{M}_1$
(b)  $\mathbf{M}_2$
(c)  $\mathbf{M}_3$
Figure 5: Matching matrices visualization, darker area means larger value.
more noisy patterns are introduced to the model (e.g., the group of [32_8, 32_16, 32_32]). Besides, a similar trend exists in the hierarchies with the same value  $b$  and different value  $a$  (e.g., the group of [16_8, 32_8, 64_8]). We conduct other experiments by changing the window size in [2,3,4,5] and the number of convolution layers in [1,2,3]. Results show that the optimal hierarchy is two-layered CNNs, with  $32 \times [3,3,3]$  filters for the first layer and  $16 \times [3,3,3]$  filters for the second layer.
We further compare different combinations of the number of dilated convolution filters and stacked layers in the HDC news representation module. Figure 4(c) demonstrates the results, where darker areas represent larger values. We observe a consistent trend over settings with different number of filters at each layer, i.e., there is a significant improvement during the first few stacked layers, and then the performance decreases a lot when the depth grows to 5. The results indicate that depth of representation layers indeed matters in terms of matching and recommendation accuracy. The optimal setting of the number of stacked layers and convolution filters is 3 and 150 respectively. We think the reason might be that in this scenario, the perceived field of dilated convolution filters at each layer ranges among [3-7-13] (with dilation rates as [1-2-3]), which is sufficient for modeling multi-grained n-gram features through hierarchical composition of local interactions, compared to the average length of news word sequences.
We also investigate the effectiveness of incorporating two-level category annotations of news as inputs. The results are shown in Figure 4(d). We can find that incorporating either categories or subcategories can benefit the performance of our model. This is interpretable since category annota
tions are helpful to reveal user's interested aspects more explicitly. In addition, enhancing news representations with subcategories is better than with categories. This is probably because compared to the general category labels, subcategories can provide more concrete and detailed information to indicate the core topic of news content. Overall, jointly incorporating the two-level category annotations can achieve the best performance.
5.2 Visualization
In this subsection, we further study the effectiveness of constructing hierarchical news representations and performing multi-grained interest matching. Figure 5 gives visualizations of the multi-grained matching matrices (defined as formula 2) between historical browsed news and candidate news for a user, where  $\mathbf{M}_l$  denotes a matching matrix of a news pair at the  $l$ -th representation level. We observe that the important matching information captured by the 1st-level matching matrix is mainly lexical relevance. For example, the words "football", "nfl", "playoff", "playoffs" and "quarterbacks" are more correlated and assigned higher matching values in  $\mathbf{M}_1$ , which may due to their similar co-occurrence information encoded in word embeddings. Differently, higher-level matching matrices have the ability to identify more sophisticated semantic structures and latent long-term dependencies. From Figure 5(b), the interactive areas between the segments "weight loss" in the candidate news and "lost pounds" in the browsed news significantly gain larger matching scores among the 2-nd level semantic representations. In the matching matrix  $\mathbf{M}_3$  in Figure 5(c), the subsequences about "trump walks out" are distinguished, since the expressions have correlated meanings. Mean
while, the results also indicate that our model has the ability to identify important segments of a sentence and ignore the parts with less information, which is helpful to capture user's interested topics or events more accurately.
In this paper, we propose a new architecture for neural news recommendation based on multi-grained representation and matching. Different from previous work that first integrates user's reading history into a single representation vector and then matches the candidate news representation, our model can capture more fine-grained interest matching signals by performing interactions between each pair of news at multi-level semantic granularities. Extensive experiments on a real-world dataset collected from MSN news show that our model significantly outperforms the state-of-the-art methods. In the future, we will do more tests and surveys on the improvement of business objectives such as user experience, user engagement and service revenue.
Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019. Neural news recommendation with long- and short-term user representations. In ACL, pages 336-345.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Trapit Bansal, Mrinal Das, and Chiranjib Bhattacharyya. 2015. Content driven user profiling for comment-worthy recommendations of news and blog articles. In RecSys, pages 195-202.
Andrew P Bradley. 1997. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern recognition, 30(7):1145-1159.
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016. Wide & deep learning for recommender systems. In DLRS, pages 7-10.
Abhinandan S Das, Mayur Datar, Ashutosh Garg, and Shyam Rajaram. 2007. Google news personalization: scalable online collaborative filtering. In WWW, pages 271-280.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. Deepfm: A factorization-machine based neural network for CTR prediction. In IJCAI, pages 1725-1731.
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013a. Learning deep structured semantic models for web search using clickthrough data. In CIKM, pages 2333-2338.
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry P. Heck. 2013b. Learning deep structured semantic models for web search using clickthrough data. In CIKM, pages 2333-2338.
Wouter IJntema, Frank Goossen, Flavius Frasincar, and Frederik Hogenboom. 2010. Ontology-based news recommendation. In EDBT/ICDT Workshops, page 16.
Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422-446.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP, pages 1746-1751.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Talia Lavie, Michal Sela, Ilit Oppenheim, Ohad Inbar, and Joachim Meyer. 2010. User attitudes towards news content personalization. International journal of human-computer studies, 68(8):483-495.
Lei Li, Li Zheng, Fan Yang, and Tao Li. 2014. Modeling and broadening temporal user interest in personalized news recommendation. Expert Systems with Applications, 41(7):3168-3177.
Lihong Li, Wei Chu, John Langford, and Robert E Schapire. 2010. A contextual-bandit approach to personalized news article recommendation. In WWW, pages 661-670.
Jianxun Lian, Fuzheng Zhang, Xing Xie, and Guangzhong Sun. 2018. Towards better representation learning for personalized news recommendation: a multi-channel deep fusion approach. In *IJ-CAI*, pages 3805–3811.
Jiahui Liu, Peter Dolan, and Elin Rønby Pedersen. 2010. Personalized news recommendation based on click behavior. In IUI, pages 31-40.
Zheng Liu, Yu Xing, Fangzhao Wu, Mingxiao An, and Xing Xie. 2019. Hi-fi ark: Deep user representation via high-fidelity archive network. In *IJCAI*, pages 3059-3065.
Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In ICML, pages 807-814.
Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017. Embedding-based news recommendation for millions of users. In KDD, pages 1933-1942.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543.
Owen Phelan, Kevin McCarthy, Mike Bennett, and Barry Smyth. 2011. Terms of a feather: Content-based news recommendation and discovery using twitter. In ECIR, pages 448-459.
Owen Phelan, Kevin McCarthy, and Barry Smyth. 2009. Using twitter to recommend real-time topical news. In RecSys, pages 385-388.
Steffen Rendle. 2012. Factorization machines with libfm. ACM Transactions on Intelligent Systems and Technology (TIST), 3(3):57.
Jeong-Woo Son, A Kim, Seong-Bae Park, et al. 2013. A location-based news article recommendation with explicit localized semantic analysis. In SIGIR, pages 293-302.
Ellen M Voorhees et al. 1999. The trec-8 question answering track report. In Trec, volume 99, pages 77-82.
Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018. Dkn: Deep knowledge-aware network for news recommendation. In WWW, pages 1835-1844.
Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019a. Neural news recommendation with attentive multiview learning. In *IJCAI*, pages 3863-3869.
Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019b. Npa: Neural news recommendation with personalized attention. In KDD, pages 2576-2584.
Chuhan Wu, Fangzhao Wu, Mingxiao An, Yongfeng Huang, and Xing Xie. 2019c. Neural news recommendation with topic-aware news representation. In ACL, pages 1154-1159.
Chuhan Wu, Fangzhao Wu, Mingxiao An, Tao Qi, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019d. Neural news recommendation with heterogeneous user behavior. In EMNLP, pages 4873-4882.
Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019e. Neural news recommendation with multi-head self-attention. In EMNLP-IJCNLP, pages 6390-6395.
Fisher Yu and Vladlen Koltun. 2016. Multi-scale context aggregation by dilated convolutions. In ICLR.
Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. 2018. Drn: A deep reinforcement learning framework for news recommendation. In WWW, pages 167-176.
Xiangyang Zhou, Lu Li, Daxiang Dong, Yi Liu, Ying Chen, Wayne Xin Zhao, Dianhai Yu, and Hua Wu. 2018. Multi-turn response selection for chatbots with deep attention matching network. In ACL, pages 1118-1127.