SlowGuess/ACL_2020_OCR huggingface dataset:
Datasets. We conduct experiments on the real-world online news datasets Adressa (Gulla et al., 2017) from a Norwegian news portal to evaluate our model. We use two datasets named Adressa-1week and Adressa-10week, which respectively collect news click logs as long as 1 week and 10 weeks. Following DAN (Zhu et al., 2019), we just select user id, news id, time-stamp, the title and profile of news to build our datasets, and preprocess the data by removing the stopwords in the news content. The statistics of our final datasets are shown in Table 1.
For the Adressa-1week dataset, we use the first 5 days' historical data for the construction of usernews bipartite graph. The 6-th day's is used to build training samples:  $\{(u,d)\}$ .  $20\%$  randomly sampled from the last day's are for validation and the remaining are regarded as test set. Note that during testing, we reconstruct the graph with all the previous 6 days' historical data. Similarly, for the Adressa-10week dataset, we construct the graph with the first 50 days' data, the following 10 days are served to generate training pairs,  $20\%$  of the last 10 days' for validation data and  $80\%$  for test. Note that, for the baselines, we also use the data from the first 5 (50) days for constructing user's historical data, the following 1 (10) days is used to generate training pairs. The validation and test set constructed with the last 1 (10) days are also the same for all the models.
Experimental settings. In our experiments, the dimension of word/entity embeddings and entity type embeddings is set as  $n_1 = n_2 = 50$ , and the dimension of input user and news embeddings  $l_{in}$  is set as 128. The embeddings of words, entities, entity types and users are randomly initialized with a Gaussian distribution  $\mathcal{N}(0,0.1)$ . In our methods, due to the large scale of the datasets, we sample a fixed-size set of neighbors (size = 10) for a user, and we set size = 30 for a news, according to the average degree of users and news respectively. The number of latent preference factors is  $K = 7$  and the dimension of each disentangled subspace is  $\Delta n = 16$ . The number of graph convolution layers is set to 2. The dropout rate is 0.5. The balance coefficient  $\lambda$  is set as 0.004. We test our model with different value of  $\lambda$  ranging from 0.001
Number	* 1week	* 10week
# users	537,629	590,674
# news	14,732	49,994
# clicks	2,107,312	15,127,204
# vocabulary	116,603	279,214
# entity-type	11	11
# average words	4.03	4.10
# average entities	22.11	21.29
to 0.02 (with step 0.001) and find that our model is insensitive to  $\lambda$  in [0.001, 0.02]. Finally, Adam (Kingma and Ba, 2014) is applied for model optimization, and the learning rate is 0.0005. The batch size is set to 128. These hyper-parameters were all selected according to the results on validation set.
It is worth noting that our model can deal with new coming news documents that have not previously existed in the user-news interaction graph  $\mathcal{G}$  during training or testing. Our model takes these news documents as isolated nodes in the graph  $\mathcal{G}$ . Their representations are based on only content feature  $h_d$  without neighbor aggregation, and can also be disentangled via Eq. 3.
We evaluate the performance of our model GNUD by comparing it with the following state-of-the-art baseline methods:
LibFM (Rendle, 2012), a feature-based matrix factorization method, with the concatenation of TF-IDF vectors of news title and profile as input.
CNN (Kim, 2014), applying two parallel CNNs to word sequences in news titles and profiles respectively and concatenate them as news features. The user representation is learned from the user's news history.
DSSM (Huang et al., 2013), a deep structured semantic model. In our experiments, we model the user's clicked news as the query and the candidate news as the documents.
Wide & Deep (Cheng et al., 2016), a deep model for recommendation which combines a (Wide) linear model and (Deep) feed-forward neural network. We also use the concatenation of news title and profile embeddings as features.
DeepFM (Guo et al., 2017), a general model that combines factorization machines and deep neural networks that share the input. We use the same input as Wide & Deep for DeepFM.
<sup>2</sup> http://reclabidi.ntnu.no/dataset/
Methods	Adressa-1week	Adressa-10week
AUC	F1	AUC	F1
LibFM	61.20±1.29	59.87±0.98	63.76±1.05	62.41±0.72
CNN	67.59±0.94	66.33±1.44	69.07±0.95	67.78±0.69
DSSM	68.61±1.02	69.92±1.13	70.11±1.35	70.96±1.56
Wide&Deep	68.25±1.12	69.32±1.28	73.28±1.26	69.52±0.83
DeepFM	69.09±1.45	61.48±1,31	74.04±1.69	65.82±1.18
DMF	55.66±0.84	56.46±0.97	53.20±0.89	54.15±0.47
DKN	75.57±1.13	76.11±0.74	74.32±0.94	72.29±0.41
DAN	75.93±1.25	74.01±0.83	76.76±1.06	71.65±0.57
GNewsRec	81.16±1.19	82.85±1.15	78.62±1.38	81.01±0.64
GNUD w/o Disen	78.33±1.29	79.09±1.22	78.24±0.13	80.58±0.45
GNUD w/o PR	83.12±1.53	81.67±1.56	80.61±1.07	80.92±0.31
GNUD	84.01±1.16	83.90±0.58	83.21±1.91	81.09±0.23
Table 2: The performance of different methods on news recommendation.
DMF (Xue et al., 2017), a CF based deep matrix factorization model without considering the news content.
DKN (Wang et al., 2018), a deep content based news recommendation framework fusing semantic-level and knowledge-level representations. We model the news title and profile as semantic-level and knowledge-level representations, respectively.
DAN (Zhu et al., 2019), a deep attention neural network for news recommendation which can capture the dynamic diversity of news and user's interests, and consider the users' click sequence information.
GNewsRec (Hu et al., 2020), a graph neural network based method combining long-term and short term interest modeling for news recommendation.
All the baselines are initialized as the corresponding papers, and in terms of neural network models we use the same word embedding dimension for fair comparison. Then they are carefully tuned to achieve their optimal performance. We independently repeat each experiment for 10 times and report the average performance.
Result analysis. The comparisons between different methods are summarized in Table 2. We can observe that our proposed model GNUD consistently outperforms all the state-of-the-art baseline methods on both datasets. GNUD improves the best deep neural models DKN and DAN more than  $6.45\%$  on AUC and  $7.79\%$  on F1 on both datasets. The main reason is that our model fully exploits the high-order structure information in the user-news interaction graph, learning better representations of users and news. Compared to the best-performed
baseline method GNewsRec, our model GNUD achieves better performance on both datasets in terms of both AUC  $(+2.85\%)$  and  $+4.59\%$  on the two datasets, respectively) and F1  $(+1.05\%)$  and  $+0.08\%$  , respectively). This is because that our model considers the latent preference factors that cause the user-news interactions and learns representations that uncover and disentangle these latent preference factors, which enhance expressiveness.
From Table 2, we can also see that all the content-based methods outperform the CF based model DMF. This is because CF based methods suffer a lot from cold-start problem since most news are new coming. Except for DMF, all the deep neural network based baselines (e.g., CNN, DSSM Wide&Deep, DeepFM, etc.) significantly outperform LibFM, which shows that deep neural models can capture more implicit but informative features for user and news representations. DKN and DAN further improve other deep neural models by incorporating external knowledge and applying a dynamic attention mechanism.
Comparison of GNUD variants. To further demonstrate the efficacy of the design of our model GNUD, we compare among the variants of our model. As we can see from the last three lines in Table 2, when the preference disentanglement is removed, the performance of the model GNUD w/o Disen (GNUD without preference disentanglement) drops largely by  $5.68\%$  and  $4.97\%$  in terms of AUC on the two datasets  $(4.81\%$  and  $0.51\%$  on F1), respectively. This observation demonstrates the effectiveness and necessity of preference disentangled representations of users and news. Com
Figure 3: Visualization of a user's clicked news which belong to different disentangled subspaces w.r.t. different preference factors. We use six keywords (translated into English) to illustrate a news.
News	Keywords
d1	norway oljebransjen (Norway oil industry), nordschehavet (Norwegian sea), helgeldanskysten (Helgeland coast), hygen (hygen), energy (energy), trondheim (a city)
d2	Statkraft (State Power Corporation of Norway), trønderenergi (tronder energy), snillfjord (snill fjord), trondheimfsjorden (trondheim fjord), vindkraft (wind power), energy (energy)
d3	Bolig (residence), hage (garden), hjemme (home), fossen (waterfall), hus (house), home (home)
d4	health-and-fitness (health and fitness), mørk sjokolade (dark chocolate), vitaminrike (vitamin), olivenolje (olive oil), grønssaker (vegetables), helse (health)
(a) # Preference factors.
(b) # Routing Iterations.
Figure 4: Influence of different number of preference factors and routing iterations.
pared to GNUD w/o PR (GNUD without preference regularizer), we can see that introducing the preference regularizer which enforces each disentangled embedding subspace independently reflect an isolated preference, can bring performance gains on both AUC  $(+0.89\%)$  and  $+2.6\%$ , respectively) and F1  $(+2.23\%)$  and  $+0.17\%$ , respectively).
To intuitively demonstrate the efficacy of our model, we randomly sample a user  $u$  and extract her logs from the test set. The representation of user  $u$  is disentangled into  $K = 7$  subspaces and we randomly sample 2 subspaces. For each one, we visualize the top news that user  $u$  pay most attention to (with the probability  $r_{d,k}$  larger than a threshold). As shown in Figure 3, different subspaces relect different preference factors. For example, one subspace (shown in blue) is related to "energy" as the top two news contain the keywords such as "oil industry", "hygen" and "wind power". The other subspace (shown in green) may indicate the latent preference factor about "healthy diet" as the related news contain the keywords such as "health", "vitamin" and "vegetables". The news  $d_3$  about home has low probability in the both subspaces. It does not belong to any of the two preferences.
Methods	Adressa-1week	Adressa-10week
AUC	F1	AUC	F1
GNUD-1	80.96	79.86	82.22	80.61
GNUD-2	84.01	83.90	83.21	81.09
GNUD-3	84.03	82.18	83.05	80.93
Table 3: The performance of GNUD with different layer numbers.
5.4 Parameter Analysis
In this section, we examine how different choices of some hyper-parameters affect the performance of GNUD.
Analysis of layer numbers. We investigate whether GNUD can benefit from multiple embedding propagation layers. We vary the layer numbers in the range of  $\{1, 2, 3\}$  on both datasets. As we can see in Table 3, GNUD-2 (2 layers) is superior to others. The reason is that GNUD-1 considers the first-order neighbors only, while using over 2 layers may lead to overfitting, which indicates that applying a too deep architecture might bring noise to the representations in news recommendation task. Therefore, GNUD-2 is regarded as the most suitable choice.
Number of latent preference factors. We fix the dimension of each latent preference subspace as 16 and check the impact of the number  $K$  of latent preference factors. As shown in Figure 4 (a), we can find that with the increase of  $K$ , the performance first grows, reaching the best at  $K = 7$ , and then begins to drop. Thus we set  $K = 7$  in our experiments.
Number of routing iterations. We study the performance with different number of routing iterations. As shown in Figure 4 (b), we can see that our model generally gets better performance with more routing iterations and finally achieves convergence after 7 iterations.
In this paper, we consider the high-order connectivity as well as the latent preference factors underlying the user-news interactions, and propose a novel graph neural news recommendation model GNUD with unsupervised preference disentangle-ment. Our model regards the user-news interactions as a bipartite graph and encode high-order relationships among users and news by graph convolution. Furthermore, the learned representations are disentangled with different latent preference factors by a neighborhood routing mechanism, enhancing expressiveness and interpretability. A preference regularizer is also designed to force each disentangled subspace to independently reflect an isolated preference, further improving the quality of user and news embeddings. Experimental results on realworld news datasets demonstrate that our model achieves significant performance gains compared to state-of-the-art methods, supporting the importance of exploiting the high-order connectivity and disentangling the latent preference factors in user and news representations.
This work is supported by the National Natural Science Foundation of China (No. U1936220, 61806020, 61772082, 61972047, 61702296), the National Key Research and Development Program of China (2018YFB1402600), the CCF-Tencent Open Fund, and the Fundamental Research Funds for the Central Universities. We also acknowledge the valuable comments from Jianxun Lian at Microsoft Research Asia.
Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019. Neural news recommendation with long-and short-term user representations. In ACL, pages 336-345.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828.
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016. Wide & deep learning for recommender systems. In DLRS, pages 7-10.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. 2018. Unsupervised representation learning by predicting image rotations. In ICLR.
Jon Atle Gulla, Lemei Zhang, Peng Liu, Ozlem Ozgöbek, and Xiaomeng Su. 2017. The adressa dataset for news recommendation. In WI, pages 1042-1048.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. Deepfm: a factorization-machine based neural network for ctr prediction. In IJCAI, pages 1725-1731.
Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In NIPS, pages 1024-1034.
Irina Higgins, Loic Matthew, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR.
Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. 2018. Learning to decompose and disentangle representations for video prediction. In NIPS, pages 517-526.
Linmei Hu, Chen Li, Chuan Shi, Cheng Yang, and Chao Shao. 2020. Graph neural news recommendation with long-term and short-term interest modeling. Information Processing Management, 57(2):102142.
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In CIKM, pages 2333-2338.
Wouter IJntema, Frank Goossen, Flavius Frasincar, and Frederik Hogenboom. 2010. Ontology-based news recommendation. In EDBT/ICDT Workshops, page 16.
Hyunjik Kim and Andriy Mnih. 2018. Disentangling by factorising. In ICML, pages 2654-2663.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP, pages 1746-1751.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. arXiv preprint arXiv:1312.6114.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. In *ICLR*.
Jiahui Liu, Peter Dolan, and Elin Rønby Pedersen. 2010. Personalized news recommendation based on click behavior. In IUI, pages 31-40.
Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. 2019a. Disentangled graph convolutional networks. In ICML, pages 4212-4221.
Jianxin Ma, Chang Zhou, Peng Cui, Hongxia Yang, and Wenwu Zhu. 2019b. Learning disentangled representations for recommendation. In NIPS, pages 5712-5723.
Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017. Embedding-based news recommendation for millions of users. In KDD, pages 1933-1942.
Steffen Rendle. 2012. Factorization machines with libfm. ACM Transactions on Intelligent Systems and Technology (TIST), 3(3):57.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. In ICLR.
Chong Wang and David M Blei. 2011. Collaborative topic modeling for recommending scientific articles. In KDD, pages 448-456.
Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018. Dkn: Deep knowledge-aware network for news recommendation. In WWW, pages 1835-1844.
Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019. Neural graph collaborative filtering. In SIGIR, pages 165-174.
Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019a. Neural news recommendation with attentive multiview learning. In *IJCAI*, pages 3863-3869.
Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019b. Npa: Neural news recommendation with personalized attention. In KDD, pages 2576-2584.
Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019c. Neural news recommendation with multi-head self-attention. In EMNLP-IJCNLP, pages 6390-6395.
Hong-Jian Xue, Xinyu Dai, Jianbing Zhang, Shujian Huang, and Jiajun Chen. 2017. Deep matrix factorization models for recommender systems. In *IJCAI*, pages 3203-3209.
Cheng Yang, Maosong Sun, Xiaoyuan Yi, and Wenhao Li. 2018. Stylistic chinese poetry generation via unsupervised style disentanglement. In EMNLP, pages 3960-3969.
Qiannan Zhu, Xiaofei Zhou, Zeliang Song, Jianlong Tan, and Li Guo. 2019. Dan: Deep attention neural network for news recommendation. In AAAI, volume 33, pages 5973-5980.