SlowGuess/ACL_2020_OCR huggingface dataset:
Linmei Hu<sup>1</sup>, Siyong Xu<sup>1</sup>, Chen Li<sup>1</sup>, Cheng Yang<sup>1</sup>, Chuan Shi<sup>1*</sup>  
Nan Duan<sup>2</sup>, Xing Xie<sup>2</sup>, Ming Zhou<sup>2</sup>
$^{1}$ School of Computer Science,
Beijing University of Posts and Telecommunications, Beijing, China
{hulinmei, xusiyong, leechen, yangcheng, shichuan} @bupt.edu.cn
{nanduan,xing.xie,mingzhou}@microsoft.com
With the explosion of news information, personalized news recommendation has become very important for users to quickly find their interested contents. Most existing methods usually learn the representations of users and news from news contents for recommendation. However, they seldom consider high-order connectivity underlying the user-news interactions. Moreover, existing methods failed to disentangle a user's latent preference factors which cause her clicks on different news. In this paper, we model the user-news interactions as a bipartite graph and propose a novel Graph Neural News Recommendation model with Unsupervised Preference Disentanglement, named GNUD. Our model can encode high-order relationships into user and news representations by information propagation along the graph. Furthermore, the learned representations are disentangled with latent preference factors by a neighborhood routing algorithm, which can enhance expressiveness and interpretability. A preference regularizer is also designed to force each disentangled subspace to independently reflect an isolated preference, improving the quality of the disentangled representations. Experimental results on real-world news datasets demonstrate that our proposed model can effectively improve the performance of news recommendation and outperform state-of-the-art news recommendation methods.
The amount of news and articles on many news platforms, such as Google News<sup>1</sup>, has been growing
Figure 1: An illustration of user-news interaction graph and high-order connectivity. The representations of user and news are disentangled with latent preference factors.
constantly at an explosive rate, making it difficult for users to seek for news that they are interested in. In order to tackle the problem of information overload and meet the needs of users, news recommendation has been playing an increasingly important role for mining users' reading interest and providing personalized contents (IJntema et al., 2010; Liu et al., 2010).
A core problem in news recommendation is how to learn better representations of users and news. Recently, many deep learning based methods have been proposed to automatically learn informative user and news representations (Okura et al., 2017; Wang et al., 2018). For instance, DKN (Wang et al., 2018) learns knowledge-aware news representation via multi-channel CNN and gets a representation of a user by aggregating her clicked news history with different weights. However, these methods (Wu et al., 2019b; Zhu et al., 2019; An et al., 2019) usually focus on news contents, and seldom consider the collaborative signal in the form of high-order connectivity underlying the user-news interactions. Capturing high-order connectivity among users and news could deeply exploit structure characteristics and alleviate the sparsity, thus improving the rec
Corresponding author:Chuan Shi(shichuan@bupt.edu.cn)
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4255-4264 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ommendation performance (Wang et al., 2019). For example, as shown in Figure 1, the high-order relationship  $u_{1} - d_{1} - u_{2}$  indicates the behavior similarity between  $u_{1}$  and  $u_{2}$  so that we may recommend  $d_{3}$  to  $u_{2}$  since  $u_{1}$  clicked  $d_{3}$ , while  $d_{1} - u_{2} - d_{4}$  implies  $d_{1}$  and  $d_{4}$  may have similar target users.
Moreover, users may click different news due to their great diversity of preferences. The real-world user-news interactions arise from highly complex latent preference factors. For example, as shown in Figure 1,  $u_{2}$  might click  $d_{1}$  under her preference to entertainment news, while chooses  $d_{4}$  due to her interest in politics. When aggregating neighborhood information along the graph, different importance of neighbors under different latent preference factors should be considered. Learning representations that uncover and disentangle these latent preference factors can bring enhanced expressiveness and interpretability, which nevertheless remains largely unexplored by the existing literatures on news recommendation.
In this work, to address the above issues, we model the user-news interactions as a bipartite graph and propose a novel Graph Neural News Recommendation Model with Unsupervised preference Disentanglement (GNUD). Our model is able to capture the high-order connectivities underlying the user-news interactions by propagating the user and news representations along the graph. Furthermore, the learned representations are disentangled by a neighborhood routing mechanism, which dynamically identifies the latent preference factors that may have caused the click between a user and news, and accordingly assigning the news to a subspace that extracts and convolutes features specific to that factor. To force each disentangled subspace to independently reflect an isolated preference, a novel preference regularizer is also designed to maximize the mutual information measuring dependency between two random variables in information theory to strengthen the relationship between the preference factors and the disentangled embeddings. It further improves the disentangled representations of users and news. To summarize, this work makes the following three contributions:
(1) In this work, we model the user-news interactions as a bipartite graph and propose a novel graph neural news recommendation model GNUD with unsupervised preference disentanglement. Our model improves the recommendation performance by fully considering the high-order connectivities
and latent preference factors underlying the usernews interactions.
(2) In our model GNUD, a preference regularizer is designed to enforce each disentangled embedding space to independently reflect an isolated preference, further improving the quality of disentangled representations for users and news.
(3) Experimental results on real-world datasets demonstrate that our proposed model significantly outperforms state-of-the-art news recommendation methods.
In this section, we will review the related studies in three aspects, namely news recommendation, graph neural networks and disentangled representation learning.
News recommendation. Personalized news recommendation is an important task in natural language processing field, which has been widely explored in recent years. Learning better user and news representations is a central task for news recommendation. Traditional collaborative filtering (CF) based methods (Wang and Blei, 2011) often utilize historical interactions between users and news to define the objective function for model training, aiming to predict a personalized ranking over a set of candidates for each user. They usually suffer from cold-start problem since news are often substituted frequently. Many works attempt to take advantage of rich content information, effectively improving the recommendation performance. For example, DSSM (Huang et al., 2013) is a content-based deep neural network to rank a set of documents given a query. Some works (Wang et al., 2018; Zhu et al., 2019) propose to improve news representations via external knowledge, and learn representations of users from their browsed news using an attention module. Wu et al. (2019b) applied attention mechanism at both word- and news-level to model different informativeness on news content for different users. Wu et al. (2019a) exploited different types of news information with an attentive multi-view learning framework. An et al. (2019) considered both titles and topic categories of news, and learned both long- and short-term user representations, while Wu et al. (2019c) represented them by multi-head attention mechanism. However, these works seldom mine high-order structure information.
Graph neural networks. Recently, graph neu
ral networks (GNN) (Kipf and Welling, 2016; Hamilton et al., 2017; Velicković et al., 2017) have received growing attentions in graph embedding because of its powerful representation learning based on node features and graph structure. Wang et al. (2019) explored the GNN to capture high-order connectivity information in user-item graph by propagating embeddings on it, which achieves better performance on recommendation. However, existing news recommendation methods focus on, and rely heavily on news contents. Few news recommendation models consider the user-news interaction graph structure which encodes useful high-order connectivity information. Hu et al. (2020) modeled the user-news interactions as a graph and proposed a graph convolution based model combining long-term and short-term interests, which demonstrates the effectiveness of exploiting the user-news interaction graph structure. Different from all these methods, in this work, we consider both the high-order connectivity information and latent preference factor underlying the user-news interactions. We propose a novel graph neural news recommendation model with unsupervised preference disentanglement.
Disentangled representation learning. Disentangled representation learning aims to identify and disentangle different latent explanatory factors hidden in the observed data (Bengio et al., 2013), which has been successfully applied in the field of computer vision (Kim and Mnih, 2018; Gidaris et al., 2018; Hsieh et al., 2018).  $\beta$ -VAE (Higgins et al., 2017) is a deep unsupervised generative approach that can automatically discover the independent latent factors of variation in unsupervised data, which is based on the VAE framework (Kingma and Welling, 2013). Recently, disentangled representation learning has been investigated on graph-structured data (Ma et al., 2019a,b). To the best of our knowledge, this is the first work to explore disentanglement in news recommendation.
The news recommendation problem can be formalized as follows. Given the user-news historical interactions  $\{(u,d)\}$ , we aim to predict whether a user  $u_{i}$  will click a candidate news  $d_{j}$  that she has not seen before.
In this paper, for a news article  $d$ , we consider the title  $T$  and profile  $P$  (a given set of entities  $E$  and their corresponding entity types  $C$  from the
news content) as features. The entities  $E$  and their corresponding entity types  $C$  are already given in the datasets. Each news title  $T$  consists of a word sequence  $T = \{w_{1}, w_{2}, \dots, w_{m}\}$ . Each profile  $P$  contains a sequence of entities defined as  $E = \{e_{1}, e_{2}, \dots, e_{p}\}$  and corresponding entity types  $C = \{c_{1}, c_{2}, \dots, c_{p}\}$ . We denote the title embedding as  $\mathbf{T} = [\mathbf{w}_{1}, \mathbf{w}_{2}, \dots, \mathbf{w}_{m}]^{T} \in R^{m \times n_{1}}$ , entity set embedding as  $\mathbf{E} = [\mathbf{e}_{1}, \mathbf{e}_{2}, \dots, \mathbf{e}_{p}]^{T} \in R^{p \times n_{1}}$ , and the entity-type set embedding as  $\mathbf{C} = [\mathbf{c}_{1}, \mathbf{c}_{2}, \dots, \mathbf{c}_{p}]^{T} \in R^{p \times n_{2}}$ .  $\mathbf{w}$ ,  $\mathbf{e}$  and  $\mathbf{c}$  are respectively the embedding vectors of word  $w$ , entity  $e$ , and entity type  $c$ .  $n_{1}$  and  $n_{2}$  are the dimension of word (entity) and entity-type embeddings. These embeddings can be pre-trained from a large corpus or randomly initialized. Following (Zhu et al., 2019), we define the profile embedding  $\mathbf{P} = [\mathbf{e}_{1}, g(\mathbf{c}_{1}), \mathbf{e}_{2}, g(\mathbf{c}_{2}), \dots, \mathbf{e}_{p}, g(\mathbf{c}_{p})]^{T}$  where  $\mathbf{P} \in R^{2p \times n_{1}}$ .  $g(\mathbf{c})$  is the transformation function as  $g(\mathbf{c}) = \mathbf{M}_{c}\mathbf{c}$ , where  $\mathbf{M}_{c} \in R^{n_{1} \times n_{2}}$  is a trainable transformation matrix.
4 Our Proposed Method
In this section, we first introduce the news content information extractor which learns a news representation  $\mathbf{h}_d$  from news content. Then we detail our proposed graph neural model GNUD with unsupervised preference disentanglement for news recommendation. Our model not only exploits the high-order structure information underlying the user-news interaction graph but also considers the different latent preference factors causing the clicks between users and news. A novel preference regularizer is also introduced to force each disentangled subspace independently reflect an isolated preference factor.
4.1 News Content Information Extractor
We first describe how to obtain a news representation  $\mathbf{h}_d$  from news content including news title  $T$  and profile  $P$ . The content-based news representations would be taken as initial input embeddings of our model GNUD. Following DAN (Zhu et al., 2019), we use two parallel convolutional neural networks (PCNN) taking the title  $\mathbf{T}$  and profile  $\mathbf{P}$  of news as input to learn the title-level and profile-level representation  $\widehat{\mathbf{T}}$  and  $\widehat{\mathbf{P}}$  for news. Finally we concatenate  $\widehat{\mathbf{T}}$  and  $\widehat{\mathbf{P}}$ , and get the final news representation  $\mathbf{h}_d$  through a fully connected layer  $f$ :
$$
\mathbf {h} _ {d} = f ([ \widehat {\mathbf {T}}; \widehat {\mathbf {P}} ]). \tag {1}
$$
Figure 2: Illustration of our proposed model GNUD.
4.2 GNUD
To capture the high-order connectivity underlying the user-news interactions, we model the user-news interactions as a bipartite graph  $\mathcal{G} = \{\mathcal{U},\mathcal{D},\mathcal{E}\}$ , where  $\mathcal{U}$  and  $\mathcal{D}$  are the sets of users and news,  $\mathcal{E}$  is the set of edges and each edge  $e = (u,d)\in \mathcal{E}$  indicates that user  $u$  explicitly clicks news  $d$ . Our model GNUD enables information propagation among users and news along the graph, thus capturing the high-order relationships among users and news. Additionally, GNUD learns disentangled embeddings that uncover the latent preference factors behind user-news interactions, enhancing expressiveness and interpretability. In the following, we present one single graph covolution layer with preference disentanglement.
4.2.1 Graph Convolution Layer with Preference Disentanglement
Given the user-news bipartite graph  $\mathcal{G}$  where the user embedding  $\mathbf{h}_u$  is randomly initialized and news embedding  $\mathbf{h}_d$  is obtained with the news content information extractor (Section 4.1), a graph convolutional layer aims to learn the representation  $\mathbf{y}_u$  of a node  $u$  by aggregating its neighbors' features:
$$
\mathbf {y} _ {u} = \operatorname {C o n v} \left(\mathbf {h} _ {u}, \left\{\mathbf {h} _ {d}: (u, d) \in \mathcal {E} \right\}\right). \tag {2}
$$
Considering that users' click behaviors could be caused by different latent preference factors, we propose to derive a layer  $\mathrm{Conv}(\cdot)$  such that the output  $\mathbf{y}_u$  and  $\mathbf{y}_d$  are disentangled representations. Each disentangled component reflects one preference factor related to the user or news. The learned disentangled user and news embeddings can bring enhanced expressiveness and interpretability. Assuming that there are  $K$  factors, we would like to let  $\mathbf{y}_u$  and  $\mathbf{y}_d$  be composed of  $K$  independent
components:  $\mathbf{y}_u = [\mathbf{z}_{u,1},\mathbf{z}_{u,2},\dots ,\mathbf{z}_{u,K}]$ ,  $\mathbf{y}_d = [\mathbf{z}_{d,1},\mathbf{z}_{d,2},\dots ,\mathbf{z}_{d,K}]$ , where  $\mathbf{z}_{u,k}$  and  $\mathbf{z}_{d,k}\in R^{\frac{l_{out}}{K}}$  ( $1\leq k\leq K$ ) ( $l_{out}$  is the dimension of  $\mathbf{y}_u$  and  $\mathbf{y}_d$ ), respectively characterizing the  $k$ -th aspect of user  $u$  and news  $d$  related to the  $k$ -th preference factor. Note that in the following of this paper, we focus on user  $u$  and describe the learning process of its representation  $\mathbf{y}_u$ . The news  $d$  can be learned similarly, which is omitted.
Formally, given a  $u$ -related node  $i \in \{u\} \cup \{d : (u,d) \in \mathcal{E}\}$ , we use a subspace-specific projection matrix  $W_{k}$  to map the feature vector  $\mathbf{h}_i \in R^{l_{in}}$  into the  $k$ -th preference related subspace:
$$
\mathbf {s} _ {i, k} = \frac {\operatorname {R e L U} \left(\mathbf {W} _ {k} ^ {\top} \mathbf {h} _ {i} + \mathbf {b} _ {k}\right)}{\| \operatorname {R e L U} \left(\mathbf {W} _ {k} ^ {\top} \mathbf {h} _ {i} + \mathbf {b} _ {k}\right) \| _ {2}}, \tag {3}
$$
where  $\mathbf{W}_k\in R^{l_{in}\times \frac{l_{out}}{K}}$  and  $\mathbf{b}_k\in R^{\frac{l_{out}}{K}}$  . Note that  $\mathbf{s}_{u,k}$  is not equal to the final representation of the  $k$  -th component of  $u$  ..  $\mathbf{z}_{u,k}$  , since it has not mined any information from neighboring news yet. To construct  $\mathbf{z}_{u,k}$  , we need to mine the information from both  $\mathbf{s}_{u,k}$  and the neighborhood features  $\{\mathbf{s}_{d,k}:(u,d)\in \mathcal{E}\}$
The main intuition is that when constructing  $\mathbf{z}_{u,k}$  characterizing the  $k$ -th aspect of  $u$ , we should only use the neighboring news articles  $d$  which connect with user  $u$  due to the preference factor  $k$  instead of all the neighbors. In this work, we apply a neighborhood routing algorithm (Ma et al., 2019a) to identify the subset of neighboring news that actually connect to  $u$  due to the preference factor  $k$ .
Neighborhood routing algorithm. The neighborhood routing algorithm infers the latent preference factors behind user-news interactions by iteratively analyzing the potential subspace formed by a user and her clicked news. The detail is illustrated in Algorithm 1. Formally, let  $r_{d,k}$  be the probability that the user  $u$  clicks the news  $d$  due to the factor  $k$ . Then it's also the probability that we should use the news  $d$  to construct  $\mathbf{z}_{u,k}$ .  $r_{d,k}$  is an unobserved latent variable which can be inferred in an iterative process. The motivation of the iterative process is as follows. Given  $\mathbf{z}_{u,k}$ , the value of the latent variables  $\{r_{d,k}: 1 \leq k \leq K, (u,d) \in \mathcal{E}\}$  can be obtained by measuring the similarity between user  $u$  and her clicked news  $d$  under the  $k$ -th subspace, which is computed as Eq. 4. Initially, we set  $\mathbf{z}_{u,k} = \mathbf{s}_{u,k}$ . On the other hand, after obtaining the latent variables  $\{r_{d,k}\}$ , we can find an estimate of  $\mathbf{z}_{u,k}$  by aggregating information from the clicked news, which is computed as Eq. 5:
Algorithm 1 Neighborhood Routing Algorithm
$$
\mathbf {s} _ {i, k}, i \in \{u \} \bigcup \{d: (u, d) \in \mathcal {E} \}, 1 \leq k \leq K;
$$
Ensure:
$$
\mathbf {z} _ {u, k}, 1 \leq k \leq K;
$$
1:  $\forall k = 1,\dots K,\mathbf{z}_{u,k}\gets \mathbf{s}_{u,k}$
2: for  $T$  iterations do
3: for  $d$  that satisfies  $(u,d)\in \mathcal{E}$  do
4:  $\forall k = 1,\dots ,K:r_{d,k}\gets \mathbf{z}_{u,k}^{\top}\mathbf{s}_{d,k}$
5:  $\forall k = 1,\dots ,K:r_{d,k}\gets \mathrm{softmax}(r_{d,k})$
7: for factor  $k = 1,2,\ldots K$  do
8:  $\mathbf{z}_{u,k} \gets \mathbf{s}_{u,k} + \sum_{d:(u,d) \in \mathcal{E}} r_{d,k} \mathbf{s}_{d,k}$
9:  $\mathbf{z}_{u,k} \gets \mathbf{z}_{u,k} / \|\mathbf{z}_{u,k}\|_2$
11: end for
12: return  $\mathbf{z}_{u,k}$
$$
r _ {d, k} ^ {(t)} = \frac {\exp \left(\mathbf {z} _ {u , k} {} ^ {(t) \top} \mathbf {s} _ {d , k}\right)}{\sum_ {k ^ {\prime} = 1} ^ {K} \exp \left(\mathbf {z} _ {u , k} {} ^ {(t) \top} \mathbf {s} _ {d , k}\right)}, \tag {4}
$$
$$
\mathbf {z} _ {u, k} ^ {(t + 1)} = \frac {\mathbf {s} _ {u , k} + \sum_ {d : (u , d) \in \mathcal {G}} r _ {d , k} ^ {(t)} \mathbf {s} _ {d , k}}{\| \mathbf {s} _ {u , k} + \sum_ {d : (u , d) \in \mathcal {G}} r _ {d , k} ^ {(t)} \mathbf {s} _ {d , k} \| _ {2}}, \tag {5}
$$
where iteration  $t = 0,\dots ,T - 1$  . After  $T$  iterations, the output  $\mathbf{z}_{u,k}^{(T)}$  is the final embedding of user  $u$  in the  $k$  -th latent subspace and we obtain  $\mathbf{y}_u = [\mathbf{z}_{u,1},\mathbf{z}_{u,2},\dots ,\mathbf{z}_{u,K}]$
The above shows a single graph convolutional layer with preference disentanglement, which aggregates information from the first-order neighbors. In order to capture information from high-order neighborhood and learn high-level features, we stack multiple layers. Specially, we use  $L$  layers and get the final disentangled representation  $\mathbf{y}_u^{(L)}\in R^{K\Delta n}$  ( $K\Delta n = l_{out}$ ) for user  $u$  and  $\mathbf{y}_d^{(L)}$  for news  $d$ , where  $\Delta n$  is the dimension of a disentangled subspace.
4.2.2 Preference Regularizer
Naturally, we hope each disentangled subspace can reflect an isolated latent preference factor independently. Since there are no explicit labels indicating the user preferences in the training data, a novel preference regularizer is also designed to maximize the mutual information measuring dependency between two random variables in information theory to strengthen the relationship between the preference factors and the disentangled embeddings.
According to (Yang et al., 2018), the mutual information maximization can be converted to the following form.
Given the representation of a user  $u$  in  $k$ -th  $(1 \leq k \leq K)$  latent subspace, the preference regularizer  $P(k|\mathbf{z}_{u,k})$  estimates the probability of the  $k$ -th subspace (w.r.t. the  $k$ -th preference) that  $\mathbf{z}_{u,k}$  belongs to:
$$
P (k | \mathbf {z} _ {u, k}) = \operatorname {s o f t m a x} \left(\mathbf {W} _ {p} \cdot \mathbf {z} _ {u, k} + \mathbf {b} _ {p}\right), \tag {6}
$$
where  $\mathbf{W}_p\in R^{K\times \Delta n}$ , and parameters in the regularizer  $P(\cdot)$  are shared with all the users and news.
4.3 Model Training
Finally, we add a fully-connected layer, i.e.,  $\mathbf{y}_u' = \mathbf{W}^{(L + 1)^\top}\mathbf{y}_u^{(L)} + \mathbf{b}^{(L + 1)}$ , where  $\mathbf{W}^{(L + 1)} \in R^{K\Delta n \times K\Delta n}$ ,  $\mathbf{b}^{(L + 1)} \in R^{K\Delta n}$ . We use the simple dot product to compute the news click probability score, which is computed as  $\hat{s}\langle u,d\rangle = \mathbf{y}_u'\mathbf{\Sigma}_d'$ .
Once obtaining the click probability scores  $\hat{s}\langle u,d\rangle$ , we define the following base loss function for training sample  $(u,d)$  with the ground truth  $y_{u,d}$ :
$$
\mathcal {L} _ {1} = - \left[ y _ {u, d} \ln \left(\hat {y} _ {u, d}\right) + \left(1 - y _ {u, d}\right) \ln \left(1 - \hat {y} _ {u, d}\right) \right], \tag {7}
$$
where  $\hat{y}_{u,d} = \sigma (\hat{s}\langle u,d\rangle)$
Then we add the preference regularization term of both  $u$  and  $d$ , which can be written as:
$$
\mathcal {L} _ {2} = - \frac {1}{K} \sum_ {k = 1} ^ {K} \sum_ {i \in \{u, d \}} \ln P (k | \mathbf {z} _ {i, k}) [ k ]. \tag {8}
$$
The overall training loss can be rewritten as:
$$
\mathcal {L} = \sum_ {(u, d) \in \mathcal {T} _ {\text {t r a i n}}} ((1 - \lambda) \mathcal {L} _ {1} + \lambda \mathcal {L} _ {2}) + \eta \| \Theta \|, \tag {9}
$$
where  $\mathcal{T}_{\mathrm{train}}$  is training set. For each positive sample  $(u,d)$ , we sample a negative sample from unobserved reading history of  $u$  for training.  $\lambda$  is a balance coefficient.  $\eta$  is the regularization coefficient and  $\| \Theta \|$  denotes all the trainable parameters.
Note that during training and testing, the news that have not been read by any users are taken as isolated nodes in the graph. Their representations are based on only content feature  $h_d$  without neighbor aggregation, and can also be disentangled via Eq. 3.