SlowGuess/NAACL_2022_OCR huggingface dataset:
Yunfan Hu*
freddy@pku.edu.cn
Zhaopeng Qiu
zhaopengqiu@tencent.com
Xian Wu†
kevinxwu@tencent.com
News recommendation is different from movie or e-commercial recommendation as people usually do not grade the news. Therefore, user feedback for news is always implicit (click behavior, reading time, etc). Inevitably, there are noises in implicit feedback. On one hand, the user may exit immediately after clicking the news as he dislikes the news content, leaving the noise in his positive implicit feedback; on the other hand, the user may be recommended multiple interesting news at the same time and only click one of them, producing the noise in his negative implicit feedback. Opposite implicit feedback could construct more integrated user preferences and help each other to minimize the noise influence. Previous works on news recommendation only used positive implicit feedback and suffered from the noise impact. In this paper, we propose a Denoising neural network for news Recommendation with Positive and Negative implicit feedback, named DRPN. DRPN utilizes both feedback for recommendation with a module to denoise both positive and negative implicit feedback to further enhance the performance. Experiments on the real-world large-scale dataset demonstrate the state-of-the-art performance of DRPN.
Online news platforms, such as Google News and Microsoft News, have attracted a large population of users (Wu et al., 2020b). However, massive news articles emerging every day on these platforms make it difficult for users to find appealing content quickly (Wu et al., 2019b). To alleviate the information overload problem, recommender systems have become integral parts of these platforms.
A core problem in news recommendation is how to learn better representations of users and
news (Hu et al., 2020b). Early works include collaborate filtering (CF) based methods (Das et al., 2007), content-based methods (IJntema et al., 2010) and hybrid methods (De Francisci Morales et al., 2012) that combine the two. These methods usually have the cold start problem when being exposed to the sparsity of user-item interactions (Zhu et al., 2019). Recently, deep learning methods have been proposed to learn better user and news representations. The techniques evolve from using recursive neural network (Okura et al., 2017), attention mechanism (Zhu et al., 2019; Wu et al., 2019c), to graph neural network (Wang et al., 2018a; Hu et al., 2020b,a; Qiu et al., 2022). These methods usually recommend news for users based on their historical feedback.
Implicit feedback is more commonly collected than explicit feedback for news because the users usually do not grade the news. Hence, current news recommendation methods naturally use positive implicit feedback like click behavior as the historical feedback to model user interests. However, there are gaps between positive implicit feedback and user real preferences (Wang et al., 2018b). For example, the click behaviors do not fully reflect the user's preferences. The user may exit the news immediately after clicking, which will involve a noise in the positive feedback. Additionally, some news that users did not click, may also attract them later. Ignoring them also impacts the recommendation performance. Our observation is that using both positive and negative implicit feedback can better model user interests. Besides, positive and negative implicit feedback can help to denoise each other by conducting inter-comparison and intra-comparison. If a news story in one feedback sequence is more similar to the news in the opposite feedback sequence rather than the news in the same sequence, it is very likely that this news story constitutes noise. We can remove this news when building user interests. This idea is shown in Figure 1.
*This work was done when Yunfan Hu was an intern at Tencent.
$\dagger$  Xian Wu is the Corresponding Author.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2320 - 2329
Figure 1: The idea to denoise the implicit feedback. Noises are found by conducting inter-comparison and intra-comparison and then reduced.
In this paper, we propose the Denoising neural network for news Recommendation with Positive and Negative implicit feedback, named DRPN. It first introduces a news encoder to represent the news in two implicit feedback sequences. Then two parallel aggregators are used to extract user representations from both positive and negative historical feedback: (1) content-based aggregator, which selects the informative news in the feedback sequences to represent the user; (2) denoising aggregator, which finds and reduces the noises in the feedback sequences. In addition to the semantic information, we introduce a graph neural network to incorporate the collaborative information to further enrich the user representation. Finally, the user and candidate news representations are used to predict the clicking probability. The contributions of this paper are summarized as follows:
- We propose a novel neural news recommendation approach DRPN which jointly models both positive and negative implicit feedback sequences to represent the user to improve recommendation performance.
- In DRPN, to minimize the impacts of the noises in the implicit feedback, the denoising aggregators are designed to refine the two feedback sequences and can help to further improve the recommendation performance.
- The experiments on the large-scale real-world dataset demonstrate that DRPN achieves state-of-the-art performance.
2.1 Recommendation with Multi-type Feedback
Few works notice the noise problem in the implicit feedback. (Zhao et al., 2018; Liu et al., 2020) use multiple types of feedback to improve recommendation. However, they ignore the noise in the im
plicit feedback. (Wang et al., 2018b) notices the noise problem but it fails to use the meaningful semantic information in the news. (Wu et al., 2020a; Xie et al., 2020; Bian et al., 2021) use the explicit feedback (such as reading time and like/dislike behaviors) to help denoise the implicit feedback. However, the explicit feedback is harder to collect than the implicit feedback. Differently, DRPN only depends on the implicit feedback (click and non-click behaviors) to conduct the denoise to better model the user preferences.
Recently, graph neural networks (GNN) have received wide attention in many fields (Wu et al., 2020c). The convolutional GNN can learn powerful node representations by aggregating the neighbors' features. Recently, some works have attempted to leverage the graph information to enhance the representations learning for news recommendation with GNNs. (Wang et al., 2018a) uses entities in news to build a knowledge graph and use the entity embeddings to improve the model performance. (Ge et al., 2020) combines the one- and two-hop neighbor news and users to enrich the representations of the candidate news and user, respectively. However, these methods also depend on the positive implicit feedback to model user representations and ignore the noise problem.
The news recommendation problem in our paper can be illustrated as follows. Let  $\mathcal{U}$  and  $\mathcal{R}$  denote the entire user set and news set. The feedback matrix for the users over the news is denoted as  $\mathcal{Z} \in \mathbb{R}^{l_u \times l_r}$ , where  $z_{u,r} = 1$  means user  $u$  gives a positive implicit feedback to news  $r$  (e.g.,  $u$  clicks  $r$ ),  $z_{u,r} = -1$  means user  $u$  gives a negative implicit feedback to news  $r$  (e.g.,  $u$  sees  $r$  but ignores it), and  $z_{u,r} = 0$  means no feedback.  $l_u$  and  $l_r$  denote the numbers of the users and news, respectively. For each specific user, his historical positive feedback sequence  $[p_1, \dots, p_{l_p}]$  and negative feedback sequence  $[n_1, \dots, n_{l_n}]$  can be gathered from the feedback matrix  $\mathcal{Z}$ , where  $p_i, n_j \in \mathcal{R}$ .
Given the feedback matrix  $\mathcal{Z}$ , the goal is to train a model  $\mathcal{M}$  (i.e., DRPN). For each new pair of user and candidate news  $(u\in \mathcal{U},r\in \mathcal{R})$ , we can use  $\mathcal{M}$  to estimate the probability that  $u$  would like to click  $r$ .
Figure 2: The DRPN framework.
Figure 2 shows the architecture of DRPN. It first employs the title encoder and id embedding layer to represent all news in two feedback sequences and the candidate news. Then two separate encoders are employed to extract the user semantic interest and collaborative interest information from both positive and negative implicit feedback sequences. Next, two fusion nets combine multiple interest representations to represent the user. Finally, we use the user and candidate news representations to estimate the clicking probability. We will detail each component in the following subsections.
4.1 Input
The inputs of the DRPN contain six parts: the titles of positive feedback sequence  $[p_1^t, \dots, p_{l_p}^t]$ , the titles of negative feedback sequence  $[n_1^t, \dots, n_{l_n}^t]$ , the candidate news title  $r_c^t$ , the IDs of positive feedback sequence  $[p_1^o, \dots, p_{l_p}^o]$ , the IDs of negative feedback sequence  $[n_1^o, \dots, n_{l_n}^o]$ , and candidate news ID  $r_c^o$ .
For each news title  $t$ , we convert its every word  $w$  to a  $d$ -dimensional vector  $\mathbf{w}$  via an embedding matrix  $\mathbf{E}_W \in \mathbb{R}^{l_w \times d}$ , where  $l_w$  is the vocabulary size and  $d$  is the dimension of word embedding. Then, the title  $t$  is transformed into a matrix  $\mathbf{T}$ .
For each news ID  $o$ , we also convert it to a  $d$ -dimensional vector  $\mathbf{o}$  via an embedding matrix  $\mathbf{E}_O \in \mathbb{R}^{l_r \times d}$ . In this manner, we can encode all news to obtain  $\mathbf{P}^o = [\mathbf{p}_1^o, \dots, \mathbf{p}_{l_p}^o]$  ( $\mathbf{p}_i^o \in \mathbb{R}^d$ ),  $\mathbf{N}^o = [\mathbf{n}_1^o, \dots, \mathbf{n}_{l_n}^o]$  ( $\mathbf{n}_i^o \in \mathbb{R}^d$ ), and  $\mathbf{r}_c^o \in \mathbb{R}^d$ .
Figure 3: The content-based aggregator (CA) on the left and the denoising aggregator (DA) on the right in semantic interest encoder. They encode positive preferences for the user. (best viewed in color)
4.2 Title Encoder
The title encoder can extract the sentence-level semantic representation of the news title. It contains two sub-layers. We take the title  $\mathbf{T}$  as an example to detail the encoding process.
The first sub-layer is a multi-head self-attention layer, which can model the contextual representation of each word. Given three input matrices  $\mathbf{Q} \in \mathbb{R}^{l_q \times d}$ ,  $\mathbf{K} \in \mathbb{R}^{l_v \times d}$  and  $\mathbf{V} \in \mathbb{R}^{l_v \times d}$ , the attention function is defined as:
$$
\operatorname {A t t n} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) = \operatorname {s o f t m a x} (\mathbf {Q} \mathbf {K} ^ {\top} / \sqrt {d}) \mathbf {V} \quad (1)
$$
Multi-head self-attention layer  $\mathrm{MH}(\cdot ,\cdot ,\cdot)$  will further project the input to multiple semantic subspaces and capture the interaction information from multiple views:
$$
\begin{array}{l} \mathrm {M H} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) = [ h e a d _ {1}; \dots ; h e a d _ {l _ {h}} ] \mathbf {W} ^ {I} \\ h e a d _ {i} = \mathrm {A t t n} (\mathbf {Q} \mathbf {W} _ {i} ^ {Q}, \mathbf {K} \mathbf {W} _ {i} ^ {K}, \mathbf {V} \mathbf {W} _ {i} ^ {V}) \end{array} \tag {2}
$$
where  $\mathbf{W}_i^Q$ ,  $\mathbf{W}_i^K$ ,  $\mathbf{W}_i^V \in \mathbb{R}^{d \times d / l_h}$  and  $\mathbf{W}^I \in \mathbb{R}^{d \times d}$  are the parameters to learn.  $l_h$  is the number of heads. [;] denotes the column-wise concatenation for matrices.
Moreover, we employ the residual connection and layer normalization function  $\mathsf{LN}$  defined in (Ba et al., 2016) to fuse the original and contextual representations:  $\widetilde{\mathbf{T}} = \mathsf{LN}(\mathbf{T} + \mathsf{MH}(\mathbf{T},\mathbf{T},\mathbf{T}))$
The second sub-layer is a gated aggregation layer (Qiu et al., 2020). It will select the important words to generate an informative title representation. The gated mechanism is employed to
decide the weight of each words. Given the word embedding matrix  $\widetilde{\mathbf{T}}$ , its sentence-level semantic representation  $\mathbf{t}$  is calculated as follows:
$$
\begin{array}{l} \mathbf {t} = \operatorname {A g g r e g a t e} (\widetilde {\mathbf {T}}) \\ = \left(\operatorname {s o f t m a x} \left(\tanh  \left(\widetilde {\mathbf {T}} \mathbf {W} _ {a} + \mathbf {b} _ {a}\right) \mathbf {W} _ {g}\right)\right) ^ {\top} \widetilde {\mathbf {T}} ^ {(3)} \\ \end{array}
$$
where  $\mathbf{W}_a\in \mathbb{R}^{d\times d'}$ $\mathbf{b}_a\in \mathbb{R}^{d'}$  and  $\mathbf{W}_g\in \mathbb{R}^{d'\times 1}$  are trainable parameters.
Finally, we can use the title encoder to model the titles of all news in two user feedback sequences to obtain  $\mathbf{P}^t = [\mathbf{p}_1^t,\dots,\mathbf{p}_{l_p}^t]$  and  $\mathbf{N}^t = [\mathbf{n}_1^t,\dots,\mathbf{n}_{l_n}^t]$ . For the candidate news, we can also obtain its title representation  $\mathbf{r}_c^t$  via the same title encoder.
4.3 Semantic Interest Encoder
The titles of the news which the user interacted usually reflect the user's interests. Hence, we can learn user interest representations by encoding the semantic representations of the news. As is shown in Figure 3, the semantic interest encoder leverages two aggregators, content-based aggregator (CA) and denoising aggregator (DA), to extract user preferences from both positive and negative feedback sequences.
4.3.1 Content-based Aggregator
Different news have different informativeness when representing users. For example, sport news are more informative than weather news in modeling user personality, since the latter are usually browsed by most users. The content-based aggregator (CA) will first evaluate the importance of different news in the feedback sequence from the content view and then aggregate the important news to represent the user. It contains two sub-layers.
The first one is a multi-head self-attention layer, which can enhance the news representations by capturing their interactions. For the positive feedback sequence  $\mathbf{P}^t$ , the multi-head self-attention layer generates  $\widetilde{\mathbf{P}}^t = \mathrm{LN}(\mathbf{P}^t + \mathrm{MH}(\mathbf{P}^t, \mathbf{P}^t, \mathbf{P}^t))$ . The MH is defined in Eq.(2) with independent parameters and the LN is the layer normalization function.
The second sub-layer is a gated aggregation layer that has the same structure as the one defined in Eq.(3). For  $\widetilde{\mathbf{P}}^t$ , it can select the more informative news to generate the user representation:  $\mathbf{p}_s^t = \text{Aggregate}(\widetilde{\mathbf{P}}^t)$ . We also use the content-based aggregator to generate another user representation from the negative feedback sequence  $\mathbf{N}^t$ ,  $\mathbf{n}_s^t$ .
4.3.2 Denoising Aggregator
Denoising aggregator will conduct what we call a refining operation, which aims to mitigate the impacts of the noises in the feedback when modeling the user interests. Intuitively, if news clicked by the user is more semantically relevant with the news in the positive feedback sequence, this news is more likely the user true preference. Otherwise, if it is more semantically relevant with the news in the negative feedback sequence, it is more likely a noise for representing the user interest. As shown in Figure 3, for each news in the positive feedback sequence, we will conduct the intra-comparisons with the news in the positive sequence and inter-comparisons with the news in the negative sequence to decide its weight when representing the user. This module contains three sub-layers.
The first sub-layer is an intra-attention layer. For news  $\mathbf{p}_j^t\in \mathbf{P}^t$  , this layer uses it as the query to aggregate all news in  $\mathbf{P}^t$  except  $\mathbf{p}_j^t$  by the attention mechanism to obtain the sequence-level representation,  $\hat{\mathbf{p}}_j^t = (\sum_{i\neq j}\alpha_{ji}^h\mathbf{p}_i^t)\mathbf{W}_v^h$  , where
$$
\alpha_ {j i} ^ {h} = \frac {\exp \left((\mathbf {p} _ {j} ^ {t} \mathbf {W} _ {q} ^ {h}) (\mathbf {p} _ {i} ^ {t} \mathbf {W} _ {k} ^ {h}) ^ {\top} / \sqrt {d}\right)}{\sum_ {e \neq j} \exp \left((\mathbf {p} _ {j} ^ {t} \mathbf {W} _ {q} ^ {h}) (\mathbf {p} _ {e} ^ {t} \mathbf {W} _ {k} ^ {h}) ^ {\top} / \sqrt {d}\right)} \tag {4}
$$
$\mathbf{W}_q^h, \mathbf{W}_k^h, \mathbf{W}_v^h \in \mathbb{R}^{d \times d}$  are learnable parameters.
The second sub-layer is an inter-attention layer. For  $\mathbf{p}_j^t$ , this layer uses it as the query to aggregate its relevant news in the negative feedback sequence  $\mathbf{N}^t$  by the attention mechanism.
$$
\hat {\mathbf {n}} _ {j} ^ {t} = \operatorname {A t t n} \left(\mathbf {p} _ {j} ^ {t} \mathbf {W} _ {q} ^ {h}, \mathbf {N} ^ {t} \mathbf {W} _ {k} ^ {h ^ {\prime}}, \mathbf {N} ^ {t} \mathbf {W} _ {v} ^ {h ^ {\prime}}\right) \tag {5}
$$
where  $\mathbf{W}_k^h$ ,  $\mathbf{W}_v^h \in \mathbb{R}^{d \times d}$  are learnable parameters.
The third sub-layer is a gated aggregation layer. The weight of the news  $\mathbf{p}_j^t$  are decided by the semantic similarities between  $\mathbf{p}_j^t$  and two sequence-level representations,  $\hat{\mathbf{p}}_j^t$  and  $\hat{\mathbf{n}}_j^t$ .
$$
s _ {j} ^ {p} = \tanh ([ \mathbf {p} _ {j} ^ {t}; \hat {\mathbf {p}} _ {j} ^ {t} ] \mathbf {W} _ {1} ^ {u} + \mathbf {b} _ {1} ^ {u}) \mathbf {W} _ {2} ^ {u} + b _ {2} ^ {u}
$$
$$
s _ {j} ^ {n} = \tanh  \left(\left[ \mathbf {p} _ {j} ^ {t}; \hat {\mathbf {n}} _ {j} ^ {t} \right] \mathbf {W} _ {3} ^ {u} + \mathbf {b} _ {3} ^ {u}\right) \mathbf {W} _ {4} ^ {u} + b _ {4} ^ {u} \tag {6}
$$
$$
\alpha_ {j} = \frac {\exp (s _ {j} ^ {p} - \mathrm {R e L U} (\gamma) * s _ {j} ^ {n})}{\sum_ {i = 0} ^ {i = l _ {p}} \exp (s _ {i} ^ {p} - \mathrm {R e L U} (\gamma) * s _ {i} ^ {n})}
$$
where  $\mathbf{W}_1^u, \mathbf{W}_3^u \in \mathbb{R}^{2d \times d}, \mathbf{W}_2^u, \mathbf{W}_4^u \in \mathbb{R}^{d \times 1},$ $\mathbf{b}_1^u, \mathbf{b}_3^u \in \mathbb{R}^d, b_2^u, b_4^u \in \mathbb{R}$  and  $\gamma$  are learnable parameters. Then, this layer will aggregate all news
according to their weights to obtain the denoised representation,  $\mathbf{p}_h^t = \sum_{j=0}^{j=l_p} \alpha_j \mathbf{p}_j^t$ .
For the negative feedback sequence  $\mathbf{N}^t$ , we take a dual denoising process to obtain its final representation  $\mathbf{n}_h^t$ .
4.4 Graph Neural Network
If two news,  $r_i$  and  $r_j$ , are co-clicked by the user  $u_1$  and  $r_i$  is also clicked by  $u_2$ ,  $u_2$  may also prefers  $r_j$  based on the idea of the collaborative filtering. Hence, we can further enrich the user interest representations by modeling the collaborative information. Like the knowledge graph, we build a collaborative graph  $\mathcal{G} = \{(r_i, r_j) | r_i, r_j \in \mathcal{R}\}$  over the news set  $\mathcal{R}$  based on the co-clicking relationships in the historical feedback matrix  $\mathcal{Z}$ .  $(r_i, r_j)$  indicates they are neighbors in the graph and have been clicked by the same user. To incorporate the collaborative information, we employ the graph transformer neural network (Shi et al., 2021) to model the news in the user feedback sequence.
First, for each news node  $\mathbf{r}^o$  in  $\mathbf{P}^o$  and  $\mathbf{N}^o$ , we compute the attention weights between it and its neighbors  $\mathcal{N}(\mathbf{r}^o)$  in  $\mathcal{G}$ .  $\mathcal{N}(\mathbf{r}^o)$  denotes the neighbor set of node  $\mathbf{r}^o$ . Take its neighbor  $\mathbf{r}_k^o$  ( $k \in \mathcal{N}(\mathbf{r}^o)$ ) as an example, the attention weight between  $\mathbf{r}^o$  and  $\mathbf{r}_k^o$  at the  $m$ -th head is calculated by
$$
\alpha_ {k} ^ {m} = \frac {\exp \left((\mathbf {r} ^ {o} \mathbf {W} _ {m , 1} ^ {g}) (\mathbf {r} _ {k} ^ {o} \mathbf {W} _ {m , 2} ^ {g}) ^ {\top} / \sqrt {\hat {d}}\right)}{\sum_ {q \in \mathcal {N} (\mathbf {r} ^ {o})} \exp \left((\mathbf {r} ^ {o} \mathbf {W} _ {m , 1} ^ {g}) (\mathbf {r} _ {q} ^ {o} \mathbf {W} _ {m , 2} ^ {g}) ^ {\top} / \sqrt {\hat {d}}\right)}
$$
where  $\mathbf{W}_{m,*}^{g}\in \mathbb{R}^{d\times d / l_{h}^{\prime}}$  are learnable parameters.  $l_{h}^{\prime}$  is the number of heads and  $\hat{d}$  is equal to  $d / l_h^\prime$
Next, each news node will aggregate the information of its neighbors from multiple heads according to the attention weights. For the node  $\mathbf{r}^o$ , the representation aggregated from its neighbors is:
$$
\hat {\mathbf {r}} ^ {o} = \left\| _ {m = 1} ^ {m = l _ {h} ^ {\prime}} \left\{\sum_ {k \in \mathcal {N} (\mathbf {r} ^ {o})} \alpha_ {k} ^ {m} \mathbf {r} _ {k} ^ {o} \mathbf {W} _ {m, 3} ^ {g} \right\} \right. \tag {7}
$$
where  $\{\mathbf{W}_{m,3}^{g}\in \mathbb{R}^{d\times d / l_{h}^{\prime}}\}_{m = 1}^{m = l_{h}^{\prime}}$  are trainable parameters.  $\parallel$  denotes the concatenation operation for  $l_{h}^{\prime}$  heads.
Finally, we will update the representation of each node by fusing its aggregated and original representations.
$$
\begin{array}{l} \widetilde {\mathbf {r}} ^ {o} = \operatorname {F u s e} \left(\mathbf {r} ^ {o}, \hat {\mathbf {r}} ^ {o}\right) \tag {8} \\ = \sigma \left(\left[ \mathbf {r} ^ {o}; \hat {\mathbf {r}} ^ {o} \right] \mathbf {W} _ {1} ^ {f}\right) \odot \tanh  \left(\left[ \mathbf {r} ^ {o}; \hat {\mathbf {r}} ^ {o} \right] \mathbf {W} _ {2} ^ {f}\right) \\ \end{array}
$$
where  $\mathbf{W}_1^f, \mathbf{W}_2^f \in \mathbb{R}^{2d \times d}$  are learnable parameters.  $\odot$  denotes the element-wise multiplication operation.  $\sigma$  is the sigmoid function.
We can use this graph neural network to encode all news in the user positive and negative feedback sequences to obtain  $\widetilde{\mathbf{P}}^o = [\widetilde{\mathbf{p}}_1^o,\dots,\widetilde{\mathbf{p}}_{l_p}^o ]$  and  $\widetilde{\mathbf{N}}^{o} = [\widetilde{\mathbf{n}}_{1}^{o},\dots,\widetilde{\mathbf{n}}_{l_{n}}^{o}]$
4.5 Collaborative Interest Encoder
The module aims to model user interests by aggregating the representations of two feedback sequences encoded by the graph neural network layer, which have incorporated the collaborative information. The structure of the collaborative interest encoder is similar to that of the semantic interest encoder and also contains two aggregators, a content-based aggregator and a denoising aggregator. The denoising aggregators have the same structure as the one in the semantic interest encoder. The only structural difference between two content-based aggregators of two encoders is that there is no multi-head self-attention operation in the content-based aggregator of the collaborative interest encoder. This is because the context information is already propagated by the graph neural work, which has a similar effect with the multi-head self-attention.
The inputs of this encoder are the positive sequence representation  $\widetilde{\mathbf{P}}^o$  and the negative sequence representation  $\widetilde{\mathbf{N}}^o$ . The content-based aggregator will generate two user representations,  $\mathbf{p}_s^o$  and  $\mathbf{n}_s^o$ , based on two sequence representations, respectively. Similarly, the denoising aggregator will denoise two sequences and generate two user representations  $\mathbf{p}_h^o$  and  $\mathbf{n}_h^o$ .
4.6 Fusion Net
There are two fusion nets as shown in Figure 2. They are used to fuse multiple user interest representations extracted by two interest encoders to form a comprehensive user representation. For different user-candidate news  $(u,r)$  pairs, the fusion net dynamically allocates different weights for different interest representations. Two fusion nets have similar structures but different parameters. We take the one for the semantic interest encoder as an example to detail the fusion process.
The fusion net first represents the  $(u,r)$  pair. It should mitigate the effect of two interest encoders and independently calculate the weights for the output representations of two encoders. Hence, it uses the outputs of the title encoder