SlowGuess/ACL_2020_OCR huggingface dataset:
Heyuan Wang†, Fangzhao Wu†, Zheng Liu†, and Xing Xie†
$^{\dagger}$  Microsoft Research Asia, Beijing 100080, China  
heyuanww@163.com, wufangzhao@gmail.com, {zhengliu, xingx}@microsoft.com
Personalized news recommendation is a critical technology to improve users' online news reading experience. The core of news recommendation is accurate matching between user's interests and candidate news. The same user usually has diverse interests that are reflected in different news she has browsed. Meanwhile, important semantic features of news are implied in text segments of different granularities. Existing studies generally represent each user as a single vector and then match the candidate news vector, which may lose fine-grained information for recommendation. In this paper, we propose FIM, a Fine-grained Interest Matching method for neural news recommendation. Instead of aggregating user's all historical browsed news into a unified vector, we hierarchically construct multi-level representations for each news via stacked dilated convolutions. Then we perform fine-grained matching between segment pairs of each browsed news and the candidate news at each semantic level. High-order salient signals are then identified by resembling the hierarchy of image recognition for final click prediction. Extensive experiments on a real-world dataset from MSN news validate the effectiveness of our model on news recommendation.
Recently, people's news reading habits have gradually shifted to digital content services. Many online news websites, such as Google News  $^{1}$  and MSN News  $^{2}$ , aim to collect news from various sources and distribute them for users (Das et al., 2007; Lavie et al., 2010). However, the overwhelming number of newly-sprung news makes it difficult for users to find their interested content (Wu et al., 2019c). Therefore, personalized news recommendation becomes an important technology to
Historical Browsed News
D1	Dog's hilarious reaction to carrot
D2	This woman lost 245 pounds over 5 years. Here's how she did it.
D3	Watch: Philip Rivers hilariously trolls Chiefs fans after win
D4	NFL playoff picture: Saints close to clinching; Patriots fall behind Texans
Candidate News
C1	Ranking the eight starting quarterbacks remaining in the NFL playoffs
C2	Protective golden retriever prevents puppy from being scolded by owner
C3	50 Genius Weight Loss Tricks You Haven't Tried
Figure 1: Example of one user's reading behavior from MSN News. The user has various interests including NFL sports, pets and the issue about weight loss. The highlighted text segments are crucial semantic clues, and the arrows of different colors indicate the relevant matching pairs for candidate news recommendation.
alleviate information overload and improve users' online reading experience (IJntema et al., 2010).
The key to news recommendation lies in the accurate matching of user's interests and candidate news. The same user usually has diverse interests, which are reflected in different news she has browsed. Meanwhile, the important semantic features of news are implied in text segments of different granularities. Figure 1 illustrates the challenges with an example. As demonstrated, different historical browsed news can reveal user's interests about different topics or events. The first and second historical news are about pet dogs and the issue of weight loss respectively. Naturally, they provide critical clues to select the candidate news  $C_2$  and  $C_3$  which reveal relevant information. However, they are less informative to identify the candidate news  $C_1$ , which is about the competition of National Football League (NFL). Besides, the matched segment pairs across browsed news and candidate news lie in different granularities, such as the words "Dog's"-puppy" and phrases "lost 245 pounds"-Weight Loss". Moreover, different segments in news texts have different importance
<sup>1</sup>https://news.google.com/
2https://www.msn.com/news
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 836-845 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
for selecting proper news candidates. For example, in the third historical browsed news  $D_{3}$ , "Philip Rivers" and "Chiefs" are more important than other words like "hilariously" and "after" for inferring that the user is a fan of NFL, since they refer to the famous quarterback and team of this sport.
Existing work, however, usually learns a single representation for each user by integrating all historical news that the user has browsed, then recommendations are performed by matching the final user vector and the candidate news vector (Okura et al., 2017; Wu et al., 2019e,b). For instance, Okura et al. (2017) encode news via denoising autoencoders, and learn representations of users from their browsed news via a GRU network. Wu et al. (2019e) apply multi-head self-attention to learn news representations, then learn user representations by modeling the relatedness between browsed news. Wu et al. (2019b) enhance personalized news and user representations by exploiting the embedding of user's ID to generate a query vector for attending to important words and news. Despite the improvements of these methods in news recommendation performance, they are limited in capturing fine-grained user-news matching signals, since user's various latent interests implied in distinct historical readings cannot match with the candidate news until the final step of click prediction.
In this paper, we propose a Fine-grained Interest Matching network (FIM), which is a new architecture for news recommendation that can tackle the above challenges. The advantages of FIM lie in two cores: the multi-level user/news representation and the fine-grained interest matching. Instead of representing each user as a single abstract vector, we employ hierarchical dilated convolutions in a unified module to construct multi-level representations of each news article based on the title and category annotations. By hierarchically stacking the dilated convolutions, the receptive input width at each layer grows exponentially, while the number of parameters increases only linearly. Meanwhile, the outputs of each layer are preserved as feature maps across different length of text segments, with no loss in coverage since any form of pooling or stride convolution is not applied. In this way, we can gradually obtain the semantic features of news from local correlation and long-term dependency at different granularities, including word, phrase, and sentence levels.
Furthermore, to avoid information loss, FIM
matches the text segments of the candidate news and each historical news browsed by the user at each semantic granularity. In practice, for each pair of news, the model constructs a segment-segment similarity matrix from word-level to sentence-level based on the hierarchical news representations. By this means, user's reading interests implied in the browsing history can be recognized under the supervision of candidate news, and carried into matching with minimal loss, so as to provide sufficient clues about the content relevance for recommending proper news. Afterwards, we merge the multiple matching matrices of each news pair at each granularity into a 3D image, whose channels indicate the relevant degrees of different kinds of user-news matching patterns. By resembling the CNN-based hierarchy of image recognition, higher-order salient signals are identified to predict the probability of the user clicking the candidate news.
We conducted extensive experiments on a real-world dataset collected from MSN news. Experimental results validate that our approach can effectively improve the performance of news recommendation compared with the state-of-the-art methods.
With the explosive growth of digital news, building personalized news recommender systems has drawn more attention in both natural language processing and data mining fields (Phelan et al., 2011; Zheng et al., 2018; Wu et al., 2019a). Conventional news recommendation methods focus on utilizing manual feature engineering to build news and user representations for matching (Phelan et al., 2009; Li et al., 2010; Liu et al., 2010; Son et al., 2013; Li et al., 2014; Bansal et al., 2015). For example, Liu et al. (2010) used topic categories and interest features generated by a Bayesian model to build news and user representations. Son et al. (2013) extracted topic and location features from Wikipedia pages to build news representations for location-based news recommendation.
In recent years, deep learning based models have achieved better performance than traditional methods for news recommendation, due to their capabilities of distilling implicit semantic features in news content (Okura et al., 2017; Wang et al., 2018; An et al., 2019; Wu et al., 2019e,d). For example, Okura et al. (2017) learned news representations via denoising auto-encoders, then used recurrent neural networks to aggregate historical browsed
Figure 2: Architecture of our FIM model. HDC (hierarchical dilated convolution) is the news encoder.
news to learn user representations. Wang et al. (2018) enhanced the representation of news by exploiting the embeddings of extracted entities in a knowledge graph as a separate channel of the CNN input. Wu et al. (2019e) leveraged multi-head self-attention to construct news representations based on the interactions between words, and constructed user representations based on the relatedness between news. An et al. (2019) proposed to learn long-term user preferences from the embeddings of their IDs, and learn short-term user interests from their recently browsed news via GRU network. (Wu et al., 2019a) proposed an attentive multi-view learning model to learn unified news representations from titles, bodies and topic categories by regarding them as different views of news. Different from these existing methods, in FIM, the representations of user's multiple browsed news are not fused into an abstract user vector before matching with the candidate news. Instead, we perform matching between each pair of segments in the news texts from multiple semantic levels. Therefore, more fine-grained information can be distilled for the final recommendation.
The news recommendation problem can be formulated as follows. Given a user  $u$ , the set of historical news she has browsed at the online news platform is formulated as  $s_u = \{d_1, \ldots, d_n\}$ . For a news
candidate  $c_{i}$ , a binary label  $y_{i} \in \{0,1\}$  is adopted to indicate whether  $u$  will click  $c_{i}$  in latter impressions. The aim is to build a prediction model  $g(\cdot ,\cdot)$ . For each pair of user and candidate news  $(u,c)$ , we can predict the probability that  $u$  would like to click  $c$  using the function  $g:s_u,c\to \hat{y}$ . Recommendations are performed based on the ranking of candidate news according to their click scores.
We present a Fine-grained Interest Matching network (FIM) to model  $g(\cdot, \cdot)$ . The architecture of FIM is illustrated in Figure 2, which contains three major components, i.e., a news representation module to construct hierarchical semantic features for news text segments, a cross interaction module to exploit and aggregate matching information from each pair of news at each level of granularity, and a prediction module to calculate the probability that the user will click the candidate news. Next, we introduce each component in detail.
3.2.1 News Representation Module
We design a hierarchical dilated convolution (HDC) encoder to learn representations of news from multiple semantic views. Besides titles that can reflect the central information of news, at many digital platforms such as MSN, news articles are usually labeled with a category annotation (e.g., "sports", "entertainment") and a subcategory annotation (e.g., "football_nba", "movies_celebrity") to help indicate news topics and target users' in
Figure 3: Hierarchical Dilated Convolution (HDC).
terests. HDC encodes each news by connecting its title, category and subcategory annotations into a sequence of words as input. Given the word sequence  $d = [x_{1},\ldots ,x_{N}]$ , where  $N$  is the sequence length, the model first looks up an embedding table to transform  $d$  into a matrix  $\mathbf{d}^0 = [\mathbf{x}_1,\dots ,\mathbf{x}_N]$ , where  $\mathbf{x}_j\in \mathbb{R}^d$  is a  $d$ -dimensional word embedding. Then hierarchical dilated convolution layers are applied to capture multi-grained semantic features in news texts. Different from standard convolution that convolves a contiguous subsequence of the input at each step, dilated convolution (Yu and Koltun, 2016) has a wider receptive field by skipping over  $\delta$  input elements at a time, where  $\delta$  is the dilation rate. For a context of  $\mathbf{x}_j$  and a convolution kernel  $\mathbf{W}$  of size  $2w + 1$ , the dilated convolution operation is:
$$
F \left(x _ {t}\right) = R e L U \left(\mathbf {W} \bigoplus_ {k = 0} ^ {w} \mathbf {x} _ {j \pm k \delta} + \mathbf {b}\right), \tag {1}
$$
where  $\bigoplus$  is the vector concatenation,  $b$  is the bias and ReLU (Nair and Hinton, 2010) is the nonlinear activation function. As shown in Figure 3, the darker output of each convolution layer is a weighted combination of the lighter regular spaced inputs in the previous layer. We start with  $\delta = 1$  (equals to standard convolution) for the first layer to ensure that no element of the input sequence is excluded. Afterwards, by hierarchically stacking the dilated convolutions with wider dilation rates, the length of convolved text segments expands exponentially, and the semantic features of different n-grams can be covered using only a few layers and a modest number of parameters.
Moreover, to prevent vanishing or exploding of gradients, we apply layer normalization (Ba et al., 2016) at the end of each convolution layer. Since there may be irrelevant information introduced to semantic units at a long distance, we practically design the multi-level dilation rates based on the performance in validation. The output of each stacked layer  $l$  is preserved as feature maps of the news
text at a specific level of granularity, formulated as  $\mathbf{d}^l = [\mathbf{x}_j^l]_{j=1}^N \in \mathbb{R}^{N \times f_s}$ , where  $f_s$  is the number of filters for each layer. Suppose there are  $L$  layers stacked, the multi-grained news representations can be defined as  $[\mathbf{d}^0, \mathbf{d}^1, \dots, \mathbf{d}^L]$ . By this means, HDC gradually harvests lexical and semantic features from word and phrase levels with small dilation rates, and captures long dependences from sentence level with larger dilation rates. Meanwhile, the computational path is greatly shortened, and the negative effects of information loss caused by down-sampling methods such as max-pooling can be reduced. Our news encoder is superior to the recurrent units in parallel ability and the entirely attention-based approach in reducing token-pair memory consumptions.
3.2.2 Cross Interaction Module
Given representations of the  $k$ -th browsed news  $[\mathbf{d}_k^l ]_{l = 0}^L$  and the candidate news  $[\mathbf{c}^l ]_{l = 0}^L$ , a segment-segment matching matrix is constructed for each granularity, i.e.,  $\mathbf{M}_{k,c}^{l}\in \mathbb{R}^{N_{d_k}\times N_c}$ , where  $l\in \{0,L\}$  is the semantic level,  $N_{d_k}$  and  $N_{c}$  are the length of the news  $d_{k}$  and  $c$ . The  $(i,j)$ -th element of  $\mathbf{M}_{k,c}^{l}$  is calculated by scaled dot product as:
$$
\mathbf {M} _ {k, c} ^ {l} [ i, j ] = \frac {\mathbf {d} _ {k} ^ {l} [ i ] \cdot \mathbf {c} ^ {l} [ j ] ^ {T}}{\sqrt {f _ {s}}}, \tag {2}
$$
indicating the relevance between the  $i$ -th segment in  $d_{k}$  and the  $j$ -th segment in  $c$  according to the  $l$ -th representation type. The  $L + 1$  matching matrices for the news pair  $\langle d_k, c \rangle$  can be viewed as different feature channels of their matching information.
To summarize the information of user's entire reading sequence, FIM fuses all interaction matrices across each browsed news and the candidate news into a 3D matching image  $\mathbf{Q}$ , formulated as:
$$
\mathbf {Q} = \left\{Q _ {k, i, j} \right\} _ {n \times N _ {d _ {k}} \times N _ {c}}, \tag {3}
$$
where  $n$  denotes the total number of browsed news in user history, and each pixel  $Q_{k,i,j}$  is defined as:
$$
Q _ {k, i, j} = \left[ \mathbf {M} _ {k, c} ^ {l} [ i, j ] \right] _ {l = 0} ^ {L}. \tag {4}
$$
Specifically, each pixel is a concatenated vector with  $L + 1$  channels, indicating the matching degrees between a certain segment pair of the news content at different levels of granularity.
As user's click behaviors may be driven by personalized interests or temporary demands and events, different historical browsed news has different usefulness and representativeness for matching
and recommending the proper candidate news. Inspired by Zhou et al. (2018) in the issue of dialogue system, we resemble the compositional hierarchy of image recognition, and employ a layered 3D convolution & max-pooling neural network to identify the salient matching signals from the whole image. The 3D convolution is the extension of typical 2D convolution, whose filters and strides are 3D cubes. Formally, the higher-order pixel at  $(k,i,j)$  on the  $z$ -th feature map of the  $t$ -th layer is computed as:
$$
\mathbf {Q} _ {k, i, j} ^ {(t, z)} = E L U \left(\sum_ {z ^ {\prime}} \sum_ {w = 0} ^ {W _ {t} - 1 H _ {t} - 1 R _ {t} - 1} \sum_ {h = 0} ^ {R _ {t} - 1} \mathbf {K} _ {w, h, r} ^ {(t, z)} \cdot \mathbf {Q} _ {k + w, i + h, j + r} ^ {(t - 1, z ^ {\prime})} + \mathbf {b} ^ {(t)}\right), \tag {5}
$$
where  $z^{\prime}$  denotes each feature map of the previous layer,  $\mathbf{K}^{(t,z)}\in \mathbb{R}^{W_t\times H_t\times R_t}$  is a 3D convolution kernel with the size of  $W_{t}\times H_{t}\times R_{t}$ , and  $\mathbf{b}^{(t)}$  is the bias for the  $t$ -th layer. A max pooling operation is then adopted to extract salient signals as follows:
$$
\widehat {\mathbf {Q} _ {k , i , j} ^ {(t , z)}} = \max  \left(\mathbf {Q} _ {[ k: k + P _ {w} ^ {(t, z)} - 1 ], [ i: i + P _ {h} ^ {(t, z)} - 1 ], [ j: j + P _ {r} ^ {(t, z)} - 1 ]} ^ {(t, z)}\right), \tag {6}
$$
where  $P_w^{(t,z)}, P_h^{(t,z)}$  and  $P_r^{(t,z)}$  are sizes of 3D max-pooling. Outputs of the final layer are concatenated as the integrated matching vector between the user and the candidate news, denoted as  $\mathbf{s}_{u,c} \in \mathbb{R}^v$ .
3.2.3 Click Prediction Module
In the recommendation scenario studied in this paper, recommendations are made based on ranking the candidate news articles according to their probabilities of being clicked by a user in an impression. Given the integrated matching vector  $\mathbf{s}_{u,c}$  of a user and candidate news pair, the final click probability is calculated as:
$$
\hat {y} _ {u, c} = \mathbf {W} _ {o} ^ {T} \mathbf {s} _ {u, c} + b _ {o}, \tag {7}
$$
where  $\mathbf{W}_o$  and  $b_{o}$  are learned parameters.
Motivated by (Huang et al., 2013b) and (Wu et al., 2019e), we leverage the negative sampling technique for model training. For each news browsed by a user (regarded as a positive sample), we randomly sample  $K$  news which are showcased in the same impression but not clicked by the user as negative samples. Besides, the orders of these news are shuffled to avoid positional biases. FIM jointly predicts the click probability scores of the positive news and the  $K$  negative news during training. By this means, the news click prediction problem is reformulated as a  $(K + 1)$ -way classification task. The loss function is designed to minimize the
summation of negative log-likelihood of all positive samples, which is defined as:
$$
- \sum_ {i = 1} ^ {S} \log \frac {\exp \left(\hat {y} _ {u _ {i} , c _ {i}} ^ {+}\right)}{\exp \left(\hat {y} _ {u _ {i} , c _ {i}} ^ {+}\right) + \sum_ {k = 1} ^ {K} \exp \left(\hat {y} _ {u _ {i} , c _ {i , k}} ^ {-}\right)}, \tag {8}
$$
where  $S$  is the number of positive training samples, and  $c_{i,k}$  is the  $k$ -th negative sample in the same impression with the  $i$ -th positive sample.
4.1 Dataset and Experimental Settings
We conducted experiments on the Microsoft News dataset used in (Wu et al., 2019b) $^3$ , which was built from the user click logs of Microsoft News $^4$ . The detailed statistics are shown in Table 1. Logs in the last week were used for test, and the rest for model training. Besides, we randomly sampled  $10\%$  of logs in the training data for validation.
In our experiments, the word embeddings are 300-dimensional and initialized using pre-trained Glove embedding vectors (Pennington et al., 2014). Due to the limitation of GPU memory, the maximum length of the concatenated word sequence of news title and category is set to 20, and at most 50 browsed news are kept for representing the user's recently reading behaviors. We tested stacking 1-5 HDC layers with different dilation rates. The reported results utilize [1-2-3] hierarchy (dilation rate for each convolution layer) as it gains the best performance on the validation set. The window size and number of convolution filters for news representation are 3 and 150 respectively. For the cross interaction module, we use two-layered composition to distill higher-order salient features of the 3D matching image, and the number and window size of 3D convolution filters are 32-[3,3,3] for the first layer and 16-[3,3,3] for the second layer, with [1,1,1] stride. The followed max-pooling size is [3,3,3] with [3,3,3] stride. Meanwhile, the negative sampling ratio  $K$  is set to 4. Adam (Kingma and Ba, 2014) is used as the optimizer, the mini-batch size is 100, and the initial learning rate is 1e-3.
Following the settings of state-of-the-art methods (Okura et al., 2017; Wu et al., 2019e), we use popular ranking metrics to evaluate the performance of each model, including AUC (Area
<sup>3</sup>A large-scale public version of Microsoft News dataset for news recommendation can be found at https://msnews.github.io
<sup>4</sup>https://microsoftnews.msn.com
# users	10,000	# topic categories	14
# news	42,255	# subtopic categories	284
# impressions	445,230	# positive samples	489,644
avg. # words per title	11.29	# negative samples	6,651,940
Under the ROC Curve (Bradley, 1997), MRR (Mean Reciprocal Rank) (Voorhees et al., 1999), and NDCG (Normalized Discounted Cumulative Gain) (Järvelin and Kekäläinen, 2002). We independently repeated each experiment for 10 times and reported the average performance.
We compare FIM with the following methods:
Manual Feature-based Methods: Traditional recommendation methods which rely on manual feature engineering to build news and user representations, including (1) LibFM (Rendle, 2012), a feature-based matrix factorization model that is widely used in recommendations. We extract TF-IDF features from users' browsed news and candidate news, and concatenate them as the input for LibFM; (2) DSSM (Huang et al., 2013a), a deep structured semantic model with word hashing via character trigram and multiple dense layers. All browsed news are merged into a long document as the query; (3) Wide & Deep (Cheng et al., 2016), a popular recommendation method that combines a wide channel for linear transformations and a deep channel with multiple dense layers. The same features with LibFM are used for both channels; (4) DeepFM (Guo et al., 2017), combining factorization machines and deep neural networks with the same features as LibFM.