TADI: Topic-aware Attention and Powerful Dual-encoder Interaction for Recall in News Recommendation
Authors: Junxiang Jiang
Venue: EMNLP 2023 Findings (2023)
Keywords: news recommendation
recommendation
TL;DR: TADI: Topic-aware Attention and Powerful Dual-encoder Interaction for Recall in News Recommendation

Abstract: News recommendation is one of the widest commercialization in natural language processing research area, which aims to recommend news according to user interests. New recall plays an important role in news recommendation. It is to recall candidates from a very large news database. Recent researches of news recall mostly adopt dual-encoder architecture as it provides a much faster recall scheme, and they encode each word equally. However, these works remain two challenges: irrelevant word distraction and weak dual-encoder interaction. Therefore, we propose a model Topic-aware Attention and powerful Dual-encoder Interaction for Recall in news recommendation (TADI). To avoid irrelevant word distraction, TADI designs a Topic-aware Attention (TA) which weights words according to news topics. To enhance dual-encoder interaction, TADI provides a cheap yet powerful interaction module, namely Dual-encoder Interaction (DI). DI helps dual encoders interact powerfully based on two aux targets. After performance comparisons between TADI and state-of-the-arts in a series of experiments, we verify the effectiveness of TADI.

TADI: Topic-aware Attention and Powerful Dual-encoder Interaction for Recall in News Recommendation
Junxiang Jiang ex.jiang@outtoook Look.
Abstract
Researches of news recall (Wu al., 2022) (or new candidate generation (Covington et al., 2016) or news retrieve (Wang et al., 2023)) can be catc- gorized into feature- and content- baaee mod- els. Feature-based models focus on feature in- teraction modeling such YoutubeNet (Coving- ton et al.. 2016) and Pinnersage (Pal et al., 2020). Feature- Feature-based based models require summarize fea- tures by manual evitably lose useful information. With the devel- opment of content understanding technology, re- searchers incorporate content understanding with feature interaction, that content-based models, such as (Okura et al., 2017). Different from man- ual text mining of feature-based models, content- based models directly learn representations of user and news by modeling news contents. However, content-based models ignore irrelevant word dis- traction problem. Every word encoded equally, that is why irrelevant words would bring side- effect for news recommendation. For example, football fans are more interested in the news "Li- onel Messi comes back Spain for taking a holiday" than tourists, but only two words "Lionel Messi" in the title are related.
News recommendation is one of the widest commercialization in natural language pro- cessing research area, which aims mend news according to user interests. News recall plays an important role news recom- mendation. It is recall candidates from very large news database. Recent researches of news recall mostly adopt dual-encoder ar- chitecture it provides a much faster recall scheme, and they encode each word equally. However, these works remain challenges: irrelevant word distraction and weak dual- encoder interaction. Therefore, we propose model Topic-aware Attention and powerful Dual-encoder Interaction for recall in news recommendation (TADI). avoid irrelevant word distraction, TADI designs Topic-aware Attention (TA) which weights words accord- ing news topics. To enhance dual-encoder interaction, TADI provides cheap yet power- ful interaction module, namely Dual-encoder Interaction (DI). DI helps dual encoders in- teract powerfully based on two auxiliary tar- gets. After performance comparisons between TADI and ttate-of-the-art in series of exper- iments, we verify the effectiveness of TADI.
mining and thus they in-
1 Introduction
News Candidates
News Representations
News recommendation is one of the widest com- mercialization in natural language processing search area, which feeds rich and suitable news users based on their interests. Currently, news recommendation generally used on online news websites, movie review websites and etc (such as MSN News), it thus has become an useful tools to provide masses of custom information in one go. Generally, recall and ranking are two main steps of news recommendation (Wu et al., 2022). The first one is to recall candidates from a very large database, while the second one is rank news candidates for display. News recall deter- mines the room of recommendation performance, and thus this paper discusses about
News Encoder
News Recommendation Model
User Clicked News
User Encoder
Representations
Figure I: The typical dual-encoder model architecture (Mao et al., 2022).
To recall news candidates, researches mostly rely on dual-encoder architecture which is shown in Figure Dual-encoder architecture able serve efficiently in real time for large scale news. Because encodes user and news inde-

---

pendently, and solves the top-k nearest neighbor search problem in sublinear complexity, by con- verting the problem to Bokimum--Inne-PPoddeee Search (MIPS) (Yao et al., 2021). However, there exists a classical challenge of dual-encoder archi- tecture (Khattab and Zaharia, 2020), that is weak dual-encoder interaction. Specifically, the click predictor (e.g., dot product), which unfortunately is the only interation between dual encoders in that architecture. Weak interaction makes one encoder difficultly utilize the information of the other en- coder. Therefore, model underestimates actual correlation between dual encoders, resulting in se- vere performance degradation.
based models and content-based models. Further- more, we introduce rescarches of news ranking be- cause its technologies are available for news recall.
2.1 News Recall.
Feature-bused models focus on feature interaction modeling, and they are usually utilized in prod- uct recommendation and movie recommendation. In our common live, YoutubeNet and Pinnersage are well known featured-based baselines in news recall (Wu et al., 2022). YoutubeNet uses the av- erage of clicked news embeddings for recall. Pin- nersage recall items based on hierarchical cluster- ing. However, difficulty of effective content min- ing lead to information loss, which limits perfor- mances of feature-based models.
To response to the aforementioned challenges, we propose
recall model, namely, Topic- aware Attention and powerful Dual-encoder Interaction for recall in news recommenda- tion (TADI)'. First, we design the Topic-aware Attention (TA) mechanism to avoid irrelevant word distraction. Because news topic is one of the most important interest indicators which di- rectly reflects preference of potential target users. So it is reasonable to weight words by using TA. Secondly, TADI involves the Dual-encoder Interaction (DI) module which helps dual en- coders interact more powerful than typical dual- encoder models. detail, DI provides two aux- iliary targets to enhance dual encoders interaction on training, without changing the mechanism of online news recall in dual-encoder architecture. Therefore, TADI can leverage the efficiency of dual-eneoder architecture news recall while si- multaneously gaining the ability to more powerful interaction. Afterwards, the effectiveness of TADI is verified by conducting a series of experiments on benchmark dataset MIND (Wu et al., 2020).
In contrast feature-based models, content- based models pay attention content understand- ing modeling. Most content-based recommenda- tion models (Wu et al., 2019b) learn user repre- sentation from sequential user clicked news, and learn news representations from news candidates. Besides, regarding to description of diverse and multi-prained user, a few researchers find that se- more suit-
ries of user interrest representation able than single one. Authors of HieRec (Qi et al., 2021) research user interest reporesentations more deeply. They split user interest into cate- gory, sub-category and overall, so that to learn multiple representations for a user. Compared with single representation models, multiple repre- sentation models achieve better performance, but consume times of computing resources click prediction. (Yu et al., 2022) aims to improve both the effectiveness and the efficiency of pre- trained language models for news recommenda- tion, and finally achieves significant performance improvement. But it consumes more computing resources by training M+1 teacher models and dis- tilling twice to get student models.
In summary, our contributions are four-fold: (1)
We propose news recall model TADI. (2) We design the topic-aware attention TA to avoid the distraction of irrelevant words. (3) We propose the dual-eneoder interaction DI to enhance dual- encoder interaction. (4) Extensive experiments are conducted on the benchmark datasets, which demonstrate the effectiveness of TADI.
We review many researches respectively on two main branches of news recall: feature-based and content-based models. However, irrelevant word distraction might impact model with learning con- fusion. TADI avoids the problem by utilizing news topic. Many researches (Qi et al., 2021; Wu et al., 2019c) involves news topic in modeling, but they seldom aware the distraction and take action solve Furthermore, weak interaction makes one encoder difficultly utilize the information of the other encoder. TADI exploit powerful interaction
2 Related Works
Researches of news recall have rather matured works, mainly could be divided into fearture-
'The source code of TADI is available at https:// gitbub. come com/jx-jji aiaggg//////aaa

---

between dual encoders for information utilization.
user and news, in order to predict click probability. The dual-encoder interaction provides capability which helps dual encoders interact more powerful.
2.2 News Ranking.
Researches of news ranking can be also catego- rized into feature-based and content-based mod- cls. FM and DeepFM are well known featured- based baselines in news ranking. FM models second order feature interactions, while DeepI'M models higher order feature interactions. Re- cently, researchers (Kang and McAuley, 2018; Sun et al., 2019) additionally model user sequen- tial behaviors for performance improvement, e.g.,
Predictor
Dual encoder Interaction
f
Sig
Sig
User Encoder
News Encoder
SDM (Lv et al., 2019) respectively models long- term and short-term user sequential behaviors. Turn to content-based models, MINE (Li et al., 2022) makes the number of user insterest repre- sentaions controlable by tunable And the model achieve best performance when hy- perparameter at 32. (Mao et al., 2022) enriches the semantics of users and news by building user and news graphs.
Cates
Sub-cates
Titles
Entbies
Cate
Sub-cate
Title
Entiyy
Figure 2: The overview of TADI architecture.
4.1 Basic Components
Before discussing the main modules of TADI, we introduce basic components at first, they are feature encoder, aggregation attention and topic- aware attention.
3 Problem Formulation
Feature Encoder. The purpose of feature en- coder is to transform feature into dense embed- ding. Each news is representated by title, category, sub-category and title entity. First of all, similar to previous works (Qi et al., 2021), we adopt pre- trained models such as word2vec (Mikolov et al., 2013) and BERT to map word tokens of titles into dense embeddings. In experiment, vantages of word2vec and BERT in detail, for title encoding. For clarity, we name this type of feature encoder as title encoder. Secondly, category and sub-category are embeded by using GloVe (Pen- nington et al., 2014). Thirdly, entity embedding is learnt from knowledge graphs which is provided by datasets. Fourth, a shallow transformer encoder (2-layer and 4-head) are used to learn feature cor- relation of title and title entity. Fifthly, only for user encoding, all type embeddings are processed by shallow transformer encoders so that to learn cross news information.
In this section, we give main notations and define the news recall problem. Features of news and user are the same previous works (Qi et al., 2021). First, for news candidate defination, we make represented by four type features: title, category, sub-category and title entity. The news title 1t word sequence. Denote the category and sub-category of the news are c" and s". De- note the title entity as dm where consists of en- tities. Secondly, we assume a user has N his-
discuss ad-
torical clicked news, and structure of each his- torical news representation is the same as news candidate. Denote titles as T categories as C [e ch ..., c.). sub-categories as S = [s] sh. .... s and title entities as D = ldf. di...., ). The target of news recall is to learn the mapping from users to the most relevant news. Technically, the target is to minimize the gap between the ground truth and the predicted label ý via optimizing model parameters.
(% ..., (%).
gggregation Attention. Aggregation attention is used to integrate embeddings by using a query, set of keys and values. The query and keys are used to calculate attention weights which measure the importance of values, and then the weighted sum of values is output. Suppose input is X
4
The Proposed Model
TADI is divided into four modules, i.e., user en- coder, news encoder, predictor, and dual-encoder interaction, which is shown in Figure 2. The user encoder and the news encoder respectively gen- crate embeddings of user and news. The predic- tor calculates dot product between embeddings of
[x1, X2; ..., IM] where TMs E dd and de em- bedding dimension, and M the number of em- beddings which need to be aggregated. Inspired by the attention used in Poly-encoder (Humeau

---

al., 2020), aggregation attention is designed in the same manner. The query is trainable vector qe E Rd, Keys A = [kg, kf:... AM where kis E IRd,, they are the output of Fully Connected Network (FCN) whose input is X, and values are the input itself X' The attention weight is A [af, 0ς2,... .... [ where a'M E R. In sum- mary, we give the mathematical formula of aggre- gation attention as follows:
each news has several words and entities of the title, the aggregation module uses the aggregation attention to integrate embeddings of words and en- tities into gat and gMtr. By doing so, we obtain title embedding and title entity embedding of each news. And then we use the aggregation attention twice to integrate title embeddings and title entity embeddings of historical clicked news for unified embeddings gr and gd obtaining.
Topic-aware Encoding. Models would dis- tracted by irrelevant words because such words are treated fairly with relevant words in feature encod- ing. The distration would be sevious especially when few related words in long title are avaiable to predict target. Therefore, user interest under- standing by fair title encoding is not enough. As supplement, model could utilize category and sub- category of news to identify relevant words in ti- tle. To do this, we use topic-aware attention to pay more attention to the topic correlated information. User Aggregation. The target of user encoder is to learn unified user embedding from histor- ical clicked news. Therefore, all aggregated em- bedddings g"r, e".e, f e".., fu. and g"ve are concatenated at first, then we map it to the user embedding " via a FCN.
2/
of a" (aftmax(1att (1)
Topic-aware Attention. Topic-aware attention aims intergrate word embeddings by using top- ies, which getting rid of irrelevant word distrac- tion. First, a topic embedding and word embed- ings are used to generate a query and a series of key-value pairs. In detail, we map news topic em- beding to d-dimension query g' Re by FCN. And then we respectively map word embedings to keys AC' [k4. k2 ... Ky where KMM E Re and values Vi [u], u/y ... Me where UM Rd by two FCNs. Secondly, we obtain attention weights a' = [af. as, and where a E R. We scale down the dot product of o' and K by the square root of length de, and then normalize it to the atten- tion weights by using softmax function. Thirdly, we aggregate D by using attention weights. The mathematical formula is below:
Regergation
Agarogation
FCN
Σ alv a' softmax Kt
ATUL
(2)
Agg
Agg
Co
Feature Encoder
Feature Encoder
4.2 User Encoder
User Encoder is used to learn a user embedding from historical clicked news. The architecture of
g
Agg
Agg
Agg
Agg
user encoder is shown in Figure 3, we introduce the main procedure below:
Feature Encoder
Feature Encoder
Feature Encoder
Feature Encoder
Feature Aggregation. Tites, categories, sub- categories and title entities of historical clicked news are transformed into dense embeddings a. gu.e, gu. and Eu by using feature encoder. The above embeddings of historical clicked news need to be aggregated for unified embeddings obtain- ing. The aggregation operation is divided into two types according to feature type. First, category and sub-category aggregation. Categories and sub- categories embeddings of historical clicked news are respectively integrated into two embeddings ghee and g by using the aggregation attention. Secondly, title and title entity aggregation. Since
Cates
Sub-cates
Titles
Entities
Figure 3: The architecture of user encoder.
4.3 News Encoder
The target of news encoder is to represent news by embeding learning. The architecture of news en- coder is shown in Figure 4. In contrast to user encoder, the input of news encoder is only one news, which reduces a few aggregation operations. The procedure is similar to user encoder, so we

---

give brief introduction below. First of all, title, calegory, sub-category and entity sequence of news candidate are transformed into dense embed- dings E ",",",,",, em and E by using feature en- coder. Secondly, because the aforementioned em- beddings of title and title entity are token-witg wise, so they are aggregated and we obtain gr and g" Thirdly, to avoid distraction of irrelevant words, we category and sub-catogory to identify rel- evant information with topic-aware attention. As a result, we get category- and sub-catogory-we embeddings a and am.a. Finally, we integrate all embeddings and obtain news embedding e" by using a FCN.
The second one is earlier interaction (EI) tar- get, which aims to help model interact earlier, which is to predict target by using category- and sulb-caeegorywwiss aggregated embeddings. To utilize hierarchical information between category and sub-category, we design a block. Specifically, the block first uses FCN to process sub-category- wise concatenated embeddings c². Then, the block concatenates the above output with category-wise embeddings (fur and fr."). After processing by FCN, we get c². Finally, ef is used to predict labels g" via a FCN and a sigmoid function. The formula is below:
concat( (mi))
c concat(f"e fmc, FCN((()) (4)
m
Joir Apgropation
1" sigmoid(FCN(CC)
feomogation
FCN
lecddig
4.5 Optimization
g"
g
The loss function L of TADI divided into three parts: L, L' and Ć" É is the loss between the pre- dicted label and the ground truth y. while L and L" are losses of two auxiliary targets. Technically, they measure differences between their predicted labels (j' and f/") and the ground truth. The math- ematical formula of L is below:
FCN
FCN
Att
Att
Agg
Agg
g
Feature Encoder
Feature Encoder
Feature Encoder
-= + bL + (1-a-b)
Encodar
where a 0, b > 0, 0 < <atb<k (5)
Cate
Sub-cate
Title
Entity
where and b are to 0.8 and 0.1 in expriments. Following pre- vious works (Qi et al., 2021), 2 utilizes Noise Contrastive Estimation (NCE) loss. Given the i- th positive sample (a clicked news) in a batch of datasct, we randomly select K negative samples (non-clicked news) for it. The selection is from the same news impression which displayed to the user. The NCE loss requires positve sample which assigning higher score than negative one and it is formulated as:
they are set
Figure 4: The architecture of news encoder.
4.4 Dual-encoder Interaction
In order make dual encoders interact more pow- erfully, dual-encoder interaction module provides two auxiliary targets by utilizing some layer out- puts of dual encoders. These two auxiliary tar- gets are only used on training, so they do not change the mechanism of online news recall in
dual-encoder architecture. The first one is pow- erful interaction (PI) target. It is to get more pow- erful interaction than only using dot product, by making use of top-level embeddings e and e to predict target. The formula below. First of all, concatenating top-level embeddings and get the concatenated embedding cf. Then, a FCN and a sigmoid function are used to predict labels i/.
No. - Σ log
exp( exp(0, Σ exp(
(6)
where No is the batch size. L' and L" utilize Binary Cross Entropy (BCE) loss. Take L' as example, the formula is below:
N,
of concat(c"u. (
C'
=
log 0l. + (1 - log(1= - i
N
(3)
ÿ' sigmoid(FCN(ed))
(7)

---

5 Experiment
the first 10 entities. To explore the influence of pre-trained models title encoder, we adopt the 300-dimensional GloVe and MiniLM (MiniLM- 121-384d, a distilled BERT) (Wang et al., 2021) to initialize title encoder, because MiniLM can save more time consumption than BERT. Embeddings of category and sub-category are initialized by us- ing GloVe, and they are unfixed during training. The K of Eq. 6 is set to 4 during training, which means each positive news is paired with 4 negative news. We employ Adam (Kingma and Ba, 2015) as the optimization algorithm.
We now uurn our attention to empirically testing TADI, and conduct expriment analyses are to ver- ify the effectiveness of our work.
Table 1: Dataset Statistic.
MIND-small
Marg-large
News
65,238
161,013
Categories
20
Sub-categories
270
294
Impressions
230.117
15,777.377
# Clicks
347,727
24.155,470
5.2 Compared Models
5.1 Experiment Setup
Considering characteristic of title encoder, we cat- egorize models into W2V and BERT types. W2V- based and BERT-based models mean that using W2V (such as word2vec, GloVe) or BERT-like (such BERT, MiniLM) model to encode titles.
Experiments are conducted on MIND which a benchmark in real-world news recommenda- tion. The dataset includes two versions: MIND- large and MIND-small. Table illustrates dataset statistics. MIND-large contains more than 15 million impression logs generated by million users, from which MIND-small randomly sam- ples 50,000 users. An impression log includes the clicked news, non-clicked news and histori- cal clicked news of the user before this impres- sion. Besides, each news contains title, cate- gory, sub-category and title entity. Following pre- vious works (Qi et al., 2021), we employ four ranking metrics, i.e., AUC, MRR, nDCG@5 55, and nDCG@ 10, for performance evaluation. The eval- uation metries in our experiments are used on both news ranking models and recall models, such as previous works (Wang et al., 2023; Khattab and Zaharia, 2020; Cen et al., 2020). On the pur- pose of utilizing experimental results from previ- ous works (such as (Qi et al., 2021; Li et al., 2022; Wu et al., 2020)), our experiments apply the same. The test ste of MIND-large does not have labels, the evaluation is an online 2 Our experiments are conducted on 12 vCPU Intel(R) Xeon(R) Platinum 8255C CPU@2.50GH., 43GB memory and GPU RTX 3090. We count the time consumption of model training on MIND-small: running one epoch respectively consumes about 28 minutes and 120 minutes when using GloVe and MiniLM as title encoder.
W2V. (1) DKN (Wang al., 2018): It uses CNN to learn news representation, and target- aware attention network to learn user representa- tion. (2) NPA (Wu et al., 2019b): It learns news and user representations by considering user per- sonality. (3) NAML (Wu et al., 2019a): It learns user and news representations by using multi-view learning, and it is the Kfaee-OfThhe-a of single representation models with GloVe. (4) LSTUR (An et al., 2019): It models both short- term and long-term user interests by using GRU networks and user ID embeddings. (5) NRMS (Wu et al., 2019d): It employs multi-head self- attentions to learn user and news representations: (6) HieRec (Qi et al., 2021): To represent user, it learns overall embedding, embeddings for each category and embeddings for each sub-category. HieRec costs about 300 times of time consumption than single representation model for news recall.
(SOTA)
BERT. (1) BERT: (Bi et al., 2022) only uses BERT for recall in news recommendation. (2) LSTUR+BERT (Wu et al., 2021): It uses BERT as title encoder LSTUR. (3) NRMS+BERT: NRMS uses BERT as title encoder, which the SOTA of single representation models with BERT.
5.3 Experiment Analysis
We utilize users' most recent 40 clicked news learn user representations. From each news, we use NLTK to split title into words, then se- lect the first 30 words. For title entity, we select
In this section, first analyze model perfor- mance. By doing so, we conduct an ablation anal- ysis. Finally, extensive analyses illustrate the ef- feet of embedding dimension and model perfor- mance on different title encoders.

---

Table 2: Performance Analysis. Baseline performances are provided by (Li et al., 2022; Qi et al., 2021; Wu et al., 2021: Zhang et al., 2021). Bold means the best performance, while underline means the best performance of bascline models. We repeated experiment TADI 3 times and reported average result with standard deviation.
MIND-small
MIND-large
Feature
Model
AUC
MRR
nDCG@5
nDCG@10 10
AUC
MRR
nDCG@s
nDCG@ 10
LibFM
59.74
26.33
27.95
34.29
61,85
29.45
31.45
37.13
Manual
DeepFM
59.89
26.21
27.74
34.06
61.87
29.30
31.35
37.05
DKN
62.90
28.37
30,99
37.41
64.07
32.92
38.66
NPA
64.65
30.01
33.14
39.47
65.92
32.07
34.72
40.37
NAML
66.12
31.53
34.88
41.09
66.46
32.75
35.66
41.40
W2V
LSTUR
65.87
33.95
67.08
35.15
NRMS
65.63
30.96
34.13
40.52
67.66
33.25
36.28
HieRec
67.95
32.87
36,36
42.53
69.03
33.89
37.08
68.28
33.05
36.75
42.91
69.53
34.35
37.42
43.12
TADI
0.12
0.15
+ 0.13
0.03
0.03
- 0.03
BERT
68.26
32.52
35.89
42.33
BETU
68.28
32.58
35.99
42.32
69.49
34.72
37.97
43.70
BERT
NRMS+BERT
68.60
32,97
36.55
42.78
69.50
34.75
37.99
43.72
69.39
33.68
37.55
43.65
70.00
34.49
37.69
43.45
TADI
(+ 0.17)
0.24
0.2
+ 0.21
0.18
0.15
0.14
5.3.1 Performance Analysis
dot product of embeddings. However, HicRec trains multiple embeddings for each user and each news. For example, in the MIND small dataset, each user and each news respectively have 289 embeddings (I overall embedding, 18 category- wise embeddings, and 270 sub-category em- beddings) for rating. Therefore, when recalling news, the time consumption of HieRee is 289 times that of TADI.
We compare model performances to demonstrate effectiveness of our work, in the perspective of ti- te encoders. Table illustrates the performance of each model MIND-small and MIND-large, from which we have following observations:
W2V. First of all, TADI the best perfor- mance model, which verifies the effectiveness of our work. Secondly, performance gaps are large comparing TADI with single representation mod- els both MIND-small and MIND-large. From the comparisons, we find that TADI achieves sig- nificant improvement over baseline models, and the comparison results powerfully support the ef- fectiveness of our work. Thirdly, TADI better than multiple representation models no matter on performance online speed. Performance gaps are smaller between TADI and HieRec than pre- vious comparisons, but TADI is much faster than HieRec for news recall. Efficiency the key fea- ture when we considering TADI. The reason why TADI able to achieve good efficiency is because DI only exists on model training, to help the model obtain interactive information and achieve better performance on news recall. Therefore, it does not add additional computing complexity to em- bedding inference and news recall. When mea- suring efficiency, news recall only considers the time consumption of rating because user embed- ding and news embedding can be inferred offline. The way that TADI recalls news similar with basic dual- encoder models, that is calculating the
Table Ablation Analysis MIND-small.
Module
AUC
MRR
nDCG@5
nDCG@ 10
W2V
68.36
33.15
36.81
43.01
w/o Pl
68.33
33.13
36.80
42.98
w/o EI
33.08
36.78
42.96
w/o DI
67.81
32.85
36.49
42.66
w/o all
67.23
32.45
36.11
42.30
w/ BERT 69.28
33.51
37.39
43.48
wo PI 69.31
33.55
37.48
43.52
w/o EI 69.33
33.60
37.49
43.55
w/o DI
33.45
37.17
43.34
w/o all
68.42
32.78
36.38
42.67
BERT. First, similar to the previous analyses on W2V, TADI better than baseline models with large performance gap. This observation demon- strates the effectiveness of TADI with BERT. Sec- ondly, compared with using W2V to encode titles, TADI with BERT performs better. From the com- parison, we find that it is worth to use BERT on title encoding, even thought it brings com- puting complexity.
Summary. The proposed TADI outperforms

---

the SOTA whenever using W2V or BERT type models to encode title. Furthermore, large perfor- mance gaps between TADI and single representa- tion models illustrate that TADI achieves signif- icant improvement. Finally, TADI demonstrates that single representation models are competitive in contrast to multiple representation models.
tail, performance gaps between the optimal perfor- mance and the rest are larger than 0.4%. Secondly, different from using GloVe, TADI with MiniLM achieves the best performance when the dimen- sion is set to 1024. Specifically, the performance is continously rising until the dimension reaches to 1024, and the performance declines when the dimension large than 1024. In summary, it is opti- mal to set the embedding dimension 768 when TADI uses GloVe, and set the embedding di- mension as 1024 when TADI uses MiniLM.
5.3.2 Ablation Analysis
To understand the importance of TA and DI, we conduct an ablation analysis, as shown in Table 3. Observations are that: First of all, we verify the importance of DI. Performance degradates af- ter removing DI, which reveals that DI is neces- sary for TADI. Secondly, we furtherly remove TA to analyze its effect. After removing TA, we ob- serve that the performance of TADI further de- clines. Therefore, the effect of TA is enormous for TADI. Thirdly, we verify the importance of PI and EI targets in DI. After comparing their per- formances, we find that one of them for TADI is enough. Fourth, combine Table 2 and Table 3, TADI without TA and DI is already outperform- ing most of the baselines. The reasons are that: First, title encoder uses strong component (trans- former encoder) to learn cross information among history clicked news, while NAML ignores them and LSTUR adopts weak component (GRU). Secondly, TADI concatenates shallow and deep in- formation when integrating all information after feature encoder, while baselines don't. Therefore, TADI without TA and DI but also achieves better performance.
502
364
502
174
6.09
BB0 BBCT-aM---nooed
Encodar
Figure 6: Title Encoder Analysis on MIND-small.
5.3.4 Title Encoder Analysis
We conduct an experiment to analyze the influence of title encoder. Previously, we have analyzed TADI performance by using GloVe and MiniLM as title encoder, but MiniLM is only one of dis- tilled BERT versions. Besides, title encoders of a few baselines are different from them. There- fore, we compare the performance and inference speed of TADI by using more title encoders. From comparisons, title encoders are respectively set to Glo MiniLM-6L (MiniL 61--ddd,, MiniLM- 12L (MiniL.M- and BERR--bas--uncased. From Figure 6, we observe: First, TADI with MiniL achieves the best performance. Be- cause MiniLM-1221 Icarns more useful informa- tion from pre-training than GloVe and MinILM- 6L. And the data scale of MIND-small might not fine-tune BERT-base-uncased well, which makes it perform worse. Secondly, the inference speed of TADI with GloVe is the fastest. With the increas- ing of model complexity, inference speed becomes slower and slower. In summary, we prefer to use Glo Ve or MiniLM- 12L as title encoder in terms of performance and inference speed.
181
637
Me
14%
055
031
429
256
706
1034
1250
Dimession
Figure 5: Dimension Analysis on MIND-small.
5.3.3 Embedding Dimension Analysis
We explore the optimal embedding dimension of TADI with GloVe or MiniLM on MIND-small, as shown in Figure 5. We observe that: First, TADI with GloVe achieves the optimal perfor- mance when the dimension is set to 768. In de-
5.3.5 Topic-aware Attention Analysis
We conduct case studies to verify the effective- ness of TA. Five news are randomly selected and

---

Table 4: Case Studies of TA.
Top 5 Relevant Words (desc)
Top Relevant Words (dese)
Title
Category
Sub cub-category
Semior Trump official embellished résumé, had face on fake
Senior, had, fake, Trump. embellished
résumé, embellished, fake, on, Trump
politics
Time cover 2020 Ford Explorer launch hardll went according to plan,
Ford, admits, company, launch, Explorer
Explorer, 2020, hardly, company, plan
autos
autos news
company admits
Mother of missing
girl, missing, human, remains, charged
missing, girl, Florida, charged, remains
Florida girl charged: human remains found Wealthy investors are bracing for sharp stock sell-off in 2020 Chrissy Teigen's 's weekend basically double-header, plus
news
news crime
for, sell-off, investors, in, stock
sell-off, investors, bracing, Wealthy, stock
finance
markets
Souble-header, Chrissy, Teigen, weekend, basically
entertainment celebrity
Teigen, Chrissy, double-header, plus, basically
entertainment
the news encoder is used to calculate the attention weights of their words. Words sorted in de- scending order by the attention weight, and list top five words in Table 4. From the table, we ob- serve that: First all, with topic-aware attention, both category and subcategory capture the correl- ative words. Top five words are generally correla- tive to the category or subcategory. Secondly, the information of category and subcategory captur- ing is complementary, which improves the perfor- mance of TADI. The above comparisons verify the effectiveness TA.
the word size of the title. Finally, we select POS with frequency large than 50, and count their av- erage ranks. By doing so, list the top 5 POS and the last POS in Table We observe that TA pays more attention informative words, for example, JJT obviously more informative than BEZ, so TA is effective.
Conclusion
In this paper, we propose the model TADI for news recommendation. To avoid irrelevant word dis- traction, TADI designs the Topic-aware Attention (TA). TA uses news topic to weight words since news topic is one of important insterest indica- tors which directly reflects preference of poten- tial target users. To make dual encoders interac- tion more powerful, TADI provides Dual-encoder Interaction (DI), DI helps dual-encoder interact more powerfully by providing two auxiliary tar- gets. After conducting series of experiments, the effectiveness of the proposed TADI verified. Be- sides, extensive analyses demonstrate the robust- ness the proposed model. In the future, we plan to achieve better performance by TADI optimiza- tion and data augmentation usage.
Table 5: Quantitatively Analysis of Topic-aware Atten- tion. The number within parentheses the normalized rank. JJT means morphologically superlative adjective. VBD means verb, past tense. VBG means verb, present participle/gerund. OD means ordinal numeral. VBN means verb, past participle. DT means singular deter- miner. WDT means wh- determiner. BEZ means the word "is". DO means the word "do".
POS (Dese by the rank)
JJT (0.3512), VBD (0.3883), VBG (0.4351), OD (0.4447), VBZ (0.4478) (0.7167), (0.7177), WDT (0.7298), BEZ (0.7303), DO (0.7417)
Top
Last
Limitations
To quantitatively analyze the effectiveness of TA based on whole dataset, additionally pro- vide Part-Of-Speech (POS) rank. We firstly use NLTK tag POS of each title and rank POS by us- ing topic attention weights (category-wisc). Sec- ondly, to normalize the rank, we divide the rank by
Although TADI achieves good performance, but remains limitations. First, DI is inflexible. DI could not be directly utilized to other models. Secondly, lack of data cannot fully fine-tune pre- trained feature encoder.

---

References
28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing. China, November 3-7, 2019, pages 2635-2643.
Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019. Neural news rec- ommendation with long- and short-term user repre- sentations. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Vol- ume 1: Long Papers, pages 336-345.
Zhiming Mao, Jian Li, Hongru Wang. Xingshan Zeng, and Kam-Fai Wong. 2022. DIGAT: modeling news recommendation with dual-graph interaction. In Findings of the Association for Computational Lin- guistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-1 7-11, 2022, pages 6595-6607,
Qiwei Bi. Jian Li, Lifeng Shang, Xin Jiang. Qun Liu, and Hanfang Yang. 2022. MTRec: Multi-task learn- ing over BERT for news recommendation. In Find- ings of the Association for Compulational Linguis- fics: ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages 2663 2663-2669.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation word represen- tations in vector space.
Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017. Embedding- news recom- mendation for millions of users. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1933- 1942.
Yukuo Cen, Jianwei Zhang, Xu Zou, Chang Zhou, Hongxia Yang, and Jie Tang. 2020. Controllable multi-interest framework for recommendation. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 2942- 2951.
Aditya Pal, Chantat Eksombatchai, Yitong Zhou, Bo Zhao, Charles Rosenberg, and Jure Leskovee. 2020. Pinnersage: Multi-modal user embedding framework for recommendations at pinterest. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event. CA, USA, August 23-27, 2020, pages 2311- 2320.
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommenda- tions. In Proceedings of the 10th ACM Confer- ence on Recommender Systems, Boston, MA, USA, September 15-19, 2016, pages 191-198.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architec- tures and pre-training strategies for fast and accurate multi-sentence scoring. In 8th International Confer- ence on Learning Representations, ICIR 2020, Ad- dis Ababa, Ethiopia, April 26-30, 2020.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation, In Proceedings of the 2014 confer- ence on empirical methods in natural language pro- cessing (EMNLP), pages 1532-1543.
Tao Qi. Fangzhao Wu, Chuhan Wu, Peiru Yang. Yang Yu, Xing Xie, and Yongfeng Huang. 2021. Hieree: Hierarchical user interest modeling for personalized news recommendation. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 1 11th International Joint Conference Natural Language Processing (Vol- ume 1: Long Papers), pages 5446-5456.
Wang--Chng Kang and Julian McAuley. 2018 Self- attentive sequential recommendation. In IEEE In- ternational Conference on Data Mining, ICDM 2018, Singapore, November 17-20. 2018, pages 197-206. IEEE
Omar Khattab and Matei Zaharia. 2020. Colbert: Ef- ficient and effective passage search via contextual- ized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 39-48.
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019. Bert4rec: Se- quential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th A ACM International Conference on Informa- rion and Knowledge Management, CIKM 2019, Bei- jing, China, November 3-7, 2019, pages 1441-1450.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization.
Jian Li. Jieming Zhu, Qiwei Bi, Guohao Cai, Lifeng Shang, Zhenhua Dong, Xin Jiang, and Qun Liu. 2022. MINER: multi-interest matching network for news recommendation. In Findings of the Asso- ciation for Computational Linguistics: ACL 2022. pages 343-352.
Hongwei Wang. Fuzheng Zhang. Xing Xie, and Minyi Guo. 2018. DKN: deep knowledge-aware network for news recommendation. In Proceedings of the 2018 World Wide Web Conference on World Wide Web. WWW 2018, Lyon, France, April 23-27, 2018 pages 1835-1844.
Fuyu Lv, Taiwei Jin, Changlong Yu, Fei Sun, Quan Lin, Keping Yang, and Wilfred Ng. 2019. SDM: sequential deep matching model for online large- scale recommender system. In Proceedings of the
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2021. Minilmv2: Multi-head self- attention relation distillation for compressing pre- trained transformers. In Findings of the Association

---

for Computational Linguistics: ACL/JJCNLP 2021, Online Event, August 1-6, 2021, pages 2140-2151
Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 3597- 3606.
Yuan Wang. Peifeng Yin. Zhiqiang Tao, Hari Venkate- san, Jin Lai, Yi Fang, and PJ Xiao. 2023. An em- pirical study of selection bias in pinterest ads re- trieval. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing. KDD 2023, Long Beach, CA, USA, August 6-10, 2023, pages 5174-5183.
Tiansheng Yao, Xinyang Yi, Derek Zhiyuan Cheng, Felix Yu, Ting Chen, Aditya Menon, Lichan Hong, Ed H Chi, Steve Tjoa, Jieqi Kang, et al. 2021. Self- supervised learning for large-scale item recommen- dations. In CIKM '21: The 30th ACM Interna- tional Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 5, 2021, pages 4321-4330.
Chuban Wu, Fangzhao Wu, Mingxiao An. Jianqiang Huang. Yongfeng Huang. and Xing Xie. 2019a. Neural news recommendation with attentive multi- view learning. In Proceedings of the Twenty-Eigh International Joint Conference on Artificial Intelli- gence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 3863-3869.
Yang Yu, Fangzhao Wu, Chuhan Wu, Jingwei Yi, and Qi Liu. 2022. Tiny-newsree: Efffective and effi- cient plm-based news recommendation. In Proceed- ings of the 2022 Conference on Empirical Meth- ods in Naturul Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 5478 5478-5489.
Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongleng Huang, and Xing Xie. 2019b. Npa: neural news recommendation with personal- ized attention.
Qi Zhang, Jingjie Li, Qinglin Jia, Chuyuan Wang. Jieming Zhu, Zhaowei Wang, and Xiuqiang He. 2021. UNBERT: user-news matching BERT for news recommendation. In Proceedings of the Thir- tieth International Joint Conference on Artificial In- telligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19.27 August 2021, pages 3356-3362.
Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pages 2576-2584.
Chuban Wu, Fangzhao Wu, Mingxiao An, Yongfeng Huang, and Xing Xie. 2019c. Neural news recom- mendation with topic-aware news representation. In Proceedings gs of the 57th Conference of the Associa- tion for Computational Linguistics, ACL 2019, Flo- rence. Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 1154-1159.
Chuban Wu, Fangzhao Wu, Suyu Ge, Tao Qi. Yongfeng Huang, and Xing Xie. 2019d. Neu- ral news recommendation with multi-head self- attention. In Proceedings the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Nat- ural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 6389-6394.
Chuhan Wu, Fangzhao Wu. Tao Qi, and Yongfeng Huang. 2021. Empowering news recommenda- tion with pre-trained language models. In SIGIR '21: The 44th International ACM SIGIR Confer- ence Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 1652- 1652.
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2022. Two birds with one stone: Unified model learning for both recall and ranking in news recommendation. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ire- land, May 22-27 2022, pages 3474-3480.
Fangzhao WU, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wa, al. 2020. MIND: a large-scale dataset for news recommendation. In

---

A Appendix
We response some questions.
Q1: TADI seems to overlook a crucial problem: the training and inference ranking objective is dis- crepant.
First of all, agree with the viewpoint: "the training and inference ranking objective is dis- crepant". But we could not say it's prob- lem. talgi-t losses allow the main target to learn knowledge from other targets. By weight- ing losses of multi-targets through parameters, we can retain beneficial knowledge to the main tar- get, thereby the main target could achieve better performance. Sccondly, agree that multi-task gradient conflict sometimes makes the main tar- get achieve bad performance, but in our experi- ments, we find that the main target performs bet- ter. In experiment, we have considered the con- flicts among losses and have tried some methods to alleviate conflicts, such as GradNorm [1] and PCGrad [2]. These methods consume lot of extra training time but contribute not much performance improvement. Considering the goal of our paper, we believe it worth have more discussion in separate paper. Finally, is correct on the point of view "the interaction performance and efficiency are always in a trade-off relation" when using user and news embeddings to recall news. Nonetheless, researchers think about to do something to help dual-encoder models better such ColBERT [3]. To achieve better performance, TADI enhance in- teraction of dual-encoder models training pro- cedure. Therefore, TADI will not reduce the effi- ciency of the model when recalling news, it still rates news by calculating the dot product of two embeddings.


Reviews:
[{"review_id": "mImbqnWmAa", "reviewer": "Reviewer_o5wg", "summary": "", "questions": "Why do you use the term 'recall' in news recommendation? I'm confused by the term as the task conducted in the experiment is simply the news recommendation (ranking). Is there any reference mentioning the term in news recommendation field?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. We response reasons of rejection point by point.\n\n>The paper does not include an important related work [1] while the performance of [1] on MIND-large dataset is higher than the numbers reported in this paper. Therefore, the paper should compare the proposed approach with the recent related work [1].\n\nThanks for your kind reminding of this newly published work[1]. It is worthy and not late for us to compare 2 papers, after then we will supplement it in the Related Works and the Appendix.\n\n\nWe analyze [1] and our work as below:\n\n1. We compare [1] with our work. The goal of [1] is to enhance news encoder to make performance improvement, by achieving that [1] proposed Tiny-NewsRec. In our paper, we noticed irrelevant word distraction and weak interaction might reduce recommendation performance, by alleviating them we proposed TADI. Honestly Tiny-NewsRec is 1% better than TADI on AUC. We do analysis and the reasons might that: (1) Tiny-NewsRec enhances news encoder with an additional dataset, so theoretically Tiny-NewsRec are supposed to be better. In this case, comparisons between TADI and Tiny-NewsRec seem inappropriate.  (2) In contrast to TADI which trains only one model, Tiny-NewsRec trains M+1 teacher models and distills twice to get 2 student models. Tiny-NewsRec gets more information by using a much more complex but effective manner. Therefore, Tiny-NewsRec and TADI have their own advantages and disadvantages, people could do option based on actual requirements. \n2. From goal achievement perspective, we successfully verify the effectiveness of TADI by comparing with baselines and conducting ablation analyses, no matters if we compare TADI with Tiny-NewsRec or not.\n\n>Analysis on topic-aware attention was missing. The goal of topic-aware attention is to avoid irrelevant word distraction. However, in the experiment, the paper does not show whether the goal is indeed achieved or not.\n\nSorry to confused you on the verification of topic-aware attention. Please allow us to clarify here. We conduct a case study to verify the effectiveness of TA. Five news are randomly selected and the news encoder is used to calculate the attention weights of their words. Words are sorted in descending order by the attention weight, and we list top five words in the below table. From the table, we observe that: First, with topic-aware attention, both category and subcategory capture the correlative words. Top five words are generally correlative to the category or subcategory. Secondly, the information of category and subcategory capturing is complementary, which improves the performance of TADI. The above comparisons verify the effectiveness of TA.\n\n|Title|Category|Top 5 relevant words (desc)|Sub-category|Top 5 relevant words (desc)|\n|----|----|----|----|----|\n|Senior Trump official embellished résumé, had face on fake Time cover|news|Senior, had, fake, Trump, embellished|news politics|résumé, embellished, fake, on, Trump|\n|2020 Ford Explorer launch hardly went according to plan, company admits|autos|Ford, admits, company, launch, Explorer|autos news|Explorer, 2020, hardly, company, plan|\n|Mother of missing Florida girl charged; human remains found|news|girl, missing, human, remains, charged|news crime|missing, girl, Florida, charged, remains|\n|Wealthy investors are bracing for a sharp stock sell-off in 2020|finance|for, sell-off, investors, in, stock|markets|sell-off, investors, bracing, Wealthy, stock|\n|Chrissy Teigen 's weekend was basically a double-header, plus more news|entertainment|double-header, Chrissy, Teigen, weekend, basically|entertainment celebrity|Teigen, Chrissy, double-header, plus, basically|\n\n>Why do you use the term 'recall' in news recommendation? I'm confused by the term as the task conducted in the experiment is simply the news recommendation (ranking). Is there any reference mentioning the term in news recommendation field?\n\nThanks for pointing out that, it is an excellent question related to the crucial point of dual-encoder model, as well as the initial enlightenment of this paper, which is no harm to discuss more.\nUsing the term \"recall\" instead of \"ranking\" because: First, from definition perspective, news recall [2, 3] (or new candidate generation [4] or news retrieve [5]) is mainly to retrieve a small amount news from database.  Secondly, from the usage of dual-encoder models, previous studies [6] and industry practice show that,the models are mostly used to recall news because their high efficiency. On the other hand, Dual-encoder models has its own inherent challenge on weak interaction, which is less competitive comparing with strong interaction models. And almost news ranking models have strong interaction, that is why we use the term \"recall\" instead of \"ranking\". \n \nTo response your question on \"the task conducted in the experiment is simply the news recommendation (ranking)\", we would like to point out that, our experiments are the same as previous works [1,2,3]. The evaluation metrics in our experiments are used on both news ranking models and recall models, such as previous work AUC [5], MRR [6], and NDCG [7]. On the purpose of utilizing experimental results from previous works, our experiments apply the same.\n\n[1] Yang Yu, Fangzhao Wu, Chuhan Wu, Jingwei Yi, and Qi Liu. 2022. Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation. EMNLP'22.\n\n[2] Chuhan Wu et al. Two birds with one stone: Unified model learning for both recall and ranking in news recommendation. Findings of ACL'22\n\n[3] Jian Li et al. MINER: Multi-Interest Matching Network for News Recommendation. Findings of ACL'22\n\n[4] Paul Covington et. al. Deep Neural Networks for YouTube Recommendations. RecSys2016\n\n[5] Yuan Wang et. al. An Empirical Study of Selection Bias in Pinterest Ads Retrieval. KDD'23\n\n[6] Omar Khattab et al. ColBERT: Eficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR '20\n\n[7] Yukuo Cen et al. Controllable Multi-Interest Framework for Recommendation. KDD'20", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed deep learning architecture achieves the highest performance compared to baseline models.\n2. The paper is generally well written, and thus readers will be easy to fully understand the idea in the paper.", "reasons_to_reject": "1. The paper does not include an important related work [1] while the performance of [1] on MIND-large dataset is higher than the numbers reported in this paper. Therefore, the paper should compare the proposed approach with the recent related work [1].\n\n2. ~~Analysis on topic-aware attention was missing. The goal of topic-aware attention is to avoid irrelevant word distraction. However, in the experiment, the paper does not show whether the goal is indeed achieved or not.~~  \n\n[1] Yang Yu, Fangzhao Wu, Chuhan Wu, Jingwei Yi, and Qi Liu. 2022. Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation. EMNLP'22.", "questions_to_authors": "Why do you use the term 'recall' in news recommendation? I'm confused by the term as the task conducted in the experiment is simply the news recommendation (ranking). Is there any reference mentioning the term in news recommendation field?", "paper_topic_and_main_contributions": "This work proposes a news recommender system by designing topic-award attention and dual-encoder interaction modules. The proposed model was evaluated on the MIND datasets, showing the highest performance compared to baseline models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Change 'new recall' to 'news recall'"}, "extra_text": {"Missing_References": "Yang Yu, Fangzhao Wu, Chuhan Wu, Jingwei Yi, and Qi Liu. 2022. Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5478–5489, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics."}}, {"review_id": "6Pfs7uwiBa", "reviewer": "Reviewer_Gomy", "summary": "", "questions": "Could you provide inference time compared to the baselines to justify the efficiency of DI?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We'd like to thank the reviewer for detailed notes and feedback. We response your questions below.\n\n>The authors seem to overlook a crucial problem: the training and inference ranking objective is discrepant...\n\nFirst of all, I agree with your viewpoints: \"the training and inference ranking objective is discrepant\". But we could not say it's a problem. Multi-target losses allow the main target to learn knowledge from other targets. By weighting losses of multi-targets through parameters, we can retain beneficial knowledge to the main target, thereby the main target could achieve better performance. Secondly, we agree that multi-task gradient conflict sometimes makes the main target achieve bad performance, but in our experiments, we find that the main target performs better. In experiment, we have considered the conflicts among losses and have tried some methods to alleviate conflicts, such as GradNorm [1] and PCGrad [2].These methods consume a lot of extra training time but contribute not much performance improvement. Considering the goal of our paper, we believe it is worth to have more discussion in a separate paper. Finally, it is correct on the point of view \"the interaction performance and efficiency are always in a trade-off relation\" when using user and news embeddings to recall news. Nonetheless, researchers think about to do something to help dual-encoder models better such as ColBERT [3]. To achieve better performance, TADI enhance interaction of dual-encoder models on training procedure. Therefore, TADI will not reduce the efficiency of the model when recalling news, as it still rates news by calculating the dot product of two embeddings.\n\n>Because this paper claims an efficient interaction module, efficiency ...\n\nFirst of all, efficiency is the key feature when we considering TADI. The reason why TADI is able to achieve good efficiency is because DI only exists on model training, to help the model obtain interactive information and achieve better performance on news recall. Therefore, it does not add additional computing complexity to embedding inference and news recall. Secondly, we respond to the question \"why TADI is more efficient than HieRec on news recall\". When measuring efficiency, news recall only considers the time consumption of rating because user embedding and news embedding can be inferred offline. The way that TADI recalls news is similar with basic dual-encoder models, that is calculating the dot product of two embeddings (i.e., user and news embeddings). However, HieRec trains multiple embeddings for each user and each news. For example, in the MIND small dataset, each user and each news respectively have 289 embeddings (1 overall embedding, 18 category-wise embeddings, and 270 sub-category-wise embeddings) for rating. Therefore, when recalling news, the time consumption of HieRec is 289 times that of TADI.\n\n>Experiments three times on MIND-small and once on MIND-large is not statistically supportive ...\n\n Once Experiments on MIND-large is because the scale of MIND-large dataset is very large, many methods have given up conducting experiments on this dataset, including the mentioned HieRec [4]. More experiments are for robustness purpose, therefore,  once experiments on MIND-large is acceptable which not impact our conclusion, and it also not impact the comparison between MIND-large and MIND-small. But we repeat experiment two times on MIND-large when using GloVe and BERT as  title encoder, to make our work more convincing. The average results with the standard deviation are reported as below. For hyper-parameter analysis, the mentioned MINER [5] is also performed on MIND-small, so we are reasonable to analyze hyper-parameters on MIND-small.\n|Feature|AUC|MRR|nDCG@5|nDCG@10|\n|----|----|----|----|----|\n|W2V|69.53 $\\pm$ 0.11|34.35 $\\pm$ 0.03|37.42 $\\pm$ 0.03|43.12 $\\pm$ 0.03|\n|BERT|70.00 $\\pm$ 0.18|34.49 $\\pm$ 0.13|37.69 $\\pm$ 0.15|43.45 $\\pm$ 0.14|\n\n\n>Topic-aware attention was widely studied in previous papers, which is somehow incremental.\n\nAdmittedly, there is a lot of works that have involved news topics into modeling, but seldom works have utilized topic-aware attention to alleviate irrelevant word distraction.\n\n[1] GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks\n\n[2] Gradient Surgery for Multi-Task Learning\n\n[3] Omar Khattab et al. ColBERT: Eficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR'20\n\n[4] HieRec: Hierarchical User Interest Modeling for Personalized News Recommendation. ACL 2021. \n\n[5] MINER: Multi-Interest Matching Network for News Recommendation. ACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The writing is clear and easy to understand.\n\n2. The proposed model is modularized making it easily adapted to larger content-based encoders, for example, recent LLM models.\n\n3. The motivation is clear and the model design is rational.\n", "reasons_to_reject": "1. The authors seem to overlook a crucial problem: the training and inference ranking objective is discrepant. Specifically, the training objective of is $a\\mathcal{\\hat{L}}+b\\mathcal{\\hat{L}'}+(1-a-b)\\mathcal{\\hat{L}''}$, whereas the inference target is only related to $\\mathcal{\\hat{L}}$. This training/inference discrepancy is not beneficial to model performance (even harmful from my empirical experience on score-oriented recommendation, but I keep neutral to this point). The major reason could be multi-task gradient conflict [1, 2]. That is, the multi-task objectives tend to be disentangled and conflicted, one objective cannot well represent the other. In this paper, $\\mathcal{\\hat{L}}$, $\\mathcal{\\hat{L}'}$ and $\\mathcal{\\hat{L}''}$ are jointly optimized in training, solely evaluating based on $\\mathcal{\\hat{L}}$ may be inferior. The author may consider gradient surgery [2] to alleviate this problem. Frustratingly, previous studies and industry practice in single/dual recommender encoders show that the interaction performance and efficiency are always in a trade-off relation.\n\n2. Because this paper claims an efficient interaction module, efficiency comparisons with baselines in inference are very important and should be done, but we cannot see it in the experiment tables. The authors claim the overhead of dot product on embeddings is \"1/300 in contrast to HieRec\" in L500. I think this is not rigorous because the authors neither give detailed inference time comparisons nor give more comprehensive explanations, given the fact that dot-product on embeddings is just a very lightweight computation compared to previous FCN and Transformer computation.\n\n3. Experiments three times on MIND-small and once on MIND-large is not statistically supportive. This is because MIND-small is literally small and always leads to high experiment variance. This may be the reason why the curve in Figure 5 fluctuates. I think more experiment trials like previous papers [3, 4] can better support the paper.\n\n4. Topic-aware attention was widely studied in previous papers [3, 4, 5], which is somehow incremental.\n\n\n[1] Gradient Surgery for Multi-Task Learning. Neurips 2020.\n\n[2] Conflict-Averse Gradient Descent for Multi-task Learning. Neurips 2021.\n\n[3] HieRec: Hierarchical User Interest Modeling for Personalized News Recommendation. ACL 2021.\n\n[4] MINER: Multi-Interest Matching Network for News Recommendation. ACL 2022.\n\n[5] Neural news recommendation with topic-aware news representation. ACL 2019.", "questions_to_authors": "Could you provide inference time compared to the baselines to justify the efficiency of DI?", "paper_topic_and_main_contributions": "This paper studies news recommendation. The authors propose a model TADI that can obtain good performance via dual interaction and the interaction efficiency is high. The core contribution is the proposed topic-aware attention and dual-encoder interaction module, which can achieve effective embedding learning and interaction between news and user sides and keep good efficiency in the inference stage.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "L86: \"To response the\" -> \"To response to the\"\n\nL380 Equation (5): \"s.t.\" abbreviates \"subject to\" meaning the constraint term in an optimization problem, here \"where\" is more appropriate.\n\nL506: \"encode title\" -> \"encode titles\"\n\nL527: \"further decline\" -> \"further declines\"\n\nA suggestion: the word \"aux\" should be \"auxiliary\" in formal paper writing."}}, {"review_id": "Lg1obfhveb", "reviewer": "Reviewer_fcXJ", "summary": "", "questions": "1. Can you used more datasets to verify your method?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and suggestions. In the following, we give point-by-point responses to your questions.\n\n>It would be better if more datasets are used for experiments. Currently, only two versions of MIND dataset is used.\n\nExperimentation on more datasets can achieve higher credibility, and we also hope to evaluate the performance of TADI on more datasets. However, there is currently no publicly available dataset like MIND. As with many previous works, we conducted experiments on two versions of MIND datasets and verified the performance of TADI.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed studied in this paper is practical, important and interesting.\n\n2. The proposed method is solid and reasonable in my opinion.\n\n3. The experimental results are promising and convincing.", "reasons_to_reject": "1. It would be better if more datasets are used for experiments. Currently, only two versions of MIND dataset is used.\n\n2. There are a few writing flaws, such as \"Table 2 and 3\" and \"Ablation Analysis on MIND-small\".", "questions_to_authors": "1. Can you used more datasets to verify your method?", "paper_topic_and_main_contributions": "The paper introduces a model called TADI (Topic-aware Attention and powerful Dual-encoder Interaction) for news recommendation to improve recall. TADI addresses irrelevant word distraction by using Topic-aware Attention (TA), which weights words based on news topics, reflecting the preferences of potential target users. Additionally, TADI enhances the interaction between dual encoders with Dual-encoder Interaction (DI), providing two auxiliary targets. Through experiments, the effectiveness and robustness of TADI are verified and analyzed.\n\n\n\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "There are a few writing flaws, such as \"Table 2 and 3\" and \"Ablation Analysis on MIND-small\"."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission studies news recommendation, which is an important application of text mining and text retrieval. This submission focuses on dual-encoder architecture and aims to address its two challenges: irrelevant word distraction and weak dual-encoder interaction. Two components are proposed to address these two challenges. However, this submission lacks discussion and comparison with recent work and needs further experiments on datasets beyond MIND such as Addressa (its textual content can be obtained by request).
Decision: Accept-Findings