PUNR: Pre-training with User Behavior Modeling for News Recommendation
Authors: Guangyuan Ma
Hongtao Liu
Xing W
Wanhui Qian
Zhepeng Lv
Qing Yang
Songlin Hu
Venue: EMNLP 2023 Findings (2023)
Keywords: News Recommendation
Pre-training
User Behavior Modeling

Abstract: News recommendation aims to predict click behaviors based on user behaviors. 
How to effectively model the user representations is the key to recommending preferred news. 
Existing works are mostly focused on improvements in the supervised fine-tuning stage. 
However, there is still a lack of PLM-based unsupervised pre-training methods optimized for user representations.
In this work, we propose an unsupervised pre-training paradigm with two tasks, i.e. user behavior masking and user behavior generation, both towards effective user behavior modeling. Firstly, we introduce the user behavior masking pre-training task to recover the masked user behaviors based on their contextual behaviors. In this way, the model could capture a much stronger and more comprehensive user news reading pattern. Besides, we incorporate a novel auxiliary user behavior generation pre-training task to enhance the user representation vector derived from the user encoder.
We use the above pre-trained user modeling encoder to obtain news and user representations in downstream fine-tuning.
Evaluations on the real-world news benchmark show significant performance improvements over existing baselines.

PUNR: Pre-training with User Behavior Modeling for News Recommendation
Guangyuan Ma Hongtao Liu", Xing Wu!
Wanhui Qian', Zhepeng Lv², Qing Yang'. Songlin Hu
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
Duu Xiaoman Finance, Beijing, China (maguangyuan, wuxing, husonglin ie an.
(liuhongtao01, qianwanhui, 1yzhepeng, ygngqing)geeee eduxiammann aoman. com
Abstract
Det Product
News recommendation aims predict click behaviors based on user behaviors. How to ef- fectively model the user representations the key to recommending preferred news. Exist- ing works are mostly focused on improvements in the supervised fine-tuning stage. However, there is still lack of PI.M PL.--ased unsupervised pre-training methods optimized for user repre- sentations. In this work, we propose unsu- pervised pre-training paradigm with two tasks, iee. user behavior masking and user behavior generation, both towards effective user behay- ior modeling. Firstly, we introduce the user behavior masking pre- pre-training task to recover the masked user behaviors based on their con- textual behaviors. In this way, the model could capture much stronger and more comprehen- sive user news reading pattern. Besides, we incorporate a novel auxiliary user behavior gen- eration pre-training task to enhance the user representation vector derived from the user en- coder. use the above pre-trained user mod- cling encoder to obtain news and user represen- tations in downstream fine-tuning. Evaluations on the real-world news benchmark show signif- icant performance improvements over existing baselines.
CLS
Vector
News
User Vactor
CLS
Share
News Encoder
User Encoder
Parameter
News Embeddings
Embeddings
Concat
N=
Cancidate News
User History News
Figure 1: The architecture of user modeling method for new recommendations in downstream fine-tuning. Siamese encoder used both modeling the news and user embeddings. Note that we use the textual titles of user-clicked news as inputs in pre-training, ruther than pure news id.
get news embeddings of user behaviors. Then a hierarchical shallow user encoder is used on top of the news encoder, in order to aggregate the user behaviors to get user representation. Such dual- tower PLM-based user modeling architecture is widely adopted in current works of news recom- mendations. However, the separated forward of the individual user behaviors (i.e. users' clicked news titles) and hierarchical aggregation result in poten- tially weak user modeling ability and complicated model architectures.
1 Introduction
News recommendation is essential technology for predicting candidate news via the history of user behaviors. widely used in online news feeds and platforms to model user behaviors and push personalized news to their users.
On the contrary, UNBERT (Zhang et al., 2021b) proposes to concatenate the candidate news and user behavior news together, and predict the match- ing score via a classification head with the word- level vector and the news-level vector. AMM (Zhang et al., 2021 concatenates the cross pairs of title, abstract, and body together between the users' history news and candidate news, extract multi- field matching representation in both within-field and cross-field. Such single-tower classification techniques result in good user-candidate behavior modeling but are relatively computationally expen-
Existing news recommendation methods often employ Pre-trained Language Models (PLMs) to produce news vectors and user vectors, and match personalized candidate news based on the dot prod- uet distances of these two vectors. For example, NRMS-BERT (Wu al., 2019c, 2021) replaces multi-head self-attention with pre-trained PLM, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and UniLM (Dong et al., 2019), to

---

User Behavior Masking (MLM)
User Behavior Generation
mask the whole behavior spans in the inputs of the user encoder. The task for the user encoder is to recover the masked behavior spans with its contextual user history news texts. The modeling ability of the user encoder is improved and makes stronger recommending performance.
(CLM)
CLS
User Encoder
Decoder
Inspired by recent representation generative pre- training technology in IR community (Lu et al., 2021; Wu et al., 2022), the information can be com- pressed into the representation vector through a generative pre- tre-training task with a shallow Trans- former decoder. This auto-regression pre-training schema enlights us to explore new pre-training paradigms for better modeling user vector in news recommendations.
CLS
Masked Embeddings
CLS
User Embeddings
Behavior Masking
Behavior News
Figure 2: Pre-training design of our proposed method, PUNR. We incorporate a special user behavior masking strategy and user behavior generation task the pre- training stage of the user encoder.
User behavior generation task. A single- layer auto-regression Transformers decoder is added on top of the user encoder. The decoder takes the user vector and user history news as in- put. The task for the decoder is to generate the whole user history news in an auto-regression way assisted by the user vector. This auto-regression pretask enables a more powerful user representa- tion learning.
sive. In order to achieve joint modeling of user behaviors while maintaining a balanced computa- tional offload, as is shown in Figure 1, we simplify the architecture of the user encoder and only use a Siamese encoder for modeling both news vec- tors and user vectors. The concatenation of user history news brings better alignment of user behav- iors. And we refer to this joint modeling of users' clicked histories as User Modeling.
To the best of our knowledge, we make the first effort to bring unsupervised pre-training tasks to the PLM-based news recommender system. The joint use of user behavior generation and user be- havior masking design for user vector optimization makes steady recommendation improvement. We also propose a simple yet effective Siamese archi- tecture for encoding both the news vectors and user vectors. Experiment results on the real-world news recommendation task show considerable perfor- mance gains over multiple baselines. In addition, conduct several ablation studies to verify the robustness of our method.
Effectively recommending content to users relies heavily on the ability to accurately model their be- haviors (Sun et al., 2019). PTUM (Wu et al., 2020a) proposes pre-training multi-head self-attention lay- ers from scratch to predict masked behaviors and the next K behaviors. However, PTUM only uti- lizes a small amount of in-domain data and does not leverage the power of PLMs. Unfortunately, pre- training for news recommendation has not yet been extensively explored, as most existing methods fo- cus on improving the fine-tuning stage by adapting an off-the-shelf PLM as the news encoder. We ar- gue that pre-training a PLM specifically tailored for news recommendation essential for achieving optimal performance. Moreover, research on mod- eling user behaviors for news recommendations is still in its early stages.
Our contribution can be summarized as follows:
We propose an effective user modeling Siamese encoder architecture for both encoding the news and user representations.
We incorporate user behavior generation and user behavior masking tasks into the unsupervised pre- training of the PLM-based news recommendation system.
In our work, we propose PUNR, Pre-training with User behavior modeling for News Recom- mendation, seeking to incorporate unsupervised pre-training signals for better user modeling ability. On top of the Siamese architecture, we incorporate two pre-training tasks tailored for user modeling, Detailed designs are shown in Figure 2.
Experiments show that our work achieves consid- erable performance gains over multiple baselines.
Approach
In this section, we describe the problem statement of news recommendation and the detailed imple-
User behavior masking task. We randomly

---

User
CLS
Masked User Embeddings
Embeddings
Token
CLS
X1
SEP
[M]
[M]
[M]
SEP
Y1
SEP
Embeddings
+
+
+
+
Positional
PosO
Pos1
Pos2
Pos3
Pos4
Pos5
Pos6
Pos7
Pos8
Pos9
Embeddings
+
+
News Segment Embeddings
Seg0
SegO
SegO
Seg1
Seg1
Seg1
Seg1
Seg2
Seg2
Seg2
Whole History Masking
User History News
N=
Figure The implementation of user model embeddings for PUNR. The title fields of the clicked news are concatenated together for joint modeling of each user's behavior. The joint user modeling not only simplifies the architecture the user encoder to a single BERT model but also enables stronger recommendation effects.
mentations of the pre-training and fine-tuning.
The percentage of mask tokens used by user behav- ior masking is B. And the rest of the tokens randomly masked. We denote the set of masked tokens ML.
2.1 Problem Statement
Given user with set of clicked news (D, Di.... D u (denoted as bchaviors), the task for news recommendation to distin- guish preferable (positive) news from candidate sets [DY, ..,..., Divi based on user's be- haviors.
The masked text m(T) is firstly converted to user embeddings (mm(T)), and then forwarded through the L-layer Transformers -based user encoder (Enc) to recover all masked positions with the MLM loss. The loss is formulated as:
2.2 Pre-training with User Behavior Modeling
Lmmm = Σ log Jogp(t)(En((((((T)) (2)
We employ a Transformers-based Pretrained Lan- guage Model (PLM), e.g. BERT, to serve as a user encoder to generate the representation of the user's behaviors. Specifically, given a clicked news set U, we tokenize each news input! and con- catenate them together line of input tokens T (to, t.) to produce token embeddings Etoken Following (Zhang et al., 2021b), we em- ploy positional embeddings Epos and news seg- ment embeddings Eseg to represent positional and segmenting information. The input embeddings E is the sum of the above three embeddings, as is shown in Figure 3.
SEM
During the recovery of user behavior masking, the user model predicts the entirely masked behav- iors with unmasked partially masked contexts. This masking strategy will bring consistent user behavior alignment in user modeling.
User Behavior Generation For each layer / E [1, L} of the user encoder, the output hidden states are denoted follows:
H' (h/ h/ bly
(3)
Here we simply use the last hidden states the [CLS] position, i.e. h/aat as the user vector. Based on the concatenated input of user vector CLS and user embeddings, single-layer Transformers- based auto-regression decoder (Dec) is introduced to perform the generation task of the entire user behavior texts.
(TT)= Ftokon
Eepos +
Fese
(1)
User Behavior Masking During the pre-training stage, employ user span masking strategy for modeling the user behaviors with token-level Mask Language Modeling (MLM) task. Given total mask ratio a, we especially use [MASK] tokens to replace the whole piece of the news segment.
FFollowigg previous works, we use the title of news as inputs.
Lase =
Σ log pogp(t]] E(T))) (4)
1E

---

Similar to the discussion in (Lu al., 2021; Wu al., 2022), the single-layer decoder has limited modeling capacity. So it needs to rely on the infor- mation from the user vector to finish the generation of entire textual user behaviors. Thus the token- level behavior information is compressed into the user vector, resulting in stronger user behavior mod- cling.
pre-training corpus Wiki-BookCorpus for the ini- tialization of the decoder. It contains about 5.6 million documents. NLTK sentence tokenizer is used to split the corpus and dynamically padded to the target max length.
The user encoder in the pre- training stage is initialized from the pre-trained BERT-base checkpoint. We first freeze the BERT encoder and only pre-train the single-layer auto- regression with general Wiki+BookCorpus. The AdamW optimizer is used with a batch size of 512, a learning rate of 3e-4, and pre-training steps 1.0k for the full initialization of the decoder on the general corpus. Then we pre-train the full model on MIND corpus using Adam optimizer with batch size of 256, a learning rate of le-5, and pre-training steps of 10k. A linear scheduler with warmup ratio of 0. used for the learning rate schedule. We set the total masking ratio a 0.3 and the user behavior masking ratio in the total masked token sets 3 to 0.3 because this combina- tion brings the best results. We use 8 Tesla V100 GPUs and takes about I day to finish the whole pre-training. We vary the pre-training seeds and take average of 3 scores our main results.
Implementation
The pre-training loss is the sum of the encoder
MLM loss and the decoder CLM loss.
C Lmim + Ldec
(5)
2.3 Fine-tuning
Fine-tuning is conducted on recommendation benchmarks for verifying the effectiveness of our work. For simplicity of downstream design, we share the parameters between the news encoder and the user encoder, for both modeling the candi- date news and the user's clicked news. The encoder is initialized from the above pre-trained PLM. The clicked news is jointly forwarded through the user encoder to obtain the user vector the CLS position. And the candidate news is separately for- warded through the news encoder to obtain the in- dividual news vectors v. Following TRMS-BERY (Wu et al., 2021), we use the negative sampling strategy, utilizing a cross-entropy loss for minimiz- ing the distance between the user vector u and the positive news vector v² while pushing away the negative news vectors e
3.2 Fine-tuning
Datasets The MIND benchmark (Wu et al., 2020b) is used to evaluate the effectiveness of our method. It is read-world English news recom- mendation dataset, which contains 160k news texts, million users, 15.8 million impressions, and 24 million click behaviors collected from Microsoft News in six weeks. MIND has two versions of datasets with different sizes. MIND-large includes 2.2 million samples in the training set and 365K samples in the development set. MIND-small is randomly sampled version of the whole dataset, which contains 50k users and their behavior logs from the MIND dataset.
exp(d(u, v+ Ca - -log Σ exp((d(u, ut) + ((u,)))) (6)
where d means the dot product operation.
3 Experiments
In this section, introduce the experimental set- tings of work, including the pre-training, fine- tuning, and baseline methods. Then we analyze the main experimental results.
The pre-trained encoder is used to initialize the downstream news and user en- coders. The parameters are shared across these two encoders because we found that using a Siamese encoder improves performance. We fine-tume the encoder using AdamW optimizer with batch size of 128, and learning rate of 3e-5 for 3 epochs. The linear scheduler with a warmup ratio of 0.1 is also used in the fine-tuning stage. Following (Wu et al., 2021), I positive and negative news are randomly sampled from the candidate sets. The
Implementation
3.1 Pre-training
Datasets The MIND dataset (Wu et al., 2020b) is used the pre-training corpus. We use 2.2 mil- lion click behaviors from the training set of MIND for the pre-training of our model. The target max length is set to 512.
Since the auto regressinns decoder is not initial- ized, we use the original BERT (Devlin et al., 2019)

---

MIND-Small
MIND-Large
Methods
AUC
MRR
nDCG@5
nDCG@ 10
AUC
MRR
nDCG@5
nDCG@
LibFM
59.74
26.33
27.95
34.29
61.85
29.45
31.45
37.13
DeepFM
59.89
26.21
27.74
34.06
61.87
29.3
31.35
37.05
DKN
61.75
27.05
28.9
35.38
64.07
30.42
32.92
38.66
NPA
63.21
29.11
31.7
37.81
65.92
32.07
34.72
40.37
NAMI
65.5
30.39
33.08
39.31
66.46
32.75
35.66
41.4
LSTUR
64.38
29.46
31.89
38.17
67.08
32.36
35.15
40.93
NRMS
64.83
30.01
32.52
38,92
67.66
33.25
36,28
41.98
FIM
65.02
30.26
32.91
39.1
67.87
33.46
36.53
42.21
NRMS BERT
65.524
31,003
33.87f
40.381
69.5
34,75
37.99
43.72
UNBERT 67.62
31.72
34.75
41.02
70.68
35.68
39.13
44.78
67.96
32.98
36.64
42.77
PUNR
68.89
33.33
36.94_a.11
43.10
71.03=0.04
35.17
39.04.0.04
45.41-0.05
Table 1: Main results MIND datasets. The best scores marked in bold. 'Two-tailed t-tests demonstrate statistically significant improvements in our method over baselines ( p--value S 0.05 ). tThe scores are based on our reproduction.
max number of user behaviors for the user encoder is set to 50, and the max length for each news title is set to 30, The seed for fine-tuning is fixed to 42 for reproducibility.
NAML (Wu et al., 2019a) applies the neural news recommendation with attentive multi-view learning for news representation extraction.
LSTUR (An et al., 2019) applies the neural news recommendation with long- and short-term user representations for news recommendation. NRMS (Wu et al., 2019c) utilizes multi-head self- attention for news recommendation.
Evaluation Metrics Following (Wu et al., 2020b), several evaluation metrics, including AUC, MRR, nDCG@5, and nDCG@ 10, are used for eval- uating the performance. The scores averaged across all impression logs. We train the model on MIND training sets and report the results MIND development sets.
FIM (Wang et al., 2020) employs a fine-grained in- terest matching for neural news recommendation. NRMS-BERT (Wu et al., 2021) is dual-tower PLM- neural news recommendation method that uses PLM, such BERT, for news feature extraction.
3.3 Baselines
We compare the performance of our model with multiple baselines, including general TF-IDF (Ramos et al., 2003) feature-based methods, con- volutional neural network (CNN-based) methods, deep neural network (DNN-based) methods, and PLM-based methods. Most of the experiment re- sults come from (Zhang et al., 2021b), while results of the PLM-based methods come from their origi- nal papers.
UNBERT (Zhang et al., 202 is single-tower PLM- baseed neural news recommendation method that concatenates candidate news and users' clicked news together and utilizes word-level and news- level features for click probability prediction. AMM (Zhang et al., 2021a) is single-tower PLM- based neural news recommendation method that concatenates the eross pairs of title, abstract, and body together from candidate news and users' clicked news to extract multifield matching rep- resentation for click probability prediction.
LibFM (Rendle, 2012) utilizes TF-IDF features of users' clicked news and candidate news in factor- ization machines.
3.4 Main Results
DeepFM (Guo et al., 2017) also uses TF-IDF fea- tures in deep factorization machines.
The main results are presented in Table 1. Scores MIND-Sman and MIND-Large show sig- nificant improvement over multiple baselines, in- cluding recent PLM-based neural recommendation methods like BEMS-BERT, UNBERT, and AMM. Our method completely beats the previous dual- tower methods TRRM--BRR' When comparing
DKN (Wang et al., 2018) uses information from knowledge graph and CNN networks for news feature extraction.
NPA (Wu et al., 2019b) utilizes the CNN network for news feature extracting and personalized atten- tion network incorporating user IDs.

---

AUC
MRR
nDCG@5
nDCG@ 10
LEMS-BERT (Baselinc)
69.5
34.75
37.99
43.72
w/ User Modeling
70.52 (+1.02)
34.60
38.38 (+0.39)
44.81 (+1.09)
w/ Only User Behavior Masking
70.79 (+1.29)
34.86 (+0.11)
38.65 (+0.66)
45.11 (+1.39)
w/ Only User Behavior Generation
70.92 (+1.42)
35.02 (+0.27)
38.86 (+0.87)
45.28 (+ (,66,)
w/ Both Pre-training Tasks
71.07 (+1.57)
35.19 (+0.44)
39.07 (+1.08)
45.45 (+1.73)
Table 2: Ablation study for user modeling and different pre-training tasks on MIND-Large benchmarks. Performance gains compared to the NRMS-BERY -ERT baseline are marked red.
71.1
45.65
45.6
71.05
71
45.5
(DDCG@10
71
45.45
45.4
70.95
45.35
AUC
70.8
AUC
45.3
70.9
45.25
nDCG10
nDCG10
70.85
45.15
70.7
45.2
15%
30%
45%
60%
10%
30%
50%
70%
90%
Total Mask Ratio
User Behavior Masking ratio
Figure 4: Performances with different total mask ratios a on MIND-Large Benchmark.
Performances with different user behavior masking ratios 3 on MIND-Large Benchmark.
Figure 5:
with the single-tower baseline, although AMM uti- lizes multifield matching information with single-
regression decoder necessary for the pre-training? Q5. How does performance change when choosing Siamese fully separated encoder fine-tuning? Q6. Will different pooling methods affect the per- formances?
tower encoder for click probability prediction, our dual-tower method still outperforms it by +0.93 on AUC, +0.35 on MRR, +0.3 on nDCG@5, +0.33
on nDCG@ 10 of MIND-Small benchmark. Our method outperforms UNBERT by +0.35 on AUC and +0.63 on nDCG@1 10.
Detailed analyses (A) are conducted below one by one. The seed for all experiments in this section is fixed to 42 for reproducibility.
These performance gains could be attributed to our special user modeling design with user behav- ior masking and user behavior generation tasks. These tasks are tailored for user modeling and user vector pre-training, thus bringing continuous unsu- pervised signals for effective user representation learning. We will perform detailed analysis in the next section.
4.1 Influence of User Modeling and User Behavior Pre-training Tasks
Table 2 shows the ablation results for user modeling and user behavior pre-training tasks.
A1. NRMS-BERT a typical PLM-based news recommendation baseline method. After we sim- plify the architecture of the user encoder and switch to the Siamese encoder for user modeling fine- tuning, our model achieves + +1.02 points AUC over the previous baseline.
4 Analysis
Given the performance gains of our user behav- ior pre-training methods, we systematically ana- lyze the following scientific questions in this sec- tion. Q1. How does user modeling contribute to the performances? Q2. How do behavior masking and user behavior generation contribute the performances? Q3. How do total mask ratio a and behavior mask ratio influence the performances? Q4. Is the initialization of the auto-
We attribute this increment to better user mod- eling in the encoder. The concatenation of user-clicked histories brings token-level deep inter- action within Transformers blocks, thus bringing joint modeling and better alignment to the user representation.

---

AR-Dee
AUC
MRR
nDCG@5
nDCG@10
AUC
MRR
nDCG@5
nDCG@IU 10
Pro-trained
71.07
35.19
39.07
45.45
CLS Average Attention
71.07
35.19
39.07
45.45
Random init
70.43
34.68
38.48
44.93
70.95
35.03
38.92
45.28
70.77
34.92
38.79
45.20
Table 3: Ablation study for the initialization of Auto- Regression Decoder (AR-Dec) with general corpus on MIND-Large benchmarks.
Table Ablation study for different pooling methods on MIND-Large benchmarks.
AUC
MRR
nDCG@5
nDCG@
masking ratio too high will cause much more chal- lenging task, thus unsuitable for the reconstruction of masked behaviors.
Siamese
71.07
35.19
39.07
45.45
Separated
70.32
34.61
38.37
44.84
Table 4: Ablation study for the Siamese or separated encoder fine-tuning on MIND-Large benchmarks.
4.3 Impact for the Initialization of the Auto-Regression Decoder
Since the single-layer Auto-Regression De- coder (AR-Dec) not be initialized from pre- trained checkpoint, we freeze the encoder and specifically train the AR-Dec with the same gen- eral pre-training corpus as BERT. We compare the pre-trained and random initialized AR-Dec in Ta- ble 3. Results show that the decoder needs a proper initialization on the general corpus. A randomly started decoder will corrupt the further pre-training with user behavior generation task.
A4.
A2. Besides the user modeling, we discuss the impact of different pre-training tasks. After incor- porating the user behavior masking and user be- havior generation tasks into the pre-training stage. model achieves + +1.29 and +1.42 AUC com- pared the baseline. The usage of both two tasks achieves +1.57 on AUC, which demonstrates the effectiveness both pre-training tasks.
As discussed in Section 2, the user behavior masking task encourages the model to predict the entirely masked textual span with other unmasked contextual information, thus bringing stronger signal for user behavior modeling. The user be- havior generation task utilizes a single-layer auto- regression decoder to fully generate the entire user behaviors based on the user vector, thus enrich- ing the ability the user representation during the backpropagation update from the decoder layer.
4.4 Impact Siamese or Fully Separated Encoder
A5. Performances of using Siamese fully sep- arated encoder for fine-tuning is presented in Table 4. Similar to the observation the IR community (Dong et al., 2022), sharing parameters between the news and user encoder brings better alignment of news and user vectors, thus achieving higher performance.
4.2 Influence of the Total Mask Ratio and User Behavior Masking Ratio
4.5 Impact of Different Pooling Methods
A3. Performances with different total mask ratios are shown in Figure 4. We fix the user behavior mask ratio 3 30% and change the total mask ratio between 15% and 60%. We empirically found that our pre-training method can tolerate a wide range of total mask ratios, and slightly increasing it to 30% or 45% will be helpful to achieve higher performance.
A6. Performances with different pooling meth- ods are listed in Table 5. Following (Wu et al.. 2021), three different pooling methods com- pared in works, including CLS: Directly using representation at CLS position as user vector, which is widely used to obtaín sentence embed- ding. We use this method in our main results. 2) Average: Using the average of hidden states from the last layer. 3) Attention: Using a parameterized MLP weighting network to aggregate sentence embedding from the hidden states of the last layer. Different from previous works (Wu et al., 2021). our method achieves higher scores on parameter- free pooling methods, e.g. CLS and Average, but has a lower performance on Attention pooling
The results "w/ Only User Behavior Generation" in Table 2 show that a suitable user behavior mask- ing is helpful for higher performance. Thus also adjust the user behavior masking ratio to suitable setting. The total mask ratio is fixed at 30%. We found that keeping the user behavior masking ratio less than 30% achieves higher perfor- mance. We believe that making the user behavior

---

methods. We believe that our pre-training method with user behavior gencration is tailored for direct user representation learning. And a parameterized pooling method will weaken the effort of this pre- training process.
dual-tower architecture that utilizes PLM for news embedding extraction. A hierarchical user encoder used for aggregating the user embeddings. The distances between the news embeddings and the user embeddings are calculated with dot product or cosine similarities. PLM-NR works as a classical PLM-based news recommendation baseline. Simi- lar to PLM-NR, we also use dual-tower architec- ture for inferencing. But clicked histories, rather than separated encoding, for effective user modeling.
5 Related Works
In this section, introduce traditional neural news recommendation systems and PLM-based neural news recommendation systems.
concatenated user
UNBERT (Zhang et al., 202 proposes to con- catenate candidate news and users' clicked news together and use single-tower BERT encoder for direct click probability prediction. It aggregates word-level and news-level features and achieves competing recommendation performances. AMM (Zhang et al., 2021a) pushes the steps further and proposes concatenate the cross pairs of title, ab- stract, and body together from candidate news and users' clicked news. Thus it extracts multifield matching representation for click probability pre- diction. Similar to the above methods, we jointly model the user-clicked behaviors. But we do not concatenate the candidates with the users' behav- iors, because our work focuses on user representa- tion optimization. Our work remains a dual-tower architecture for balancing effectiveness and effi- ciency. Furthermore, to the best our knowl- edge, we are the first to bring unsupervised pre- training tasks tailored for user modeling into the pre-training stage of the user model. Our pre- training methods bring better modeling ability the user encoder, which is proven to be effective with extensive experiments.
5.1 Neural News Recommendations
News recommendation systems aim to predict users' preferred news based on their browsing histo- ries. Early manual feature-based recommendation systems (Rendle, 2012; Guo et al., 2017) often em- ploy deep factorization machines for click-through rate prediction. In recent years, traditional news recommendation systems utilize neural networks, e.g. convolutional neural network (CNN) and deep neural network (DNN), for neural-based news and user feature extraction. DKN (Wang et al., 2018) utilizes knowledge-graphenee graneee-ggrp CNN networks for news feature extraction. NPA (Wu et al., 2019b) also utilizes CNN networks with personalized at- tention networks incorporating user IDs. NAML (Wu et al., 201 9a) classical DNN-based neural news recommendation with attentive multi-view learning for news representation learning. LSTUR (An et al., 2019) applics long and short-term user representations into baNed neural news rec- ommendation. NRMS (Wu et al., 2019c) is another classical DNN-based neural news recommendation method that utilizes multi-head self-attention. FIM (Wang et al., 2020) incorporates fine-grained in- terest matching into DNN-based methods. Tradi- tional neural news recommendation systems are mostly lightweight networks, designed for online inference of news recommendations.
Conclusion
This paper proposes to pre-train with user behavior modeling for better news recommendations. A sim- ple yet efficient user modeling encoder is used for better alignment of user behaviors. On top of this, we further incorporate the user behavior masking task and the user behavior generation task for ef- fective unsupervised pre- pre-training. Experiments real-world news benchmarks show that our works significantly improve the performances of PLM- based news recommendations.
5.2
PLM-based News Recommendations
The bloom of Pre-trained Language Models (PLMs), such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and UniLM (Dong et al., 2019), invigorates the development of PLM- based neural recommendation systems for higher recommendation performances. With sufficient supervised pre-training corpus, PLMs are powerful for representation extraction. Thus they are used as the backbone of PLM-based news recommenda- tion systems. PLM-NR (Wu et al., 2021) designs a
Limitations
As our pre-training method tailored for PLM- based news recommendations, it requires relatively

---

larger GPU resources than non-PLM-based meth- ods. Moreover, the joint modeling of the user en- coder brings increased performance, but the news embeddings from user behaviors cannot be pre- encoded and thus uncacheable. This hinders fast inferencing in low-resouree scenarios. We will investigate the effect of downscaling PLMs or dis- tillation for faster inferencing, and leave this to further works.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, 1901.19079129
Shugi Lu, Di He, Chenyan Xiong, Guolin Ke. Waleed Malik, Zhicheng Dou, Paul Bennett, Tin-Yan Liu, and Arnold Overwijk. 2021. Less is more: Pretrain a strong Siamese encoder for dense text retrieval using weak decoder. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 2780-2791, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ethics Statement
The authors declare that they have no conflict of interest.
Juan Ramos et al. 2003. Using tf-idf to determine word relevance in document queries. In Proceedings the first instructional conference on machine learning. volume 242. pages 29-48. Citeseer.
References
Mingxiao An, Fangzhao Wu, Chuban Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019. Neural news rec- ommendation with long- and short-term user repre- sentations. In Proceedings of the 57th Conference of
Steffen Rendle. 2012. Factorization machines with libfm. ACM Transactions on Intelligent Systems and Technology 3-3)1122.
the Association for Computational Linguistics, ACL 2019, Florence, Italy July 28- August 2, 2019, Vol- ume 1: Long Papers, pages 336-345, Association for Computational Linguistics.
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019. Bert4rec: Se- quential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on informa- tion and knowledge management, pages 1441-1450.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Heyuan Wang, Fangzhao Wu, Zheng Liu, and Xing Xie. 2020. Fine-grained interest matching for neural news recommendation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 836-845 Online. Association for Computational Linguistics.
Hongwei Wang. Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018. DKN: deep knowledge-awarc network for news recommendation. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018, pages 1835 1844. ACM.
Li Dong, Nan Yang, Wenhui Wang. Furu Wei, Xi- aodong Liu, Yu Wang. Jianfeng Gao, Ming Zhou, and Hsiao- Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Pro- cessing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13054-
Chuhan Wu, Fangzhao Wu, Mingxiao An. Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019a. Neu- ral news recommendation with attentive multi-view learning. In Proceedings of the Twenty-Eightt Inter- national Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 3863-3869, ijcai.org.
Zhe Dong, Jianmo Ni, Dan Bikel, Enrique Alfonseca, Yuan Wang Chen Qu, and Imed Zitouni. 2022. Ex- ploring dual encoder architectures for question swering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9414-9419, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019b. NPA: neural news recommendation with personalized at- tention. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pages 2576-2584, ACM.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. Deepfm: A factorization- machine based neural network for CTR prediction. In Proceedings of the Twetty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 1725-1731, ojcai.org.
Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi. Yongfeng Huang, and Xing Xie. 2019c. Neural news recom- mendation with multi-head self-attention. In Pro-

---

ceedings of 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference Natural Language Processing (EMNLP-ICCLLP), pages 6389-6394, Hong Kong, China. Association for Computational Linguistics.
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2021. Empowering news recommendation with pre-trained language models. In Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval, SIGIR '21, page 1652-1656, New York, NY, USA. Association for Computing Machinery.
Chuhan Wu, Fangzhao Wu, Tao Qi, Jianxun Lian, Yongfeng Huang, and Xing Xie. 2020a. PTUM: Pre- training user model from unlabeled user behaviors via self-supervision. In Findings of the Association for Computational Linguistics: EMNLP 2020. pages 1939-1944, Online. Association for Computational Linguistics.
Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. 2020b. MIND: A large-scale dataset for news recommenda- tion. Proceedings of 58th Annual Meeting of the Association for Computational Linguistics, pages 3597-3606, Online. Association for Computational Linguistics.
Xing Wu, Guangyuan Ma, Meng Lin, Zijia Lin, Zhongyuan Wang, and Songlin Hu. 2022. Contextual masked auto-encoder for dense passage retrieval.
Qi Zhang, Qinglin Jia, Chuyuan Wang. Jingjie Li, Zhaowei Wang, Xiuqiang 2021a. AMM: attentive multi-field matching for news recommen- dation. In SIGIR '21: The 44th International ACM SIGIR Conference on Research Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 1588 1592. ACM.
Development in
Qi Zhang. Jingjie Li, Qinglin Jia, Chuyuan Wang, Jiem- ing Zhu, Zhaowei Wang, and Xiuqiang He. 2021b. lb. UNBERT: user-news matching BERT for news rec- ommendation. In Proceedings of the Thirtieth Inter- national Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 3356-3362. ijcai.org.


Reviews:
[{"review_id": "AqMwwF4NiN", "reviewer": "Reviewer_n3WE", "summary": "", "questions": "1. Can we look at case studies to see why this structure is more efficient than other baselines? \n2. What is the effect of word representations using random initialization?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments. We will answer your questions as follows.\n\n**Q1.** The idea of user behavior pre-training is very common in recommendation systems.\n\n**A1.** Yes, user behavior modeling is prevalent in recommendation systems. However, we argue that:\n1) Previous studies often focus on training with user IDs [1, 2]. Different from previous studies, our work focuses on pre-training with textual behavior sequences to enhance the user representations, i.e. [CLS] from the user encoder. \n2) We mark our core contributions and effectiveness as designing User Behavior Masking and User Behavior Generation tasks tailored for pre-training stronger user representations. \n\n[1] PTUM: Pre-training User Model from Unlabeled User Behaviors via Self-supervision. EMNLP (Findings) 2020\n\n[2] U-BERT: Pre-training User Representations for Improved Recommendation. AAAI 2021\n\n**Q2.** According to the results in Table 2, the improvement of the model effect is gradual. Also, why not use UNBERT as a baseline for comparison, in Table 2?\n\n**A2.** Table 2 is the **ablation studies** for investigating the effects of each proposed pre-training task. The model effect is shown gradually for disassembling the performance gains and verifying that our proposed pre-training tasks all contribute to higher performances. \n\nWe choose NRMS-BERT as baselines because we derive from a dual-encoder recommendation architecture, e.g. NRMS-BERT, which first generates news vectors and user vectors and then ranks the candidate news with scores of dot product or cosine similarities.\n\nIn contrast, UNBERT is a single-encoder architecture that concatenates each candidate news and each user sequence and forwards the PLM with all combinations. UNBERT has a deep interaction of the candidate news and user sequences with self-attention, which is **not a fair comparison baseline** for our ablation studies.\n\n**Q3.** Can we look at case studies to see why this structure is more efficient than other baselines?\n\n**A3.** Here we present the case study as follows. The historical clicked news is listed in Table 1. The news recommendation task aims to rank the candidate news with historical clicked news (Or User Behaviors). Here we show the top-3 results for our method and NRMS-BERT and mark the correct recommendation in bold text. \n\nAs is shown in Table 2, our method recommends more diverse news with categories ranging in \"Travel\", \"Lifestyle\" and \"Food and Drink\", and returns the correct news as the second result. In contrast, in Table 3, NRMS-BERT biases the top-3 results all towards the \"Sports\" category. We contribute our efficient recommendation as special pre-training tasks tailored to enhancing user modeling abilities.\n\n**Table 1.** Historical Clicked News of Impression ID: 96, User ID: U118952.\n| Category                            | Title                                                                                                |\n|-------------------------------------|------------------------------------------------------------------------------------------------------|\n| News                                | Analysis: A Mike Pence Presidency Is No Longer Just Dinner Party Chatter in DC. Is He Up to the Job? |\n| News                                | Analysis: Democrats' impeachment gamble paying off in court of public opinion for now.               |\n| Sports                              | Las Vegas prepares to welcome Raiders, but is it a bad bet?                                          |\n| Travel                              | 19 of the best ski resorts in North America that don't cost a fortune.                               |\n| Sports                              | 3 ways to fix the NFL's onside kick problem.                                                         |\n| Food and Drink                      | Panera Bread worker fired after TikTok exposed frozen mac and cheese.                                |\n| Finance                             | President Trump's trillion-dollar hit to homeowners.                                                 |\n| TV                                  | 'Wheel Of Fortune' Guest Delivers Hilarious, Off The Rails Introduction.                             |\n\n**Table 2.** Recommended by Our Methods. The title marked in **bold** is the ground truth of this example.\n| Category                            | Title                                                                                                |\n|-------------------------------------|------------------------------------------------------------------------------------------------------|\n| Travel                              | Unwanted tourist types: The kinds of travelers becoming more unwelcome                               |\n| Lifestyle                           | **Why I'm Done Celebrating The Accomplishments Of Couples**  |\n| Food and Drink                      | The Real Reason McDonald's Keeps the Filet-O-Fish on Their Menu                                      |\n\n**Table 3.** Recommended by NRMS-BERT\n| Category                            | Title                                                                                                |\n|-------------------------------------|------------------------------------------------------------------------------------------------------|\n| Sports                              | Wilson deletes tweet of him standing over concussed Smith-Schuster                                   |\n| Sports                              | Warriors on pace for worst drop in winning percentage in NBA history                                 |\n| Sports                              | LeBron James on he and Tom Brady: 'We're gonna play until we can't walk'                             |\n\n\n**Q4.** What is the effect of word representations using random initialization?\n\n**A4.** Pre-training with user behavior generation needs an initialized single-layer decoder. Because we don't have a single-layer GPT to initialize this decoder, as described in Section Experiment, we first train it with Wiki+BookCorpus (which is also the BERT pre-training corpus). Random initialization in Table 3 means we omit this step, verifying that this will hurt the performances.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Experiments fully prove the effectiveness of the proposed method.\n2. The description is logical and easy to follow.\n3. The proposed architecture is straightforward and easy to understand.\n", "reasons_to_reject": "1. The idea of user behavior pre-training is very common in recommendation systems.\n2. According to the results in Table 2, the improvement of the model effect is gradual. Also, why not use UNBERT as a baseline for comparison, in Table 2?", "questions_to_authors": "1. Can we look at case studies to see why this structure is more efficient than other baselines? \n2. What is the effect of word representations using random initialization?\n", "paper_topic_and_main_contributions": "The authors propose an architecture that includes news and user representation using pre-training, which incorporate user behavior generation and user behavior masking tasks, to solve news recommendation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CZZmqI7wwN", "reviewer": "Reviewer_YPna", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments. We will answer your questions as follows.\n\n**Q1.** Data Contamination issue. \n\n**A1.** Thanks for your very detailed insights. \n\n1) We believe that our work does not have such a data contamination issue. In paper [1], it pre-trains with both training corpus and **human labels** in downstream test sets. However, we only pre-train with 2.2 million click behaviors from the training set of MIND, which does not contain any labels from impression logs. \n2) Our work can be viewed as in-domain pre-training techniques. In-domain pre-training is actually prevalent in NLP communities. For example, GPT-3 is pre-trained with Wikipedia and Web pages. Popular downstream tasks like Natural Questions [2] and TriviaQA [3] are also made from Wikipedia. A popular pre-training architecture in the IR community, named Condenser [4], is pre-trained with MS-MARCO web collections [5]. It also fine-tuned and tested on MS-MARCO and TREC [6] benchmarks, which are all related to MS-MARCO collections.\n3) Overall, we think that the data scopes in pretraining and fine-tuning could overlap, especially in the in-domain tasks, under the premise of ensuring that the labels do not leak.\n\n[1] Data Contamination: From Memorization to Exploitation, ACL-2022\n\n[2] Natural Questions: A Benchmark for Question Answering Research. Trans. Assoc. Comput. Linguistics 7: 452-466 (2019)\n\n[3] TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. ACL (1) 2017: 1601-1611\n\n[4] Condenser: a Pre-training Architecture for Dense Retrieval. EMNLP (1) 2021: 981-993\n\n[5] MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. CoCo@NIPS 2016\n\n[6] Overview of the TREC 2019 deep learning track. CoRR abs/2003.07820 (2020)\n\n**Q2.** Advanced news recommendation baselines should be compared.\n\n**A2.** Thanks for your suggestions. \n\n1) Our work focuses on the pre-training stage, which is orthogonal to fine-tuned methods. Compared with advanced fine-tuned recommendation architectures may not be a very fair practice. For example, HieRec uses hierarchical user interest matching. MINER uses multi-interest user modeling with multi-field information. MTRec uses additive attention and gradient surgery. \n2) Our architecture merely uses a Siamese encoder (one BERT model) as both the user and news encoder for recommendation. It's simple and uses fewer parameters than the above methods.\n\nWe will consider exploring combining our pre-training methods with advanced fine-tuning architecture in our extended works.\n\n\n**Q3.** Equation 4 is not right.\n\n**A3.** Thanks for pointing this out. We will correct this in our next versions.\n\n**Q4.** It is too difficult for the decoder to generate such hierarchical and complex user behaviors only based on a user representation vector. \n\n**A4.** Yes, this task is difficult. We believe that besides relying on the self-attention of GPT, this difficult task will encourage the GPT to rely more on the information within the [CLS] vector to finish the generation of user behaviors. In this way, we seek to \"compress\" the information of user behaviors into the [CLS] vector. Thus the enhanced representations will do better in downstream recommendations.\n\nAs recorded in our pre-training logs, The final MLM loss for the BERT encoder is 1.8739, and the final CLM loss for the decoder is 1.6354. Thus the decoder is actually capable of finishing such a generation task. \n\n**Q5.** Initializing the generative decoder with the original BERT is irrational.\n\n**A5.** Actually, we do not initialize the generative decoder from BERT. Because there is no off-the-shelf single-layer GPT for initialization of the decoder, as described in the Experiments Section 3.1, we first pre-train the decoder with the Wiki+Bookcorpus, the same pre-training corpus as BERT. Then we use our proposed structure for pre-training with in-domain corpus MIND. Note that we discard the decoder in downstream fine-tuning, as it is only designed for the pre-training stage. We can view the decoder as an assistant for CLS representation of the encoder to learn better user behavior modeling ability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper explores user behavior pretraining, which is less studied by previous research. The idea is generally well stated and experiments are thorough.", "reasons_to_reject": "1. The major weakness of this paper is the overlap of pretraining and finetuning data scopes. In principle, pretraining and finetuning data should not be overlapped. Otherwise, data contamination [1] makes the pretrained model memorize the data and cannot be well generalized to various downstream tasks. The essence (or target) of PLM is to pretrain on large-scale unlabeled text corpus and finetune on various downstream tasks. For example, GPT-3 and T5 [2,3] are pretrained on Common Crawl, WebText and Wikipedia corpus and finetuned on downstream supervised tasks. Pretraining on task-specific data makes the model difficult to be adapted to other tasks. The authors pretrain the model on MIND and also finetune on MIND, which is irrational and should not be called pertaining (it can be regarded as typical training), unless the author tests the model on other datasets or tasks. Pretraining and finetuning on the same dataset MIND is not a good setting for PLM study. Especially note that MIND-small is a subset of MIND-large, the MIND-small test set is also included in pertaining MIND-large corpus.\n\n2. As the authors use relatively high-cost computation resources to \"pretrain\" the model of large parameters, advanced news recommendation baselines should be compared, including [4,5,6,7]. Fairly speaking, the chosen baselines are not updated to advanced news recommendation works of the recent two years.\n\n3. Equation 4 is not right where $t_{<i}$ is either missing or not self-consistent with the notation of $E(T)$.\n\n4. Minor concern of mine: Masked user behavior modeling is intuitive. However, incorporating GPT-like user behavior generation by introducing an additional decoder seems redundant and unnecessary, though the authors claim with experiments that it can help to learn the representation of user vectors. Since the user behavior and its containing news articles are represented in a hierarchy, I also doubt that it is too difficult for the decoder to generate such hierarchical and complex user behaviors only based on a user representation vector. Considering that the user representation vector [CLS] is a one-dimension vector (not like the $n\\times d$ hidden states of Transformer or T5 encoder outputs), I doubt if this vector can generate the whole complex user behaviors.\n\n5. Initializing the generative decoder with the original BERT is irrational. Moreover, using BERT's architecture to model generation is not a good choice. BERT's architecture is designed and pretrained for Mask Language Modeling, and the authors revised it to do autoregressive Language Modeling (CLM in the paper). The pretraining gap between Mask Language Modeling and Language Modeling exists.\n\n\n[1] Data Contamination: From Memorization to Exploitation, on ACL-2022\n\n[2] Language Models are Few-Shot Learners, on NIPS-2020\n\n[3] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, on JMLR-2020\n\n[4] HieRec: Hierarchical User Interest Modeling for Personalized News Recommendation, on ACL-2021\n\n[5] MINER: Multi-Interest Matching Network for News Recommendation, on ACL-2022\n\n[6] MTRec: Multi-Task Learning over BERT for News Recommendation, on ACL-2022\n\n[7] DIGAT: DIGAT: Modeling News Recommendation with Dual-Graph Interaction, on EMNLP-2022", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a pretraining model PUNR for news recommendation. PUNR is pretrained by two objectives of BERT-like masked behavior modeling and GPT-like autoregressive behavior generation. Comprehensive experiments show that PUNR beats several news recommendation baselines.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In line 005, 'works are mostly focused on' should be 'works mostly focus on'.\n\nThe terminology \"auto-regression\" should be \"auto-regressive\" which is standard used in our NLP community."}, "extra_text": {}}, {"review_id": "1YsBYrLcTO", "reviewer": "Reviewer_ed5M", "summary": "", "questions": "1. Why are  some methods mentioned in related works especially those for PLM-based news recommendation not compared in experiments?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments. We will answer your questions as follows.\n\n**Q1.** About writing flaws and more datasets.\n\n**A1.** Thank you for your suggestions. We will correct them immediately and include more datasets in our extended works.\n\n**Q2.** Methods mentioned in related works are not compared in experiments.\n\n**A2.** Note that NRMS-BERT is actually from PLM-NR [1], which has already been compared in our works. We choose NRMS-BERT for a fair comparison. Nevertheless, we will include more related latest baselines in our next version. Thanks again for your suggestions.\n\n[1] Empowering News Recommendation with Pre-trained Language Models. SIGIR 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem studied in this paper is interesting and important, but rarely researched. Thus, this research has some novelty.\n\n2. The proposed pretraining methods, i.e., user behavior masking to recover masked behaviors and user behavior generation to enhance user representations, are reasonable, although they are standard techniques in language modeling.\n\n3. The experiments are extensive and solid. The improvement brought by the proposed method is significant.", "reasons_to_reject": "1. There are a few writing flaws. For example, there should be some punctuations after the equations.\n\n2. Besides the two MIND datasets, more datasets could be used in experiments.", "questions_to_authors": "1. Why are  some methods mentioned in related works especially those for PLM-based news recommendation not compared in experiments?", "paper_topic_and_main_contributions": "This research proposes an unsupervised pre-training method for news recommendation based on user behaviors. The authors introduce two tasks: user behavior masking to recover masked behaviors and user behavior generation to enhance user representations. Their approach outperforms existing methods in real-world news recommendation benchmarks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "There should be some punctuations after the equations."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a pretraining model PUNR for news recommendation. The proposed PUNR is pretrained by two objectives of BERT-like masked behavior modeling and GPT-like autoregressive behavior generation. 


Pros:
The proposed paper is backed by thorough experiments, demonstrating its effectiveness. The straightforward and understandable architecture of the proposed method simplifies its comprehension. Furthermore, the pretraining methods are considered reasonable and well-reasoned, aligning with established practices in language modeling.

Cons:
Despite these strengths, there are some concerns raised by the reviewers. The novelty of this paper is relatively limited.  Furthermore, the gradual improvement observed in the experimental results has been identified as a drawback. 
Concerns about data overlap between pretraining and finetuning data are significant, with suggestions that this could lead to data contamination and impact the generalization of the model. There are also recommendations to compare the proposed method with recent news recommendation baselines, address equation inconsistencies, evaluate the necessity of an additional decoder, rectify minor writing flaws, and consider using a more diverse set of datasets in experiments to enhance the paper's overall robustness.
Decision: Accept-Findings