Bag of Tricks and A Strong Baseline for Neural News Recommendation

BAG OF T RICKS AND A S TRONG BASELINE FORN EURAL N EWS R ECOMMENDATION
Yichao LuLayer 6 AI
yichao@layer6.ai

A BSTRACTPersonalized news recommender systems play an important role in improving online news reading experience. However, news recommendation still remains agreatly unexplored research field, partly due to the lack of a high-quality benchmark dataset in the news recommendation community. The MIND News Recommendation Competition, which is based on the Microsoft News Dataset (MIND),is a first-of-its-kind competition dedicated to benchmarking existing news recommendation algorithms and to advancing the state-of-the-art. This technical reportpresents a strong baseline for neural news recommendation, built on top of theNRMS model with a bag of task-specific tricks. The proposed baseline approachachieved the second place out of over 200 teams in the MIND News Recommendation Competition, with less than 0.4% relative difference in terms of AUC fromthe first place team.

1I NTRODUCTION

With the explosive growth of digital news, personalized news recommender systems have become
an ever-present part of online news browsing platforms. Through learning user preferences fromuser behavior logs, personalized news recommender systems are able to present the right news to
the right user in the right time.
News recommendation is a challenging task in the sense that (i) it is important for news recommendation models to understand news content in a fine-grained manner so as to capture user interests,(ii) there is often no explicit feedback from users, and (iii) news articles are produced daily andtheir ephemeral timelife requires accurate modelling of short-term user preferences. Therefore offthe-shelf recommendation algorithms typically cannot achieve desirable performance in the task ofnews recommendation. Despite the necessity of recommendation algorithms specifically designed
for news recommendation, not least because the news recommendation community lacks an authoritative benchmark dataset.
The Microsoft News Dataset (Wu et al., 2020) is dedicated to facilitating the research in news recommendation. To date the Microsoft News Dataset (MIND) is the largest news recommendation
dataset, consisting of about 160k English news articles and more than 15 million impression logs
generated by 1 million users. The accompanying MIND News Recommendation 1 serves as a testbed
for evaluating the performance of various news recommendation models. Given the anonymized behavior logs collected from the Microsoft News website 2 , the task is to rank the candidate news ineach impression log so that clicked news appear at the top of the list, while non-clicked news appear
at the bottom of the list.
This technical report demonstrates that a properly tuned NRMS model (Wu et al., 2019b), whencombined with a bag of task-specific tricks, serves as a very strong baseline for neural news recommendation, showing highly competitive performance in the MIND News Recommendation Competition. The final solution, which is based on an ensemble of NRMS models only, achieved thesecond place out of over 200 teams, with less than 0.4% relative difference in terms of AUC fromthe first place team. In addition, this technical report describes a number of alternative approaches
12

https://msnews.github.io
https://microsoftnews.msn.com

1Bag of Tricks and A Strong Baseline for Neural News Recommendationthat have been explored during the competition, together with a discussion of the potential cause of
their inferior performance compared to the optimized NRMS model.

2A PPROACH

The NRMS model (Wu et al., 2019b) is a state-of-the-art neural news recommendation approachwith multi-head self-attention. NRMS learns news representations from news titles, using a multihead self-attention network. Briefly speaking, the news encoder first maps each word in the news
title to the the corresponding vector, and then uses the self-attention network to learn word-level representations. Finally, a query vector is used to locate the important words in the news title, and anattention-based pooling method is used to aggregate the word-level representations into the learned
title representations. In order to learn user representations from their browsed news, NRMS again
uses the multi-head self-attention network on top of the learned news representations. The probability of a user clicking a candidate news is given by the dot product between the user representations
and the news representations.
2.1

B I -LSTM WITH ATTENTION M ECHANISM

While self-attention based methods have shown superior performance in a variety of natural language processing tasks, especially in pre-trained language models with Transformers, the bidirectional LSTM remains a popular alternative when modelling short texts, and when pre-training is notused. The promising performance of bidirectional LSTM in terms of news modelling has also beenreported in Wu et al. (2019b).
According to the experiments during the competition, several tricks can further improve the performance of the bidirectional LSTM based model in the task of neural news recommendation. These
inculde: using fastText instead of Glove for initializing the word vectors, orthogonal initialization ofthe LSTM weights, data augmentation on the text input (e.g., shuffle, replace, and using synonyms),
and using the lazy Adam optimizer for sparse word embeddings update.
2.2

P OSITION E MBEDDINGS

While the self-attention mechanism can effectively capture the long-term dependency within sequences, it does not have the inherent ability of RNN based approaches to reason about the absolution and relative positions. In other words, the standalone self-attention network model is inherentlyposition-agnostic. However, the order in which the user viewed the news is important for modellingshort-term user preference. One would thus expect that enabling position-aware self-attention for
the NRMS model would lead to a boost in terms of performance.
The common approach to enabling position-aware self-attention is to introduce the position embedding layer, which essentially encodes each position into a vector representation. It should be noted
that, in order to deal with the variable lengths of user history, it is important to ensure that the userbrowsing history is reversed before feeding into the position embedding layer. In other words, thelast viewed news for all users share the same embedding vector, and the second-to-last viewed news
for all users share the same embedding vector, etc. Additionally, experimental results suggest thatthe sinusoidal initialization of the position encoding can result in improved performance in comparison with random initiliazation.
2.3

M ULTI - VIEW R EPRESENTATION L EARNING

The original NRMS model only learns news representations from titles. However, news articles
often contain rich information such abstract and body, which is helpful for learning better news
representations. In Wu et al. (2019b), the authors show that the attentive multi-view learning (AMV)
approach (Wu et al., 2019a), which models different sources of news information independently
and fuses the learned representations via the attention mechanism, outperforms the simple baselinethat directly concatenates all sources of information into a long document. However, experimental
results during the competition suggest that, simply averaging the representations learned from themulti-views is able to get better performance than the AMV approach. Averaging can be regarded asa special case of the attention-based approach where the model assigns uniform view-level weights.
2Bag of Tricks and A Strong Baseline for Neural News Recommendation

This suggests that, given sufficient training data, a neural network based model can learn to perform
multi-view learning without the need of designing a specific attentive multi-view learning module.
2.4

S CALED D OT-P RODUCT ATTENTION WITH PARAMETER S HARING

For the original NRMS model, the representation learned by the k-th attention head is formulated asexp(eTi Qk ej )aki,j = PM,T
m=1 exp(ei Qk em )hi,k = Vk (MXaki,j ej ),

(1)

(2)

j=1

where Qk and Vk refer to projection matrices, and aki,j denotes the relevance score between the i-thnews and the j-th news.
Recently studies have found that in particular tasks, weight sharing in the self-attention layer canlead to better performance, presumably due to the fact that weight sharing inherently provides regularization to the model. In the MIND dataset, weight sharing again turns out to be an effective
approach. In addition, the scaled dot-product attention proposed by Vaswani et al. (2017) is an effective approach to stabilize the training of the self-attention based models by enforcing the output
to have unit variance. Thus the modified scaled dot-product attention with parameter sharing can beformulated as follows:
For the original NRMS model, the representation learned by the k-th attention head is formulated asexp( √1d eTi QTk Kk ej )kkai,j = PM,T T√1m=1 exp( dk ei Qk Kk em )hi,k = Kk (MXaki,j ej ),

(3)

(4)

j=1

where √1d is a scaling factor described in Vaswani et al. (2017).
k2.5

E NSEMBLE

The ensemble technique is a widely used strategy to improve the performance of machine learningmodels. Ensemble methods work by aggregating the predictions of a number of single models.
There are two common strategies to ensemble models: bagging and boosting. Since NRMS is aneural network based model, it is naturally a low bias and high variance model. Therefore bagging
based ensemble works better than boosting for NRMS since it can help reduce the bias.
Three different ensemble approaches have been experiment during the competition:• Rank ensemble: using the averaged rank predicted by single models as the prediction by
the ensemble model.
• Raw logits ensemble: using the average of the pre-softmax logits as the prediction by theensemble model.
• Scaled probabilities ensemble: using the average of the unnoramlized softmax activations
as the prediction by the ensemble model.
Scaled probabilities ensemble works the best among all the three explored approaches, due to thefact that the scaled probabilities more accurately reflect each single model’s confidence about thepredicted click.
3

Bag of Tricks and A Strong Baseline for Neural News Recommendation

3A LTERNATIVE A PPROACHES

To facilitate the research in news recommendation, this section describes some other explored yet
unsuccessful approaches during the competition, and discusses the potential causes of failure ofthese approaches. It should be noted that, while all these approaches fail to outperform the NRMS
model under the settings of the MIND competition, i.e., to predict the click behaviors for one week
in the future, some approaches are superior when considering a shorter horizon of time; see Section
4 for more details.
3.1

E ND - TO - END F INE - TUNING WITH BERT

Wu et al. (2020) reported an improved performance in terms of news recommendation quality whenreplacing the word embedding layer in the news encoder with a pretrained-then-finetuned BERT
model. However, such model is cumbersome and is infeasible to run in a timely manner. Therefore,in the interest of time, an alternative approach, which directly uses pooled representations fromBERT as the news representations, was explored during the competition. However, this approach
failed to achieve comparable performance against the bidirectional LSTM based approach. The
possible reason that such way of using end-to-end fine-tuned BERT is not showing improvement isthat the pooled representation of BERT is more suitable for the classification task.
3.2

F EATURE E NGINEERINGIn traditional recommender systems problems, hand-engineered features can significantly boost theperformance of machine learning models. Experiments on the MIND dataset show that, when themodel is equipped with engineered features such as the length of the user history and the numberof news in the user history with the same category as the candidate news, the model achieves better
performance on the validation set, while at the same time the test set performance drops. The
result suggests that engineered features probably have less generalization ability compared to neuralnetwork based models like NRMS, hence the need for research on neural news recommendation.
3.3T WO - STAGE R E - RANKING

The two-stage re-ranking model is a common technique used in a lot of data science competitions
Volkovs et al. (2018). On the MIND dataset, a two-stage re-ranking model which uses the NRMS
as the first stage and XGBoost as the second stage is able to improve the NRMS baseline modelby over 10% in terms of AUC on the validation set, while getting significantly worse performanceon the test set. This suggests that two-stage re-ranking models are more suitable for making shortterm predictions, while neural news recommendation models such as NRMS has more consistent
performance over a longer period of time.
3.4

T RAINING E XPERT M ODELSIn the MIND dataset, the model needs to handle different recommendation scenarios, e.g., recommending news when there is different number of available history news. Instead of applying one
model to solve all these tasks, one common practice is to train individual models each for solving
one particular task (Lu et al., 2019). The rationale behind this is that, training expert models thatspecialize in one particular task allows the model to learn task-specific patterns and at the same timeenables easier optimization. While such approach is shown to be effective on a number of othertasks, on the MIND dataset, it fails to bring about gains. This observations suggests that the NRMS
model, especially the self-attention model it uses, is well-suited for news recommendation in all
kinds of scenarios.
3.5

A DDRESSING C OLD - START WITH P OPULARITY

Cold start is a common phenomenon in news recommendation. For example, about 1.2% of theimpression data in the MIND test data is cold-start. A common approach to addressing the coldstart problem is to recommend the most popular item to the user, i.e., recommending the most clickednews in the task of news recommendation. However, the success of such method relies heavily on4Bag of Tricks and A Strong Baseline for Neural News Recommendation

Table 1: Ablation study on the MIND News Recommendation Competition.
MethodValidation AUC

Test AUC

Default NRMS
+ Bi-LSTM+ Position encoding+ Multi-view learning+ Attention with parameter sharing+ BERT fine-tuned+ Feature engineering+ Two-stage re-ranking+ Expert models+ Popularity

0.68560.68720.6951
0.70740.71220.68320.70310.77330.68450.68770.68260.68550.69230.70330.7042—–0.65450.64860.68250.6814

Table 2: Final leaderboard results for the top-5 teams. The result of the solution described in thistechnical report is in bold.
Rank

TeamAUCMRR

nDCG@5nDCG@1012
3
3
3

chenghuige
oahciyRavox
Qinne
gcc microsoft

0.71310.70960.70480.7032
0.69790.36080.35400.35050.3496
0.34790.39600.38830.38450.38300.38060.45210.44540.44160.43976
0.4373the how recent the supervised data is, so that the model can accurately estimate the popularity of thenews. Therefore, while the popularity based method show improved performance on the validation
set (predicting click behaviors in the next day), it fails to achieve desirable performance on the testset (predicting click behaviors in the next week).

4E XPERIMENTS

4.1

I MPLEMENTATION D ETAILSThe model is trained with the Adam optimizer with a batch size of 32 and a learning rate of 1e − 4.
Following (Wu et al., 2019b), the max length of the user history is set to be 50. For training thetwo-stage re-ranking model, the first six days of the training data is split to be the first stage training
data, and the last day of the training data is split to be the second stage training data.

4.2

R ESULTS

An extensive ablation study is shown in Table 1. The test performance for some of the approaches
is unavailable because the MIND News Recommendation Competition adopts a very rigorous evaluation protocol, and each team is only allowed to submit the test results for evaluation for a limited
number of times. It should also be noted that the test AUC reported here refers to the public leaderboard performance, which is evaluated on 10% of the test data.
The final leaderboard results for the top 5 teams is presented in Table 2. The final solution, whichbased on an ensemble of the optimized NRMS model, achieves competitive performance against thesolutions provided other teams.
5

Bag of Tricks and A Strong Baseline for Neural News Recommendation

5C ONCLUSION

This technical report demonstrates that a properly tuned NRMS model (Wu et al., 2019b), whencombined with a bag of task-specific tricks, serves as a very strong baseline for neural news recommendation. An extensive ablation study proves the effectiveness of each proposed trick.

R EFERENCES
Yichao Lu, Cheng Chang, Himanshu Rai, Guangwei Yu, and Maksims Volkovs. Learning effective
visual relationship detector on 1 gpu. arXiv preprint arXiv:1912.06185, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information
processing systems, pp. 5998–6008, 2017.
Maksims Volkovs, Himanshu Rai, Zhaoyue Cheng, Ga Wu, Yichao Lu, and Scott Sanner. Two-stage
model for automatic playlist continuation at scale. In Proceedings of the ACM Recommender
Systems Challenge 2018, pp. 1–6. 2018.
Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. Neural news recommendation with attentive multi-view learning. arXiv preprint arXiv:1907.05576,2019a.
Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. Neural news recommendation with multi-head self-attention. In Proceedings of the 2019 Conference on EmpiricalMethods in Natural Language Processing and the 9th International Joint Conference on NaturalLanguage Processing (EMNLP-IJCNLP), pp. 6390–6395, 2019b.
Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing
Xie, Jianfeng Gao, Winnie Wu, et al. Mind: A large-scale dataset for news recommendation. In
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.
3597–3606, 2020.

6