UniTRec: A Unified Text-to-Text Transformer and Joint Contrastive Learning Framework for Text-based Recommendation
Year: 2023

Abstract:
Prior study has shown that pretrained language models (PLM) can boost the performance of text-based recommendation. In contrast to previous works that either use PLM to encode user history as a whole input text, or impose an additional aggregation network to fuse multiturn history representations, we propose a unified localand global-attention Transformer encoder to better model two-level contexts of user history. Moreover, conditioned on user history encoded by Transformer encoders, our framework leverages Transformer decoders to estimate the language perplexity of candidate text items, which can serve as a straightforward yet significant contrastive signal for user-item text matching. Based on this, our framework, UniTRec, unifies the contrastive objectives of discriminative matching scores and candidate text perplexity to jointly enhance text-based recommendation. Extensive evaluation shows that UniTRec delivers SOTA performance on three text-based recommendation tasks.1

heading: 1 Introduction
text: Text-based recommendation (Li et al., 2010; Gu et al., 2016; Okura et al., 2017; Malkiel et al., 2020) aims to recommend relevant textual content (e.g., news articles, Twitter posts) to people based on their behaviors as represented in historical log texts. For instance, engagement recommendation (Cheng et al., 2022) on social media (e.g., Twitter and Reddit) helps users discover and engage with interested threads by modeling their browsing history.
Pretrained language models (Devlin et al., 2019; Brown et al., 2020) have made waves in recent text-based recommendation research (Zhang et al., 2021; Qi et al., 2022; Geng et al., 2022). The most common practice is using PLM encoders (BERT family) to learn representations of user history and candidate item texts. Recommendation
1Our code is available at https://github.com/Veasonsilverbullet/UniTRec.
matching scores are computed over the user and item representations and finally optimized by noise contrastive estimation (NCE) loss (Gutmann and Hyvärinen, 2010) for ranking multiple candidates.
Unlike encoding single text, using PLM to encode multi-turn texts of user history is nontrivial. Existing works (Malkiel et al., 2020; Qi et al., 2022; Geng et al., 2022) concatenate multi-turn history texts as a whole input text, then use one PLM encoder to learn the holistic user representation. This is a standard PLM encoding manner but ignores the relation among history turns, as all word tokens from different history turns are equally attended2. In contrast, previous studies point out that learning the relation among user history turns is also beneficial (Zeng et al., 2020; Qi et al., 2021). Another approach is using PLM encoders to learn representations from multi-turn history texts, followed by an additional aggregation network to fuse the multi-turn representations (Wu et al., 2021; Li et al., 2022). However, the imposed aggregation networks (with newly initialized parameters) weaken the representation power of PLM encoders which are already pretrained on large-scale corpora.
This work introduces UniTRec, a Unified text-totext Transformer framework for text-based Recommendation. In the encoder component of UniTRec, we design local- and global-attention to learn user history representations through tailored attention masking, which aims to jointly model word-level and turn-level relations of user history. UniTRec can utilize the full power of PLM encoders because it preserves the intact structure of PLM encoders without newly imposed parameters.
Different from most previous works that predict user-candidate matching scores solely based on the representations learned by Transformer encoders, we argue that conditioned on user representations
2There is no inductive bias of turn-level and history-level relations introduced to Transformer self-attention computation, where each token plays an equal role.
1160
learned by Transformer encoders, candidate text perplexity (PPL) estimated by pretrained Transformer decoders is also a straightforward yet significant signal for text-based recommendation. As shown in Figure 1, we hypothesize that the candidate text perplexity estimated by pretrained LM decoders can directly measure the text matching degree between user history and candidate texts. It is because the perplexity estimates the likelihood of candidate texts based on encoder outputs, which naturally indicates the probabilities of candidate texts given the user history. Besides, UniTRec can use the last hidden states of Transformer decoders to directly predict matching scores. Hence, this work unifies the contrastive objectives of discriminative matching scores and candidate text perplexity to jointly enhance text-based recommendation.
The contributions of this work are: (1) We propose local- and global-attention to model two-level relation of user history without additional parameters, which enjoys the full power of PLM encoders. (2) We introduce PLM perplexity to measure usercandidate text matching and unify the objectives of discriminative matching scores and candidate text perplexity to enhance text-based recommendation. (3) Experiments on three text-based recommendation datasets validate the effectiveness of UniTRec.

heading: 2 Approach

heading: 2.1 Unified User-history Modeling
text: Formally, multi-turn history of a user is represented as H = [t1, t2, ..., tN ], and each turn text ti contains |ti| words as ti = [x1i , x2i , ..., x |ti| i ]. UniTRec aims to unify learning word- and turn-level context representations in one Transformer encoder.
Local attention on word-level context. We first
concatenate the multi-turn history texts as the input tokens X = [x11, x 2 1, ..., x |t1| 1 , ..., x 1 N , x 2 N , ..., x |tN | N ]. Inspired by Dong et al. (2019), we tailor the attention masking in Transformer self-attention to learn the word-level context of each turn. Specifically, we allow word tokens from the same turn to attend to each other, while tokens from different turns are excluded from self-attention computation:
Mi,j = { 0, token xi and xj in the same turn −∞, otherwise
Attention(Q,K, V ) = softmax( QKT√
dk +M)V
(1) , where Q,K, V are self-attention query, key, and value in Vaswani et al. (2017), M is the mask matrix to achieve local-attention inside each turn text. The local self-attention blocks consist of L1 layers, by which original PLM encoders can be adapted to learn word-level context representations of turns.
Global attention on turn-level context. Over the local self-attention layers, we leverage global self-attention to model the relation among history turns. Specifically, tokens from all turns attend to each other in self-attention computation (by setting the mask matrix M = 0). In this way, Transformer encoders can perform global interaction among each token (and turn) to learn turn-level context representations of user history. There are L2 layers in the global self-attention blocks, which can also be inherited from PLM encoders directly.

heading: 2.2 Joint Contrastive Ranking Objectives
text: Conditioned on the history representation, we input the candidate text to Transformer decoders to predict how likely it should be recommended. It is worth noting that Transformer decoders can naturally perform effective cross-attention interaction between history and candidate hidden states.

heading: 2.2.1 Objective on Discriminative Scores
text: Motivated by Lewis et al. (2020), we feed the last hidden state of decoder output hT to an MLP scorehead which predicts the user-candidate matching score Sd = ScoreHead(hT ). The matching score is discriminative, as higher scores indicate higher user-candidate matching probabilities.
Following previous works (Li et al., 2022; Qi et al., 2022), we adopt negative sampling with NCE loss to optimize matching score prediction. Given the user history and its ground truth matched candidate Ci, UniTRec predicts the matching score
as Sd+i . In addition, K unmatched negative candidates {Cj}Kj=1 are sampled from the candidate set, and their matching scores are {Sd−j }Kj=1. The NCE loss is represented in a contrastive form:
Ldi = − log exp(Sd+i )
exp(Sd+i ) + ∑K j=1 exp(S d− j ) (2)

heading: 2.2.2 Objective on Candidate Text Perplexity
text: As aforementioned, UniTRec leverages perplexity to rank candidate texts. Since lower perplexity indicates higher user-candidate matching probability, regarding the candidate text Y = [y1, y2, ..., yT ], we define the perplexity-based matching score Sp as its negative perplexity3:
Sp = −PPL(Y ) = 1 T ∑T i=1 log pθ(yi|y<i) (3)
, where pθ(·) denotes the target probability output from the UniTRec Transformer decoder. Similar to Eq. (2), we optimize the perplexity-based matching score Sp in the NCE loss form. As perplexity empirically varies in a wide range, we introduce a temperature parameter τ to balance the joint NCE loss gradients following Radford et al. (2021).
Lpi = − log exp(τ · Sp+i )
exp(τ · Sp+i ) + ∑K j=1 exp(τ · S p− j )
(4) , where τ is learnable and initialized to 1. On the training dataset D, the joint contrastive learning objective is formulated as:
L = ∑|D|
i=1
( Ldi + Lpi ) (5)
3Note https://huggingface.co/docs/transformers/perplexity for LM perplexity calculation. We empirically discard the outer exponential term in the PPL formula, because it already exists in NCE loss Eq. (4) and does not affect the final ranking.

heading: 2.3 Model Initialization and Inference
text: As UniTRec is a standard text-to-text Transformer, we initialize the parameters from pretrained BART (Lewis et al., 2020). In inference, UniTRec predicts the discriminative and perplexity-based scores for each candidate item, respectively. The two separate scores Sd and Sp are normalized, averaged, and finally ranked as the output. Detailed ranking process is provided in Appendix B.

heading: 3 Experiments
text: We evaluate UniTRec on three text-based recommendation tasks: 1) NewsRec, to recommend news articles to users based on their browsing history. We use the MIND-small dataset (Wu et al., 2020) for experiments. 2) QuoteRec, to recommend quotations to users based on their conversation history. We use the Reddit-quotation dataset (Wang et al., 2021) for experiments. 3) EngageRec, to recommend social media posts for users to engage with based on their comment history. We use the dataset released by Zeng et al. (2020) for experiments. Detailed dataset statistics is provided in Appendix A.
Implementation Details. The UniTRec encoder and decoder both consist of 6 Transformer layers with 768-dimensional hidden states and 12 attention heads. We set L1 = 3 and L2 = 3. We use AdamW optimizer (Loshchilov and Hutter, 2019) to train UniTRec with cosine learning rate decay.
Baselines. We compare UniTRec with competitive baselines: 1) GRU4Rec (Balázs et al., 2016) utilizes a GRU network to learn multi-turn history. 2) SASRec (Kang and McAuley, 2018) encodes user history with a self-attention based sequential model. 3) BERT4Rec (Sun et al., 2019) employs bidirectional self-attention to model user history. 4) RoBERTa-Sim, a simple yet strong baseline men-
tioned in Qi et al. (2022), uses the hidden states of [CLS] tokens to measure user-candidate similarity. 5) UNBERT, implemented as Zhang et al. (2021), concatenates history and candidate texts as the input to BERT and predicts matching scores from the final hidden states of [CLS] tokens.
Note that we do not consider other methods that use non-text inputs (e.g., user profile, text topic labels). For fair comparison, all baseline models use pretrained 12-layer RoBERTa-base (Liu et al., 2019) as text encoders to learn embeddings of texts.

heading: 3.1 Main Results
text: Table 1 shows the performance of experiment models. From the results of NewsRec and QuoteRec, we can see that UniTRec outperforms all baseline models by a clear margin. Also, RoBERTa-Sim and UNBERT that directly use the [CLS] hidden states to represent user history, surpass other baselines that build additional aggregation networks upon the whole RoBERTa outputs. As displayed in the results, EngageRec is the most difficult task. We inspect the dataset and find that the texts on social media contain too much noise (e.g., URL and emoji), and the user history contains less number of turns. Nevertheless, UniTRec achieves better overall performance than other baseline models, validating its robustness on noisy text inputs and limited user history.

heading: 3.2 Ablation Studies and Analyses
text: We further conduct ablation studies on UniTRec. The experiment results are reported in Table 2.
Initialization of UniTRec. We train UniTRec from scratch without initialization from pretrained BART (refer to w/o BART Init). The recommendation performance significantly drops in all three tasks, which indicates that acquiring effective text understanding ability from PLM is a necessary key to UniTRec performance.
Local and global attention. We investigate the function of two-level attention modules of the UniTRec history encoder. Concretely, we set L1 = 0 in w/o Local-Att and L2 = 0 in w/o Global-Att, where L1 + L2 = 6. We can observe that removing local and global attention from the original UniTRec history encoder both lead to suboptimal performance, while the performance drop is more significant in w/o Global-Att. The results justify the effectiveness of jointly modeling two-level history contexts through adapted Transformer attention masking without additional parameters.
Discriminative and perplexity-based objectives. We probe into training UniTRec with standalone discriminative (Disc-Score only) and perplexitybased (PPL-Score only) contrastive objectives, respectively. We can see that the discriminative objective yields better performance than the perplexitybased objective. Besides, the model performance on both standalone objectives declines compared to the original joint objective. The results indicate that the discriminative and perplexity-based matching scores are complementary and can jointly provide more accurate signals of user history and candidate text matching for text-based recommendation.

heading: 4 Conclusion
text: We present a unified Transformer UniTRec for textbased recommendation. UniTRec learns two-level contexts of multi-turn user history and jointly exploits discriminative matching scores and candidate text perplexity as matching objectives. Empirical experiments on three text-based recommendation datasets corroborate the effectiveness of UniTRec.

heading: 5 Limitations
text: Our model only focuses on utilizing text information for recommendation, which is a key limitation of this work. In real-world settings, recommender systems are usually required to handle heterogeneous information inputs. UniTRec is a pure textbased recommender modeling user history and candidate texts as inputs. However, incorporating additional side information (e.g., user profile, text topic labels, and dwell time of user behaviors) could further improve the recommendation performance and alleviate the cold start problem. Furthermore, UniTRec only models two-level relations of user behavior history. Nonetheless, incorporating more user behavior information, such as implicit and negative feedback, could further enhance the recommendation performance.

heading: Acknowledgements
text: We appreciate constructive comments from anonymous reviewers. The research described in this paper is partially supported by CUHK under Project No. 3230366.

heading: A Dataset Statistics
text: The detailed statistics of the three text-based recommendation datasets are displayed in Table 3. Note that we use news titles as the text inputs for NewsRec following Qi et al. (2021). NewsRec regards the user clicked and non-clicked news as candidate texts, while QuoteRec and EngageRec regard all potential quotation texts and post texts as candidates. Different from Zeng et al. (2020) that formulates the task as recommending candidate users to given posts based on post content, we formulate the task as recommending candidate posts to given users based on user history.
Algorithm 1 Candidate Ranking Processs Input: discriminative scores Sd = {Sd1 , Sd2 , ..., SdM}, perplexity-based scores Sp = {Sp1 , Sp2 , ..., SpM}. Output: final averaged ranking R̄.
1: Derive the normalized discriminative scores Sdnorm = softmax(Sd). 2: Derive the normalized perplexity-based scores Spnorm = softmax(Sp). 3: Derive the geometric average scores S̄ = log (Sdnorm) + log (Spnorm). 4: Sort the averaged scores S̄ by descending order to derive the final ranking: R̄← Rankdes(S̄). 5: return R̄
B Inference Ranking
Given the user history and M candidate texts, UniTRec first predicts the discriminative ranking scores Sd = {Sd1 , Sd2 , ..., SdM} and perplexitybased ranking scores Sp = {Sp1 , Sp2 , ..., SpM} of the candidates. Algorithm 1 outlines an approach to aggregate the final ranking based on Sd and Sp. Note that the function Rank(S)4 denotes outputting the sorted order of elements in a score list S. There exist other ways to average the ranking of Sd and Sp, which we leave for future work to explore.
4Rank(S) works similarly to scipy.stats.rankdata(). For example in ascending order, Rankasc({0.2, 0.6, 0.7, 0.4}) = scipy.stats.rankdata([0.2, 0.6, 0.7, 0.4]) = [1, 3, 4, 2]

heading: C Qualitative Analysis
text: We show randomly sampled outputs of UniTRec, for instance, demonstrated on the news recommendation and quote recommendation tasks. Table 4 and 5 showcase the qualitative samples.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 5
7 A2. Did you discuss any potential risks of your work? We see no concern about potential risks.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? The Abstract provides the link to our code.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? In the Abstract, a Github repository with documentation is released.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A
C 3 Did you run computational experiments? Section 3
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: There exist other ways to average the ranking of Sd and Sp, which we leave for future work to explore. 4Rank(S) works similarly to scipy. stats. rankdata(). For example in ascending order, Rankasc({0. 2, 0. 6, 0. 7, 0. 4}) = scipy. stats. rankdata([0. 2, 0. 6, 0. 7, 0. 4]) = [1, 3, 4, 2]