Pivotal Role of Language Modeling in Recommender Systems: Enriching Task-specific and Task-agnostic Representation Learning
Year: 2023

Abstract:
Recent studies have proposed unified user modeling frameworks that leverage user behavior data from various applications. Many of them benefit from utilizing users’ behavior sequences as plain texts, representing rich information in any domain or system without losing generality. Hence, a question arises: Can language modeling for user history corpus help improve recommender systems? While its versatile usability has been widely investigated in many domains, its applications to recommender systems still remain underexplored. We show that language modeling applied directly to taskspecific user histories achieves excellent results on diverse recommendation tasks. Also, leveraging additional task-agnostic user histories delivers significant performance benefits. We further demonstrate that our approach can provide promising transfer learning capabilities for a broad spectrum of real-world recommender systems, even on unseen domains and services.

heading: 1 Introduction
text: Recent advances in user modeling have focused on constructing unified user models to be directly adapted to diverse applications. Many of them leverage natural language or plain text data, which enables general-purpose applicability among various domains and systems (Qiu et al., 2021; Gu et al., 2021; Geng et al., 2022; Cui et al., 2022; Hou et al., 2022; Shin et al., 2023). These strategies pave a much more efficient way for service owners to quickly adapt to various task scenarios by tuning one single model, bringing performance improvement across whole systems in parallel.
Based on the recent explosions of sequence prediction models in many domains (Chen et al., 2020; Brown et al., 2020; Ramesh et al., 2021; Chen et al., 2021; Borsos et al., 2022), it is natural to ask
§Both authors contributed equally to this research. Correspondence to: <ky.shin@navercorp.com>.
whether recommender systems can benefit from representation trained by token sequence prediction, i.e., language modeling. Moreover, several works have provided deep insights into why and how language models help address downstream classification tasks (Gururangan et al., 2020; Saunshi et al., 2021; Wei et al., 2021; Karouzos et al., 2021; Krishna et al., 2022).
Some recent studies confirm that continued pretraining of language model on few task-specific data drawn from the target task distribution, or data similar to a target domain can provide significant benefits to solve downstream classification tasks (Gururangan et al., 2020; Lee et al., 2020; Karouzos et al., 2021). Interestingly, Krishna et al. (2022) go further and validate that language models trained from scratch on task-specific or task-agnostic data1—data from other downstream tasks—can rival standard webtext language models. Another line of research provides mathematical explanations of how language model pretraining can improve performances on downstream tasks (Saunshi et al., 2021; Wei et al., 2021). More specifically, Saunshi et al. (2021) reformulate classification tasks as sentence completion tasks, thus demonstrating that linear classification using output features from fixed GPT-2 (Radford et al., 2019), i.e., no finetuning, also guarantees to solve sentence classification tasks.
Motivated by these works, we introduce a new method called LMRec, which jointly trains Language Model and Recommendation task objectives from user behavior histories transformed as plain text format. As illustrated in Figure 1, our approach is conceptually simple but practically effective. We first investigate if the recommender system jointly trained with the language modeling objec-
1Other studies, such as Gururangan et al. (2020) and Krishna et al. (2022), use the term “domain-specific data” or “cross-data” to represent task-irrelevant corpus that is not webtext data. However, we use the term “task-agnostic data” to generally refer to data from other downstream tasks.
1146
tive on task-specific data can enrich the user/item representations, thus providing better generalization even for unseen downstream tasks (Table 4 and 7). We then further verify that additional taskagnostic data can help across the various recommendation tasks, especially when using the taskagnostic data as a user feature (Figure 3). As a result, our methods significantly outperform all the baselines on all tasks, including three public benchmarks and three real-world datasets from different application service domains, and online A/B experiments. Moreover, the pretrained LMRec shows a promising ability to perform downstream transfers flexibly with simple feature-based transfer learning. We also explore several aspects of how the language modeling regime affects the model quality under various conditions, including transfer learning, corpus ablation, and model sizes.
Our major findings are as follows: Jointly training language modeling and recommendation task objectives improve recommender systems. Language modeling on the user history can produce rich user/item representations for diverse applications. These results are consistent with the effect of task-adaptive pretraining in the previous research (Gururangan et al., 2020; Karouzos et al., 2021; Krishna et al., 2022). Furthermore, our approach also boosts the transfer learning capability of the recommendation model. Extensive experimental results show the efficacy of our approach compared to training without language model objectives (Table 4 and 7). Language modeling on task-agnostic data pro-
vides strong results on user representation learning. Consistent with prior work (Gururangan et al., 2020; Krishna et al., 2022), language modeling on additional task-agnostic data alleviates overfitting to a specific history corpus and benefits the learning of robust text representations (Table 4 and 7). We explore how language model pretraining on the diverse task-agnostic data affects transfer learning performances, by comparing with models pretrained on different domain corpora (Figure 3). Virtues of more user data. Recent studies argue that increasing information on user data should be treated as a top priority for improving recommendation performances (Shin et al., 2021; Ardalani et al., 2022). We collect additional user data matched with downstream task users based on user IDs and incorporate them as an additional user feature. Table 7 verifies the data scaling strategy has shown to be beneficial to our models.

heading: 2 Approach

heading: 2.1 Language Models Help with Classification Tasks
text: The empirical and theoretical analyses from the prior work imply that the learned features from the language models trained with appropriate behavior corpus could help predict user and item interactions in recommender systems (Gururangan et al., 2020; Saunshi et al., 2021; Krishna et al., 2022). It is also consistent with the results in Table 1 that language model pretraining with appropriate corpus— related to the downstream task rather than other
corpora such as webtext—leads to performance improvement. It is worth mentioning that linear probe results of LMagnostic can achieve that of LMwebtext performance, although task-agnostic data are in a much smaller-scale than webtext data. This result strongly motivates our research.
Given a sequence of text tokens of user history, u = {h1, ..., hn} and item text tokens i = {g1, ..., gm}, the language model objective L1 is to maximize the following negative log-likelihood:
L1 = − ∑
j=1
logP (hj |hj−k, . . . , hj−1;M), (1)
where k is the context size, and the conditional probability P is modeled using language model M. Then for the downstream tasks, user and item representations zu, zi ∈ Rd are computed as follows:
zu = M(hEOS|u) (2) zi = M(gEOS|i), (3)
where EOS denotes the end of the history token. We use a vector that corresponds to [EOS] token at the last layer as a feature (Neelakantan et al., 2022). The downstream recommendation task loss, L2, of each user-item pair is defined as:
pu,i = 1
1 + exp(− 〈 Wuzu,Wizi 〉 ) , (4)
L2 = −ylogpu,i − (1− y)log(1− pu,i), (5)
where y ∈ {0, 1} is the label denoting whether the user interacted with an item or not. We use〈 ·, · 〉
for the dot product. The weight matrices Wu,Wi ∈ Rd×d linearly transform the user and item representations, respectively.
Several works have highlighted that jointly optimizing language modeling during finetuning benefits avoiding catastrophic forgetting (Chronopoulou
et al., 2019; Karouzos et al., 2021). Inspired by the merits of this strategy, we adopt a joint optimization:
L = L1 + λL2, (6)
where L is the final joint training loss. We impose weight λ on L2 loss to prevent the overfitting of recommendation tasks. As illustrated in Figure 1, a model that optimizes Equation (6) is denoted as “LMRec”. The model trained without the language model objective (L1) is “LMRec-lm”. The performance comparison between the pretrain-thenfinetune model and our approach are presented in Table 2.

heading: 2.2 Enriching Task-specific and Task-agnostic Representation
text: Leveraging task-agnostic data. Optimizing performances solely on task-specific data would restrict the potential of a unified framework. Therefore, a recent trend in user modeling research is to leverage large quantities of pretraining (or additional) data that are not directly related to the target task (Hou et al., 2022; Shin et al., 2023).
To this end, we introduce “LMRec+agnostic”, which utilizes additional task-agnostic data for language model objectives. This approach increases the generality by mitigating overfitting to a specific history corpus. Consequently, it boosts the learning of robust text representations, thus making LMRec+agnostic universal across various tasks. As a result, additional task-agnostic data further boost the performance of our default LMRec model, which already produces state-of-the-art results in all tasks and metrics. Transfer learning. There are several difficulties in applying a unified model to real-world applications: (1) target applications are commonly unknown or undefined during pretraining, (2) user ID cannot be matched across different companies, (3) large-scale recommender systems usually contain millions of
users and items, thus it is computationally expensive to finetune the large models to numerous applications directly. To overcome these obstacles, we propose a simple transfer learning framework that can easily and quickly adapt the model to diverse applications. As visualized in Figure 2, we simply plug the target task-specific inputs into the pretrained LMRec and compute user/item embeddings to perform a linear probe. We add superscript to the model as “LMRecTL” for the transfer learning framework. The LMRecTL model jointly pretrains multiple tasks, excluding the target downstream task. The final loss to pretrain is as follows:
L = ∑
t∈Ts,Ta Lt1 + λ
∑ t∈Ts Lt2, (7)
where Ts denotes a set of pretraining recommendation tasks, and Ta for additional task-agnostic data. Note that linear layers of pretraining and featurebased transfer learning are separate modules. Task-agnostic user features. Leveraging crossdomain data of users for improving recommender systems has been widely discussed (Man et al., 2017; Yuan et al., 2019; Zhu et al., 2022; Shin et al., 2023). These strategies assume that the underlying user preference in the source and the target domains can be related, and thus learning a common user semantic enhances the recommender system. Hence, we utilize additional task-agnostic data, obtained from application services whose user IDs are shared in a company level, as a user feature for target downstream tasks. The difference between task-specific and task-agnostic data in Figure 2 is
only which user features are used for transfer learning. For example, if the target downstream task is ECOMM, models are first pretrained with OBS and OTA, and then use task-specific data of ECOMM to produce task-specific user features. For leveraging the task-agnostic user feature, the pretrained model extracts user features from task-agnostic data, such as Search and News. Components other than user features, such as the pretrained model, downstream architecture (linear layer), and ground truth interacted items of users, are all the same. We can verify that the transfer learning approach benefits from leveraging additional task-agnostic data as user features, especially when it is recommending for new users (Table 7, 8 and Figure 3).
Appendix A describes the training details of our methods.

heading: 3 Experiments

heading: 3.1 Datasets
text: To make user behavioral corpora, we consider the behavior description as items, i.e., search queries of search logs, news titles of online news click logs, and content titles of social media click logs. As illustrated in Figure 1, we concatenate the behavior logs using the “→” token. This simple form of a prompt template can have behavior sequences that are very long. Furthermore, separating corpus among multiple services provides flexible transfer learning capabilities by enabling easy proliferation of behaviors and filtering out redundant representation to target applications. We use Byte-level BPE (Wang et al., 2020) to tokenize the textual description of each item in the behavior logs. Task-specific datasets. We use three in-house datasets in order to assess our approach on various applications and add three public datasets that are predominantly evaluated in recommendation communities. The in-house datasets are built from services of an online booking service (OBS), an online travel agency (OTA), and e-commerce platmform (ECOMM). For public datasets, we select two categories “Industrial and Scientific” (Scientific) and “Prime Pantry” (Pantry) from Amazon review datatsets (Ni et al., 2019) which are two completely different service domains. We further collect “Online Retail”2 dataset from an online retail platform to validate the cross-system transferability of our models.
2https://www.kaggle.com/carrie1/ecommerce-data
Task-agnostic datasets. We construct sufficiently large-scale task-agnostic behavioral corpora for inhouse datasets. These datasets are collected over two years and from four behavioral corpora, a search engine (Search), e-commerce (E-comm.), social media platform (SNS), and news website (News). As a result, the in-house dataset contains 10 million users and 94 million user history logs, and 12 billion BBPE tokens. Following the experimental setup of UniSRec (Hou et al., 2022) for public benchmarks, we select the five categories “Grocery and Gourmet Food”, “Home and Kitchen”, “CDs and Vinyl”, “Kindle Store”, and “Movies and TV” from Amazon review datasets. These datasets are used as pretraining datasets for pretrain-then-transfer models such as UserBERT (Wu et al., 2022), UniSRec (Hou et al., 2022), M6-Rec (Cui et al., 2022), and CLUE (Shin et al., 2023), while used as additional task-agnostic data for LMRec+agnostic model.
The details of datasets are outlined in Table 3.

heading: 3.2 Experimental Settings
text: In-house downstream tasks. The datasets consist of positive pairs (u, i) which means a user u interacted with an item i. The negative pairs are generated through random sampling during training. Evaluation metrics are Recall@k and top-k Normalized Discounted Cumulative Gain (NDCG@k), which are evaluated from ground truth items mixed with 100 randomly sampled negative items. To test the generalizability of user representations, we randomly split the user pool among the training (80%), validation (10%), and test sets (10%). Public downstream tasks. We filter out users and items with fewer than 5 interactions. Each user’s interaction history was listed chronologically. We use item descriptions such as titles, categories, and brands for item information. The maximum token length of item text is set to 512. Following previous works (Kang and McAuley, 2018; Sun et al., 2019; Hou et al., 2022), we adopt the leave-
one-out strategy, i.e., next item recommendation task. The last item, second last item, and other items are used as the test, validation, and training data respectively. The Recall@k and NDCG@k are computed by ranking the ground-truth item among all the other items.

heading: 3.3 Baselines
text: We compare our models against six strong baselines. Behavior Sequence Transformer (BST) (Chen et al., 2019) and LightGCN (He et al., 2020) are primarily used baselines in various tasks and domains. To reflect the recent trend of user modeling research, which adopts pretrainthen-transfer strategies, we employ several models from these lines of work. UserBERT (Wu et al., 2022) and UniSRec (Hou et al., 2022) pretrain self-supervision objectives with language embeddings and then finetune the model to downstream tasks. The most comparable unified user models to our methods are M6-Rec (Cui et al., 2022) and CLUE (Shin et al., 2023). These two methods treat user history as plain text and construct a universal encoder that can be adapted to any domain and task. Note that all the pretrain-then-transfer models, excluding CLUE, utilize webtext language models. Please see Appendix B for more details of baselines.

heading: 4 Results

heading: 4.1 Performance on Various Tasks
text: Table 4 presents the efficacy of our LMRec against baselines. Across the six datasets, LMRec trained only with the task-specific data achieves state-ofthe-art performances compared to all the baselines, even though some methods utilize additional task-agnostic data. For the in-house datasets, LMRec surpasses best performing baseline models by over 1.6 ∼ 3.2% in Recall@10. In the public datasets, LMRec shows around 5% average improvements compared to baselines. Since other
pretrain-then-transfer models leverage additional data, we introduce LMRec+agnostic, a more robust representation learning method using additional corpus for language modeling. LMRec+agnostic remarkably outperforms the other models in all tasks by a significant margin (see improvement in Table 4). We further conduct an ablation study on combining task-specific and task-agnostic corpus when the computation resources are limited. Table 5 presents the results. LMRec+agnostic (0% : 100%), i.e., language modeling on task-agnostic data only, outperforms LMRec-lm in Table 4, but
shows the worst performance in Table 5. Increasing the ratio of used task-specific data delivers performance benefits to some point (70%). However, leveraging task-specific data solely finally decreases the performance.
Previous research provides a theoretical analysis of why language model pretraining guarantees effective representation learning for downstream tasks (Saunshi et al., 2021; Wei et al., 2021). The additional analysis in Appendix C may support these results.

heading: 4.2 Linear Probe
text: We show the effectiveness of the language model pretraining then feature-based transfer strategy (Figure 2) across all tasks. Our approach empirically demonstrates the flexible generalizability of the pretrained features. Note that all the baselines, excluding CLUE, are pretrain-then-finetune methods, and the downstream computational cost (Table 6) is much more expensive than the linear probe.
As shown in Table 7, the linear probe result of LMRecTL-lm that are trained only on recommendation tasks shows worst transfer learning performances. Unsurprisingly, a model trained without language modeling cannot guarantee generalizability to other language corpora. It is worth mentioning that LMRecTL, which jointly trains language model and recommendation tasks objectives, shows decent transfer learning capability for downstream tasks. This result provides that incorporating language model pretraining with recommender system profits strong adaptability and generality compared to the recommendation model, even on the linear
probe, i.e., not trained on downstream tasks directly. As previous research (Gururangan et al., 2020; Krishna et al., 2022) confirmed, it is reasonable to believe that leveraging large quantities of additional data for language model pretraining is strictly more powerful than using small task-specific data. LMRecTL+agnostic shows enhanced transferability on linear probe. Comparing results among Table 4, 6, and 7, we can see that LMRecTL+agnostic outperforms other baselines with much fast and easy adaptation.

heading: 4.3 Virtues of More User Data
text: A line of research that studies scaling law in recommender systems argues that parameter growth will not always offer performance improvement and has low return-on-investment (ROI) in resource efficiencies (Ardalani et al., 2022; Shin et al., 2023). Hence, the data scaling scheme should be treated as a top priority for improving model performances. To verify the efficacy of the data scaling approach, we evaluate our model on downstream tasks by using task-agnostic data as user feature. Results are presented in Table 7-(Taskagnostic feature/Combine). We simply concatenate task-specific and task-agnostic data to use as inputs
for the Combine setup. Most baselines are not adequately reflecting the possibility of using additional user features due to their pretraining methods, but LMRecTL+agnostic properly considers the potential of using more user data. It is an enormous benefit to the models seeing that LMRecTL+agnostic (Combine) shows outstanding performance by combining all the user data. Interestingly, LMRecTL, which is trained without task-agnostic data, also achieves state-of-the-art or comparable performances to the baseline models. This result highlights the efficacy of our approach.
We conducted an online A/B experiment for a product collection recommendation task (see Appendix D for more details) on our in-house ecommerce platform for two weeks in August 2022. Table 8 shows the consistent superiority of our method online. For user groups ‘new’, the user representation by LMRecTL+agnostic significantly improves CTR and GMV compared to GNN (Jeong et al., 2020). We conjecture that it may benefit from additional user data from other services, thus contributing to users with no recorded behavior.

heading: 4.4 Effect of Pretraining Behavior Corpora for Transfer Learning
text: We perform ablation studies on the relations between pretraining corpora and using task-agnostic data as user features. As shown in Figure 3, the model pretrained with the specific corpus provides general and robust representations of that corpus even on unseen tasks. Interestingly, tailoring a language model to diverse corpora may bridge the gap between pretraining and taskagnostic corpus domains. For example, even though LMRecTL+search leverages only Search corpus for language model pretraining, it consistently outperforms LMRecTL-lm and LMRec TL on all the
downstream tasks with other task-agnostic features. As it can be seen in Figure 3, the performance of LMRecTL-lm in OBS task is relatively low compared to other tasks. It is due to the strong contribution of task-agnostic features (Table 7 and Figure 4) for the OBS task. In other words, when the taskagnostic features are well-transferable to the target downstream tasks, the performance differences between not pretrained (LMRecTL-lm) and the rest can be substantial.

heading: 4.5 Effect of Model Size
text: Many recent reports in NLP and computer vision have empirically demonstrated the existence of a scaling law, where performance scales strongly with model capacity (Brown et al., 2020; Kaplan et al., 2020; Zhai et al., 2021; Bahri et al., 2021). Recently, Shin et al. (2023) found the power-law learning curve as a function of model size in recommender systems. Figure 4 shows that scaling up the model leads to a strict performance improvement on the downstream tasks, consistent with the results in the prior works. However, we can also
find that models’ performances have an upper limit. It is in harmony with the trend in Ardalani et al. (2022) that the recommendation performance follows a power law plus a constant relationship to the model size, which is an irreducible error on our side.
Note that the performances of LMRecTL-lm do not vary according to the model sizes. We conjecture that the model trained without language modeling has no benefits from high model complexity, as its learning capacity is naturally limited.

heading: 5 Related Work
text: Any model that trains a text-based user model to adapt to unseen domains/systems can be viewed as prior work of our research. This line of work has been recently explored since learning text representation has been rapidly developed in the decade. In this context, Qiu et al. (2021) and Gu et al. (2021) are the earliest work we are aware of. They train the model through critical word matching in user logs and then finetune models to the downstream tasks. First, the word (item) embeddings are precomputed
using pretrained language models (PLMs). The sequence of item embeddings is then passed to the encoder to produce user representations. Recently, some researchers propose to use behavior history as plain text data (Geng et al., 2022; Cui et al., 2022; Hou et al., 2022; Shin et al., 2023). Hou et al. (2022) and Shin et al. (2023) introduce a contrastive learning framework on multiple service domains, and perform transfer learning across various downstream tasks. Another line of work (Geng et al., 2022; Cui et al., 2022) tries to construct personalized prompts for building versatile framework, i.e., “Here is the history of {gender} {age}: {history from all services}, The user is now recommended a {item}”. This approach profits from the methods that utilize language models such as GPT-2 (Radford et al., 2019), T5 (Raffel et al., 2020), and M6 (Lin et al., 2021). Their PLM-based approach can be generalized to various applications, with the ability to perform zero-shot learning. Shin et al. (2023) is the only work that trained the whole encoder from scratch rather than using PLMs. We refer readers to Liu et al. (2023) and Yuan et al. (2023) for an overview of this line of work.
A related idea to our work is the training language model on task-specific or task-agnostic corpora. It has been shown to be beneficial in a variety of works (Chronopoulou et al., 2019; Gururangan et al., 2020; Lee et al., 2020; Karouzos et al., 2021; Krishna et al., 2022). Gururangan et al. (2020) continue pretraining of LM on task-specific data and show it can improve the downstream performances of standard webtext language models. Krishna et al. (2022) point out that the effect of pretraining on standard webtext data may have been overestimated. They show that models trained only on task-specific data comparably perform to existing webtext language models. On the one hand, a line of research jointly trains language models on taskspecific data during finetuning to avoid catastrophic forgetting (Chronopoulou et al., 2019; Karouzos et al., 2021). Some of the works above also investigate if the models pretrained on task-agnostic data can be effective for downstream tasks. Gururangan et al. (2020) and Lee et al. (2020) show domain-adaptive pretraining further improves the performance of pretrained language models. Recently, Krishna et al. (2022) have observed that pretraining on task-agnostic data can provide a significant advantage compared to standard webtext data. These findings give huge insight into our
research. Note that our work aims at extending the potential of language modeling that has been successfully used for diverse applications to recommender systems.

heading: 6 Conclusion
text: Recent works have built text-based user models and demonstrated that the rich nature of text information in any domain or system could be a valuable foundation for user modeling. Our primary contribution is jointly optimizing the language modeling and recommendation task objectives and successfully tackling a broad spectrum of diverse recommendation tasks, including transfer learning for unseen domains and systems. Overall, our analysis sheds remarkable insights on user representation learning through user behavioral corpora.
Considerations and Limitations
LMRec is trained on user behavior text data that are collected from diverse service applications. These datasets are preprocessed to users’ behavior sequences as detailed in Figure 1 and Section 3.1. However, in order to improve the quality of user representations, choosing the item information differently for each application may improve the effectiveness. As such, we can consider domain-specific information for each service rather than using general item information. For example, we may leverage additional domain-specific information such as news topics or categories, names of the press agency, and keywords for the news content rather than using only news titles for the News dataset. This issue is a promising extension for practitioners to successfully apply LMRec to real-world applications.
The types of task-agnostic data will largely affect the performance gains of LMRec+agnostic and LMRecTL+agnostic. We fully utilize four types of taskagnostic data, i.e., Search, E-comm., SNS, and News, and achieve state-of-the-art results. However, this paper does not thoroughly explore their optimized combination or mixing ratio of the corpus due to the heavy computational costs, which most large LM studies suffer from. While prior work shows how the pretraining corpus sources and their combination affect diverse downstream tasks (Raffel et al., 2020; Gururangan et al., 2020; Lee et al., 2020; Krishna et al., 2022; Shin et al., 2022), there still remain limitations in finding the
generic relation between downstream performance and corpus properties; measuring the effect of the pretraining corpus on the downstream task is still underexplored. We point out that more careful study is left for future research.
Regarding reproducibility, it is difficult to open our in-house data due to legal issues caused by privacy and user agreement. Therefore, we tried our best to validate the efficacy of our LMRec with the experiments on benchmark datasets in addition to in-house data.

heading: Acknowledgements
text: All authors thank NAVER Smart Machine Learning (NSML) platform team (Sung et al., 2017; Kim et al., 2018) for their critical work on the software and hardware infrastructure on which all the experiments were performed.

heading: A Training Details
text: We utilize separate data loaders to deal with different batch sizes between language modeling and recommendation tasks. Furthermore, the early stopping strategy is employed based on the validation loss of the recommendation task and patience of 100 steps. We use the AdamW (Loshchilov and Hutter, 2019) with β1 = 0.9, β2 = 0.98, ϵ = 10−6, and Zero Redundancy Optimizer (Rajbhandari et al., 2020). We update the model using linear warm-up of the learning rate over the first 1% steps, followed by cosine decay (Loshchilov and Hutter, 2017) to decrease the learning rate to 10% of its initial value. The cosine decay is also applied to the λ value. We leverage the automatic mixedprecision (Micikevicius et al., 2018) package in Pytorch (Paszke et al., 2019) to reduce training time and GPU memory usage. Gradient norm clipping (Pascanu et al., 2013) is used with the max norm set to 0.1 to stabilize training. Unless otherwise specified, all results are reported by 125M transformer decoder (Vaswani et al., 2017). All models use a vocabulary size of 50, 258 and a max sequence length of 2, 048. The hyperparameter values for different sizes of LMRec is presented in Table 9. All the results are averaged over the 20 runs.

heading: B Details of Comparison Models
text: Behavior Sequence Transformer (BST) (Chen et al., 2019) embeds user history logs as lowdimensional vectors and passes them to the transformer layers to model underlying user preferences.
LightGCN (He et al., 2020) leverages Graph Convolution Network (Kipf and Welling, 2017) for enhancing collaborative filtering. It linearly propagates user and item embeddings of a bipartite interaction graph. The final embedding is computed by the sum of the embeddings propagated at each layer.
UserBERT (Lu et al., 2020) incorporates two self-supervision tasks for pretraining. These pretext tasks effectively capture the relations between user behaviors and inherent user interests. It finally finetuned models on target tasks.
UniSRec (Hou et al., 2022) proposes to combine parametric whitening and MoE adaptor
for learning personalized representation. UniSRec pretrains user history by sequence-to-sequence contrastive learning and then finetunes the model to downstream tasks.
M6Rec (Cui et al., 2022) employs prompt tuning of pretrained language models for building a unified framework. M6Rec fully utilizes text inputs to generalize to any domains/systems and has the ability to perform zero-shot learning. Since they did not release pretrained M6 (Lin et al., 2021), we used Huggingface RoBERTa (Liu et al., 2019) to implement it.3
CLUE (Shin et al., 2023) presents a plain text-based contrastive learning framework, considering heterogeneous services or applications as a modality and users as a common semantic. It then performs feature-based transfer learning for downstream tasks.

heading: C Effect of Language Modeling on Local Curvature
text: One of the most well-known criteria influencing neural network generalization is observing Hessian eigenvalues with respect to parameters. Since the Hessian is often treated as local curvature, the eigenvalues of Hessian determine the smoothness of loss landscapes. Many researchers have argued that the flat loss landscape leads to better generalization (Li et al., 2018; Foret et al., 2021; Chen et al., 2022; Park and Kim, 2022). We calculate and gather top-5 Hessian eigenvalues by PyHessian (Yao et al., 2020), and resulting max eigenvalues are visualized using kernel density estimation in Scikit-learn (Pedregosa et al., 2011). Results are presented in Figure 5. The language model
3https://huggingface.co/transformers/model_doc/roberta
only (LM) on the OBS task produces many negative eigenvalues, which means the loss landscape is non-convex and, thus, challenging to optimize. This result is natural since the loss of the target task computed without adaptation of models cannot bring good properties. On the other hand, eigenvalues of models (LMRec-lm and LMRec) trained with target objectives flocked together on the positive side. The magnitude of the eigenspectra of LMRecmodel is smaller than that of LMRec-lm model. It means that learning two objectives simultaneously improves the robustness and generality of model performance on downstream tasks.

heading: D Online A/B Experiment
text: We run A/B experiments on product collection recommendation tasks using LMRecTL+agnostic user feature to verify the practical usage of our method online. The product collection is a collection of products allotted by merchandisers with a particular category such as “Plush robe coats for men”, “Winter sale special offer”, and “Best backpacks for high school students”. This task is to recommend the product collection banner, linked to a page displaying a list of products.
We pretrain LMRecTL+agnostic with OBS, OTA, and ECOMM and then transfer to the product collection recommendation (target task). The mean pooled task-specific and task-agnostic user features are used as the final user features. During the 14 days of online experimentation, we measured two important metrics for the online recommender system, CTR and GMV, to track user satisfaction with the platform. CTR represents the click/view rate of recommendation, and GMV is the total value of sold products through recommendation. All models take the same amount of user traffic.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Considerations and Limitations section
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: It is described in the last paragraph of the Introduction section.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 describes the scientific artifacts used in this paper.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3 descrbies it. We used Amazon review dataset proposed in https://nijianmo.github.io/amazon/.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The license guidelines are already described at (https://s3.amazonaws.com/amazon-reviews-pds/readme.html). And we strictly followed the guides (purpose of academic research).
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The scientific artifacts used in this work are consistent with their intended use.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No offensive content exists in our datasets; even if it exists, offensive content can not harm our research since the final outputs are recommendation results. It is already anonymized, and there is no identifying information like names, phone, credit card numbers, addresses, user names, etc.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3.1 outlines our datasets.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3 and Table 3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? The computation costs of our models are shown in Table 6.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? We described it in Table 6

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix A describes the experimental details of our approach.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? At the end of Appendix A.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We point out that more careful study is left for future research. Regarding reproducibility, it is difficult to open our in-house data due to legal issues caused by privacy and user agreement. Therefore, we tried our best to validate the efficacy of our LMRec with the experiments on benchmark datasets in addition to in-house data