UserBERT: Self-supervised User Representation Learning
Authors: Tianyu Li
Ali Cevahir
Derek Cho
Hao Gong
DuyKhuong Nguyen
Bjorn Stenger
Venue:  (2021)
Keywords: user representations
representation learning
self-supervised learning
pretraining
transfer learning

Abstract: This paper extends the BERT model to user data for pretraining user representations in a self-supervised way. By viewing actions (e.g., purchases and clicks) in behavior sequences (i.e., usage history) in an analogous way to words in sentences, we propose methods for the tokenization, the generation of input representation vectors and a novel pretext task to enable the pretraining model to learn from its own input, omitting the burden of collecting additional data. Further, our model adopts a unified structure to simultaneously learn from long-term and short-term user behavior as well as user profiles. Extensive experiments demonstrate that the learned representations  result in significant improvements when transferred to three different real-world tasks, particularly in comparison with task-specific modeling and representations obtained from multi-task learning. 

Under review as a conference paper at ICLR 2021
USERBERT: SELF-SUPERVISED USER REPRESENTA- TION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
This paper extends the BERT model to user data for pretraining user representa- tions sulf--uper way. By viewing actions (e.g.. purchases and clicks) behavior sequences (i.e., usage history) in an analogous way to words in sentences, we propose methods for the tokenization, the generation of input representation vectors and novel pretext task to enable the pretraining model to learn from its own input, omitting the burden of collecting additional data. Further, our model adopts unified structure to simultaneously learn from long-term and short-term user behavior as well as user profiles. Extensive experiments demonstrate that the learned representations result in significant improvements when transferred to three different real- world tasks, particularly in comparison with task-specific modeling and representations obtained from multi-task learning.
INTRODUCTION
The choice of data representations, i.e., how to create meaningful features, imposes tremendous impact on the performance of machine learning applications (Bengio et al.) 2013). Therefore, data processing and feature engineering have been the decisive steps in developing machine learning models. To extend the applicability of the models, recent research on representation learning aims to discover the underlying explanatory factors hidden in raw data. With rapid advances in this direction, we have witnessed many breakthroughs in areas computer vision (CV) (Doersch) et al. 2013 Sharif Razavian al. 2014), SSiooSSera et al.) (013) and natural language processing (NLP) (Mikolov et al., 2013 Pennington et al., 2014 Lin et al. 2017).
Similarly, for building user-oriented industrial applications like next purchase prediction and rec- ommendation, much effort has been spent on understanding business models and user behavior for creating useful features (Richardson et al.) 2007 Covington al., 2016). This is time-consuming and application-specinn process. Also, it is challenging to reuse these features or share gained knowledge between different services and applications.
To solve the issues of isolated feature engineering and task-oriented pipeline design, the pretraining- transfer learning paradigm has been explored. For example, multi-task learning (MTL.) has shown promising results (Ni et al. 2018). Nevertheless, MTL has its intrinsic challenges, e.g., deciding which tasks to learn jointly (Standley et al.) 2019), or how weigh tasks (Kendall et al. 2018). to achieve optimal performance. More importantly, the learning still hinges on large amounts of well-annotated user labels.
Inspired by the BERT model and its variations (Devlin et al. 2019 Lan et al. 2020), this paper explores the feasibility of understanding users in a similar way to how language is understood. We think it is conceptually intuitive to make such an analogy since understanding language and users share similar goal, iee,, understanding conveyed message, but with different mediums. The former models what is said (sentences) while the latter learns from what is done (bchavior). The syntax and semantics of a sentence are comparable with the behavioral patterns and the characteristics a Hence, hypothesize the learning procedure propose to build upon BERT for pretraining user representations on unlabeled behavior data.
be consistent in methodology well, and
Our proposal, UserBERT, simultancously leams from three categories of user data, i.e., long- long-term and short-term behavior as well as user profiles, via a unified architecture. particular, differ- ent action types (e.g., page views, clicks and purchases) and attributes (e.g., shop and item genre)

---

Under review as a conference paper at ICLR 2021
chosen represent long-term and short-term user behavior. For these two behavior types, first present distinct strategies to discretize them into a sequence of behavioral words. Instead of modeling single user actions sequentially, the applied discretization leads to better generalization. The token representation of these behavioral words is computed by the concatenation and mean calculation of the word embeddings of the attribute IDs in each action, and this is followed by the summation of token, position and segment embeddings. These representation vectors are finally aligned with the word embeddings of user categorical profiles as the input to Use.BERT. With this input, we design novel pretext task, masked multi-label classification, and the UserBERT model pretrained via optimizing the multi-label classifications of the multiple attributes in the masked behavioral words.
Despite the parallels between user behavior and sentences, there are substantial differences and challenges in designing the learning procedure in coherent way. Our model is able to deal with heterogeneous user behavior data, and achieve generalization via effective tokenization and the pre- training task. While there is prior work applying BERT to task-specific user modeling (Sun et al. 2019b), this paper is built upon the assumption that behavioral patterns can be understood like the structure of a language. The UserBERT model explores integrating various types of user data in unified architecture and learning generic representations with self-supervised signals. In our exper- iments, the pretrained model is fine-tuned three different real-world tasks, and the results show that URTT outperforms task-specific modeling and multi-task learning based pretraining.
Our contributions are summarized follows:
We propose UserBERT, self-supervised learning model, pretrain user representations via analogizing actions in a user behavior sequence to words in sentence. It eliminates the needs of previous approaches for collecting additional user annotated labels.
We design the discretization of user raw data sequences, the generation of the input repre- sentation and novel pretext task for pretraining.
UserBERT adopts a unified model architecture to enable the simultaneous learning from heterogeneous data including long, short-term behavior as well as demographics.
We demonstrate the empirical power of UserBERT with extensive experiments. Our model is compared with task-specific models without pretraining and multi-task learning based pretraining models, and achieves performance gains on three real-world applications.
RELATED WORK
2.1 PRETRAINING AND TRANSFER LEARNING
Recent studies have demonstrated that pretraining on large, auxiliary datasets followed by fine- tniing on target tasks is a promising paradigm for boosting performance (Oquab et al., 2014 Don- ahue et al., 2014 Hendrycks et al. 2019 Ghadiyaram et al.) 2019). Multi-task learning has been one the commonly adopted approaches for pretraining due to its ability to improve generaliza- tion (Zhang & Yang 2017: Ruder 2017). It is shown that the pretrained MTL models can boost performance even when transferred to unseen tasks (Liu et al. 2015 Ni et al. 2018). Despite its success, MTL still has many challenges, such negative transfer and the learning adjustment be- tween different tasks (Guo et al. 2018)). Also, MTL requires large amounts of well-annotated labels produce satisfying outputs. There trained models to given target task, i.e., feature- based in which the pretrained weights are frozen, and directly fine-tuning the pretrained model (Peters al. 2019). We fine-tune pretrained models in our experiments.
common forms of adaptation when transferring the pre-
2.2 SELF-SUPERVISED LEARNING
Deep learning models can already compete with humans on challenging tasks like semantic seg- mentation in the CV area (He et al. 2015) well few language understanding tasks (Liu al.) 2019). However, such success relies on adequate amounts of quality training data, which can be extremely expensive even impossible obtain (Kolesnikov et al. 2019). As result, lot of
2

---

Under review as a conference paper at ICLR 2021
research efforts aim to liberate learning from the heavy dependency on supervised signals. Self- supervised learning (SSL), a subelass of unsupervised learning, has been drawing more attention since the recent advances in the NLP field. Instead of using supervision signals, SSL only requires unlabeled data and trains models via formulating pretext learning task. There are two main types of pretext tasks: context-based (Pathak al. [016]] Noroozi & Favaro 2016 (Sermanet al. 2018
Wu et al. 2019) and contrastive-basee (Hjelm et al. Chen et al. 2020
2.3
USER MODELING
To build user-oriented machine learning applications, the key challenge is finding an expressive representation of user data so that the followed modeling can effectively extract useful information to produce good performance. For that reason, much effort has been going towards data preprocessing and transformations, such as converting user categorical attributes to embeddings and aggregating user activities like total number of visits, clicks or amount of money spent over certain time interval or a particular product genre (Richardson et al., 2007 Zhu al. 2010). Deep learning models have successfully mitigated the dependency on human efforts due to its ability to capture underlying representations in raw data (Cheng et al. 2016 Covington et al.) 2016 [Zhou et al.] 2018). However, these models need massive supervision signals for training, and they are mostly designed for specific tasks like recommendation (Pei et al. 2019) and click-through rate prediction (Zhou et al. 2019).
Despite the success of these deep learning models, they fail to generate promising results for real- world industrial tasks with limited labeled data. To deal with this issue, the methodology that pre- training universal user representations on massive user data, and then fine-tuning them for down- stream tasks is explored. The goal is to learn a universal and effective representation for each user which be transferred to new tasks (Ni et al. 2018). However, MTL-based pretraining still quires the collection of user labels. Also, it is limited by inherent shortcomings to achieve optimal results (Kendall et al.. 2018 Guo al. 2018). It is highly desirable for user applications to have a learning paradigm that does not require large amounts of manually annotated data. Our work is inspired by the BERT model which pretrains representations for language understanding. We aim to pretrain universal user representations by analogizing actions in user behavior sequence to words in sentence, and apply transfer learning to downstream tasks, especially those with few labeled data, for boosting performance.
3 THE PROPOSED APPROACH
In this section, first review the BERT model in brief, and then elaborate how extend it to user data including behavior sequences and demographic profiles.
3.1 THE BERT MODEL
BERT is a language representation model that pretrains deep bidirectional representations by jointly conditioning on both left and right contexts in all encoding layers (Devlin et al., 2019). The input of the BERT model is a sequence of tokens that can represent both a single text sentence and a pair of sentences. These discrete tokens consist of words and a set of special tokens: separation tokens (SEP), classifier tokens (CLS) and tokens for masking values (MASK). For token in the sequence, its input representation sum of word embedding, the embeddings for encoding position and segment.
The BERT model pretrained with two tasks, masked language modeling (MLM) and next sentence prediction. In MLM, the input tokens are randomly masked and the BERT model is trained to reconstruct these masked tokens. detail, linear layer learned to map the final output features of the masked tokens to a distribution over the vocabulary and the model is trained with a cross- entropy loss. next sentence prediction, the inputs are two sampled sentences with a separator token SEP between them. The model learns to predict whether the second sentence is the successor of the first. A linear layer connecting the final output representations of the CLS token is trained to minimize cross-entropy loss on binary labels. Many recent research works focus extending the BERT model to areas beyond NLP, and successfully achieved state-of-thee off-the-art arrt results (Sun et al.
2019a Lu al. 2019 Su et al. 2020 Qi et al. 2020).
3

---

Under review as a conference paper at ICLR 2021
long-term
short-term
profiles
input sequence
shop
event
age gender place
attribute
expense channel
genie
shop
profile embedding
action
purchase
point usage
diick
page view
embedding
position embadding
within 24 hours
mins
mins
token embedding
Figure 1: Tokenization and input representation generation of long-term, short-term user be- havior and profile data. form behavioral words, we discretize long-term behavior into 24-hour intervals and segment short-term sequences when there is time interval larger than 30 minutes be- tween two actions. The word embeddings of the attribute IDs in each action are first concatenated. Then, the token representation is constructed by the mean of all action embeddings. The representa- tion in the sequence is summation of token embeddings and the embeddings for encoding position and segment.
3.2 USERBERT
Tokenization of user behavior sequences. Our goal is to learn generic user representations that based their preferences and recent interests. We decide not sequentially model single actions in long-term and short-term user data. While such modeling is suitable for certain tasks, it is susceptible to overfitting when learning generic user representations. Instead, we learn from a sequence of clustered user actions, in which a cluster represents a routine or spontaneous interest. Customers often make online purchases with specific intentions, e-g., shopping for a shirt, cartoon books or a gift for Mother's Day. Also, many customers have preferences for particular stores and sales are heavily impacted by seasonality. These continuous or related actions form 'word' in behavior sequence. Similarly, we consider the same regarding short-term user behavior. Users commonly browse web content, moving between pages on an e- commerce site. During this time period, in order to capture the user's interest, aim to estimate the theme or product genre rather than the specific order of individual actions.
characterize
Therefore, first need to segment raw action data into sequence of "behavioral words' for each user, analogous to words in a sentence. In detail, we adopt different approaches for long- long-term and short-term data. Data representing long-standing user preferences is discretized into 24-hour intervals from 4 AM of one day to AM the next day. Short-term data is discretized if there is time interval larger than 30 minutes between two actions, similar to the processing steps in Grbovic & Cheng 2018).
Input representations. In order enable bidirectional representation learning, we transform the behavioral word sequence into a sequence of input embeddings. We first introduce the concept of action type and attribute in user actions: The action type indicates what user does, e.g., making purchase or obtaining points for using a service, while the attribute of an action includes the shop name, the item genre and price range, etc, as shown Figure [ We choose different action types and attributes in our dataset to represent long-term and short-term user behavior, and propose separate tokenization strategies for them since we expect to extract inherent user preferences from regular routines over longer time periods, and short-term interests from recent, temporary interactions. In combination with demographic data, we consider the learned representations comprchensive and expressive.
To generate input representations, all attribute IDs are first mapped to fixed-length word embeddings via look-up tables. Then, the attribute embeddings of each action concatenated. Subsequently, the token representation is constructed by the mean embedding vector obtained by summing the token embeddings and the embeddings for encoding
action embeddings. Finally, the input
4

---

Under review as a conference paper at ICLR 2021
h
h2
h
h2
UserBERT
E
E
E,
E2
E,
E
zero-out
zero-out
long-term
short-term
user profiles
Figure 2: Pretraining of BERT. The representation vectors in input sequences are randomly masked (zeroed-out), and then the masked input is passed through UserBERT. The model is trained to reconstruct the attributes in these masked 'words'. For each atribute, an oulput layer is connected to the hidden representations at the masked positions, and it is learned via minimizing the prediction errors of multi-label classifications.
position and segment. Long and short-term user data share the same processing steps above, but each has their own definitions for token position. While the position of token in long-ten meng-temm sequences is the number of days counted from the starting point the collected training data, for short-term data it is the number of hours. The segment embedding is used to differentiate the given types of user behavior. In order to incorporate non-temporal user profile data to our modeling, we consider categorical attributes like gender tokens in the user input sequence. For the continuous-valus attributes like age, we segment them by heuristics and convert them to categorical attributes. After mapping attributes word embedding vectors, these are summed the segment embedding. Note that there is no position embedding for profile embeddings since no order information needs to be captured for these user attributes. The input sequence for each user is formed by aligning the generated representation vectors of user behavior as well as the embeddings of user profiles, see Figure I for illustration.
The generated input sequences allow us to make minimal changes to the BERT architecture and follow the practice Devlin et al. (2019). We then pretrain our model learn bidirectional representations. While the MLM task seems to naturally apply to modeling, recon- structing the masked 'behavioral words' requires modification since these words contain an assem- bly of user actions rather than individual words used in the original BERT model. We implement masked multi-label classification to predict the multiple attributes in the masked behavioral words. More precisely, for each target attribute in masked token, linear layer is connected the final representations and learned to map a distribution over the vocabulary in Figure 2 For one masked token, the training loss is the sum of cross-entropy losses of all the attribute predictions, e.g., the prediction of the shop IDs and genre IDs, etc. The final loss for one input sequence the sum of the losses of all masked tokens.
Pretraining tasks.
the attribute, as illustrated
For masking input tokens, we follow a similar process as BERT: 15% of tokens are selected uni- formly, where 80% of the time the token is zeroed-out and remains unchanged otherwise. We distin- guish between three segments of behavioral words from the three types of user data, i.e., long-term short-term and user profiles. For long and short-term segments, we apply the masking-prediction for pretraining our model, while we do not mask user profiles. To pretrain UserBERT, we first randomly sample a mini-batch of raw user sequences. Then, they are tokenized and transformed to input rep- resentations, which is followed by the masking step. In the end, the masked sequences are passed through the model, and the model is trained by minimizing the prediction eтror for reconstructing what attributes are inside the masked tokens. For each attribute type, linear layer is learned to map the hidden representations of masked tokens to distributions over its vocabulary for conducting the multi-label classification.
5

---

Under review as a conference paper at ICLR 2021
Let i be randomly sampled index for masking, w, and wy be the masked behavioral word and the input after masking to the UserBERT. Also, let n the number of target attributes for reconstruction prediction, and f (w,))0) be the final output vector after softmax layer for k-th attribute in the masked W;- The loss of the UserBERT model is:
L( -wwn D (
(1)
where w is a uniformly sampled input representation sequence from the training dataset D, yf is the ground truth binary vector for the k-th attribute with its corresponding vocabulary size in the masked wi and ICE is the cross entropy loss for the multi-label classification. Note that long-term and short-term user behavior have different types and numbers of attribute in actions. With the pretrained models, we leverage them for fine-tuning on downstream tasks.
4 EXPERIMENTS
We experimentally verify whether the proposed UserBERT model is able to yield generic user rep- resentations, and evaluate the performance when applying different tasks via transfer learning.
4.1 DATASETS
Datasets are collected from multitude of online ecosystem of services, including an e-commerce platform, a travel booking service, a golf booking service and others. Customers can access all services via their unique ID, and their activities across the ecosystem are linked together.
We consider two action types as long-term user behavior. The first one is the purchase action on the e-commerce platform, and the second one is the point usage history. Points earned whenever purchases are made or when certain services are used and can be spent on any service within the ecosystem. The "channel' attribute represents from which service users obtain points or where they spend points, We collected the purchase and point history data over time period of three months for our experiments. For short-term behavior, we mainly focus on recent customer activities on the e-commerce website, i.e., browsing and search history. The collected actions are clicks, page views and searches over shorter time period of seven days. The detailed information on action types and attributes in the experimental data are shown in Table 1
The user profile data registered eustomer information such as age and gender. The unique number of users in the dataset is 22.5 million, the number of daily purchase and point usage samples is approximately 5 million, and the number of short-term data samples approximately 50 million. The data is preprocessed to generate user action sequences.
Table 1: Action types and attributes in user behavior data
Action type
Attribute (vocabulary size)
usage type (4), channel (742), expense range (17),
long-term
purchase, point usage
shop (85,124) genre (1 1,438),, hour (24)
event type (3), shop (40,804), genre (10,386), device type (2)
short-term
click, search, page view
4.2 TARGET TASKS
We transfer pretrained models to three downstream tasks that aim to improve the customer experi- ence. The user targeting task is to identify potential new customers for certain services products, and it is formulated as a binary classification problem. The seed users who responded positively positive labels, while negative ones uniformly sampled from
to the target service/product the rest of the users with a 3:1 ratio. The dataset is collected after the time period of the data used for pretraining. The second task, next genre prediction, is multi-class prediction problem with

---

Under review as a conference paper at ICLR 2021
the aim to predict the next genre that user will purchase from. The dataset is created from the one-month user history following the pretraining time period. The final attribute prediction task is predicting different user attributes such whether customer owns a pet. It is also a classification problem, where ground truth labels are obtained through questionnaires. The datasets of the three target tasks are split 80-20 create training and testing datasets for fine-tuning.
4.3 BASELINES
The UserBERT is compared to direct modeling without pretraining and to MTL-based pretraining. The MTL models apply a multi-tower architecture in which each tower encodes one type of user data in our experiments. For the MTL-baaed baselines, different types of user data are passed through corresponding encoders, and the encoded representations are combined at the last layer before con- necting to multiple training tasks. The dimension of the combined representations is set to 128 for all MTL models.
We collect user labels across the services in the ecosystem and pretrain MTL models with 12 multi- class classification tasks. These pretraining tasks classify the categories of user activities such the usage frequency of certain services or attributes like type of occupation. By learning and sharing across multiple tasks, the yielded user representations are considered to be generalized and applica- ble for transferring to downstream tasks.
Diee&DDe:+ We generate fixed-length (1130-d) embeddings by aggregating behavior data and input them to the deep part of the model (Cheng et al. (016). Categorical user profile data is mapped to word embeddings and concatenated before feeding it into the wide part of the model. The wide part is linear model, while the dimensions of the 4 hidden layers for the deep side are 512, 256, 256 and 128, respectively.
LSTM+MTL: The same discretization and input generation is applied to long-term and short-term user behavior for this model. It a 3-tower model, in which two LSTMs model the two types of user behavior and user profiles are modeled in the same way as the Weee&Deep model. The dimension of the hidden state in all LSTM encoders and the length limitation of both long-term and short-term data are set to 128.
The architecture is the same as the LSTM+MTL model above but with two different Transformer encoders (Vaswani et al., 2017) model long and short-term user data sep- arately. The length of input user behavior sequence to the encoders is limited to 128 as well. We pretrain the model via minimizing the summed cross-entropy loss of the multiple training tasks.
Mra
The proposed self-super learning based pretraining model. It enables a simultane- ous learning from long, short user behavior and user profiles. Its pretraining is done by reconstruct- ing attributes in masked tokens via multi-label classifications.
UserBERT:
4.4 EXPERIMENTAL SETUP
For UserBERT, we use the same notations as BERT, and set the number of Transformer blocks L to 4, the hidden size H to 128 and the number self-attention heads to 4. The input sequence length of both long-term and short-term data is limited to 128 in the experiments. For fair comparison, pretrain all models using the Adam optimizer with learning rate of 1e-4 and a batch size of 16. We fine-tune models using the same learning rate and batch size of 128. Pretraining of 400K batches of the UST model takes approximately 12 hours using our PyTorch implementation, run on two GeForce RTX 2080 Ti GPUs.
For fine-tuning each target task, the combined encoder representations of the MTL-based models fed to an output layer, while the fine-tuning of UserBERT is done by connecting the hidden representations of the first token to an output layer for each task. After plugging in task-specifee inputs and outputs, we fine-tune the parameters of pretrained models in an end-to-ent way.
4.5 EXPERIMENT RESULTS
User Targeting. We show the results for two different services. The sizes of the datasets are 30,204 samples and 31,106, respectively. Compared to the size of the pretraining dataset, the use cases of
7

---

Under review as a conference paper at ICLR 2021
BCc Aut (Casw 1)
Assuracy (Eao
ROc AUC (Case
Accurasy (Cass
UERTERT
UberEEST
WeBe/Dep-lft Derd
reach
Figure 3: Fine-tuning performance comparison for the user targeting task. ROC and ac- curacy results on two use cases, predicting new users for two different services.
rnmtrmrrint ankormientt lowe lake
Trensforma- 4I1, Ieaariiron 1911 lown Lsees
Ter. UsetEAT
UnetETT
Toa- towdett uniperta
Figure AUCLAUC comparison between Transformer-basee MTL models with different numbers of labeled data.
Performance comparison between UserBERT with and without pretraining on user targeting task
Figure
this task only have few labeled data. Classification performance terms of accuracy and ROC AUC shown in Figure The LSTM model, which sequentially models user behavior, has relatively low accuracy. One possible explanation that the sequential order of user actions does not provide useful information for this task. From our experience the user targeting task focuses on patterns from relatively static user preferences. The Wide&Deep model shows competitive performance, which reasonable since our exploratory analysis indicates that user profiles are important features. The performance of the Transformer- based models reveal that the underlying explanatory factors for this task can be captured by attention networks. UserBERT outperforms other models in both use cases by substantial margin. We hypothesize that, compared to Transformer-based MTL, the learning of the UserBERT is not limited by the multiple training tasks and is able to learn more expressive and generic representations from the input.
To further demonstrate the advantage of the proposed method over MTL-based pretraining, we pre- train Transformer-based MTL models with different numbers of labels before fine-tuning, We eval- uate three models: without pretraining, trained on 30% of the available labels and trained using all labels. The comparison indicates that the performance of MTL is significantly affected by the number of training samples. As shown in Figure more annotated training data contributes to per- formance gain. The model without the pretraining step shows the worst performance. In contrast, the pretraining of the UserBERT does not require the additional collection of supervision signals, and therefore not impacted by either the quantity or the quality of user annotations.
We also directly apply UserBERT to these two use cases without pretraining to verify whether the user targeting task benefits from the pretraining step. The ROCLAUC comparison between User- BERT with and without pretraining is shown in Figure [5] The pretrained models outperform the direct modeling significantly. This indicates that the pretraining step can extract useful informa- tion and enables the followed fine-tuning to boost performance for downstream tasks. From the error curves during training, we also observe that models tend overfit quickly without pretraining- The pretrained UserBERT model achieves more generie user representations and yields significant accuracy improvements when adapted to new downstream tasks.
Next Genre Prediction. The test dataset contains 586,130 users, and we run 10 epochs of fine- tuning for each pretrained model. The mean average precision (mAP) comparison is shown in Table 2 The UserBERT model outperforms baseline models by large margin. This task requires understanding of both long-term preferences as well as recent interests of customers. Prediction models should be able to pick out candidate genres from user habits over longer time range,
8

---

Under review as a conference paper at ICLR 2021
RDC
AUC pel.eetee
Table mAP@ comparison after 10-epoch fine-tuning on next genre prediction task.
Model
mAP( %))
34
Widc&DDep+MMTL
7.65
LSTM+MTL
6.90
Transforme++MTL
7.10
tooch
cooch
UserBERT 8.97
Figure 6: ROC_AUC comparison attribute pre- diction task.
and then identify likely ones as prediction results from latest interest trends. More specifically, model should understand how users typically use services in the ecosystem well as what they are currently interested in. The architecture the baseline models learns from different types of user data separately and combines the last-layer representations for training. It fails to sufficiently capture the correlations. On the contrary, UserBERT benefits from the unified structure of the user data and captures more accurate correlations, not only within certain types of user behavior, but also between different behavior types via attention networks.
Since it is common that but strong baseline that sets predictions as the most popular genres ranked in descending order by the total number of purchases, and compared it against all pretrained models. The mAP@ 10 is 4.22%, demonstrating the effectiveness of the pretrained models.
make purchases from only subset of genres, we also built intuitive
Attribute Prediction. In general, is challenging predict user attributes because the predictive signals in the behavior data are very sparse. In other words, the target user attributes may not be strongly correlated to behavior data. Therefore, this prediction task evaluates the model's ability to discover hidden explanatory factors in the raw data. We show experimental results of two use cases: one is to predict whether user has a car while the other one is to predict if a user is parent. These two tasks are denoted as has has_car and is parent.
The dataset for the has car task contains 448,501 samples and the one for the parent task contains 400,268. The classification results of 10-epoch fine-tuning are shown in Figure From the hus car results, we observe that the Wide& Deep model shows good performance, although other models eventually reach similar accuracy. We believe this is due to the fact that user demographies like age and living area are important features for this task. It seems challenging for models to extract other decisive patterns from either long-tern user a parent or not seems to present different characteristics in terms of how they behave on an e- commerce or travel booking platform. These patterns be captured by deep learning models like UserBERT and Transformer-based models. UserBERT is able to match and eventually outperform the baseline models.
short-term user behavior. On the other hand, whether
CONCLUSIONS
This paper introduces novel paradigm to understand user behavior by using the analogy to language understanding, We present UserBERT, extension of the BERT model to user data, for pretraining user representations in a self-supervised way. It explores and demonstrates the possibility for user- oriented machine learning tasks to alleviate the dependency experiments show that a well-designed pretrained model with self-supervision is able to outperform fully supervised learning models when transferred to downstream applications.
large annotated datasets. Extensive
REFERENCES
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 1788 2013. ISSN 0162-8828.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint 2000022002.05709, 2020.

---

Under review as a conference paper at ICLR 2021
Meng-Tze Cheng, Levent Koe, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado. Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. Wide & deep learning for recommender systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, pp- 7-10, 2016.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems, pp. 191-198, 2016.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV). pp. 1422 1430, 2015.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Dar- rell. Decaf: A deep convolutional activation feature for generic visual recognition. In Proceedings of the 31st International Conference on Machine Learning, pp. 647-655, 2014.
Deepti Ghadiyaram, Matt Feiszli, Du Tran, Xueting Yan, H. Wang, and D. Mahajan. Large-scale pre-training for video action recognition. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12038-12047,200 2019.
Mihajlo Grbovic and Haibin Cheng. Real-time personalization using embeddings for search ranking at airbnb. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 31 311-320, 2018.
Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioriti- zation for multitask learning. In ECCV, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference Computer Vision (ICCV), pp. 1026-1034, 2015.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In ICML, 2019.
Devon Hjelm, Alex Fedorov, Samuel Lavoie Marihl-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. 2018 IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7482-7491, 2018.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting sulf-super visual repre- sentation learning. 2019 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1920-1929, 2019.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self sule-supervised learning of language representations. In ICLR, 2020.
Zhouhan Lin, Minwei Feng, C. D. Santos, Mo Yu, B. Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In International Conference on Learning Represen- tations, 2017.
Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye- Yi Wang. Representation learning using multi-task deep neural networks for semantic classification and information trieval. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 912-921, 2015.
10

---

Under review as a conference paper at ICLR 2021
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4487-4496, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for saioo-and-languaal tasks. In Advances in Neural Information Processing Systems, pp. 13-23, 2019.
Tomas Mikolov, Ily Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representa- tions of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems, pp.
2013.
Yabo Ni, Dan Ou, Shichen Liu, Xiang Li, Wenwu Ou, Anxiang Zeng, and Luo Si. Perceive your users in depth: Learning universal user representations from multiple e-commerce tasks. In Pro- ceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 596-605, 2018.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representions by solving jigsaw puzzles. In ECCV, 2016.
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level image representations using convolutional neural networks. 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1717-1724, 2014.
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2536-2544, 2016.
Changhua Pei, Yi Zhang, Yongfeng Zhang, Fei Sun, Xiao Lin, Hanxiao Sun, Jian Wu, Peng Jiang, Junfeng Ge, Wenwu Ou, and Dan Pei. Personalized re-ranking for recommendation. Proceed- ings of the 13th ACM Conference on Recommender Systems, pp. 3-11. 2019.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543,
2014.
Matthew Peters, Sebastian Ruder, and Noah Smith. To tune or not to tune? adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP, pp. 7-14, 2019.
Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti. Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data. CoRR, abs/2001.0796 2020.
Matthew Richardson, Ewa Dominowska, and Robert Ragno. Predicting clicks: Estimating the click-through rate for new ads. In Proceedings of the 16th International World Wide Web Senffernne(WWWW2000)) 2007.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. ArXiv, abs/1706.05098, 2017.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 1134 141, 2018
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. CNN features offf-the-heelf an astounding baseline for recognition. In CVPR Deep workshop, 2014.
Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Iasonas Kokkinos, Pascal Fua, and Francesc Moreno- Noguer. Discriminative Learning of Deep Convolutional Feature Point Descriptors. Proceed- ings of the International Conference Computer Vision (ICCV), pp. 118-126, 2015.
Trevor Scott Standley, Amir Roshan Zamir, Dawn Chen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? ArXiv, abs/1905.07553. 2019.
11

---

Under review as a conference paper at ICLR 2021
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre- training of generic visual- Visual-linguistic representations. International Conference on Learning Representations, 2020.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: joint model for video and language representation learning. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7463-7472, 2019a.
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequen- tial recommendation with bidirectional encoder representations from transformer. In Proceed- ings of the 28th ACM International Conference on Information and Knowledge Management, pp- 1441-1450, 2019b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st In- ternational Conference on Neural Information Processing Systems, pp. 6000-6010, 2017.
Jiawei Wu, Xin Wang, and William Yang Wang. Self-supervised dialogue learning. In ACL, pp- 3857-3867, 2019.
Yu Zhang and Qiang Yang. A survey multi-task learning. ArXiv, 0oss/177..08111 14, 2017.
Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma. Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp- 1059-1068, 2018
Guorui Zhou, Na Mou, Ying Fan, Qi Pi. Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. Deep interest evolution network for click-through rate prediction. AAAI, 2019.
Zeyuan Allen Zhu, Weizhu Chen, Tom Minka, Chenguang Zhu, and Zheng Chen. A novel click model and its applications to online advertising. In Proceedings of the Third ACM International Conference on Web Search and Data Mining, pp. 321321-3 321-330, 2010.
12


Reviews:
[{"review_id": "z0udscpw8wO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an approach to learning user representations based on activity patterns on e-commerce websites and a user profile. The method turns activity patterns into a sequence of discrete tokens based on the action type and attributes that correspond to a certain action. A self-supervised transformer is trained on this data with a masked language modeling (MLM) objective. Data is compartmentalized as long-term patterns such as a purchase or the use of reward points or short-term such as clickthrough data or user profile information such as user age, gender, or location. Separate segment and position embeddings are used within each compartment. Since each masked token is a high-level action type that may have many attributes, predicting a masked-token is cast as a multi-label classification problem over attributes.\n\nThe trained model is evaluated on downstream user targeting and user attribute prediction benchmarks.\n\nOverall, the paper is a straightforward application of BERT pre-training (with MLM only) to learn user representations. The main contribution of this work is the tokenization/discretization strategy and multi-label classification to enable masked language modeling.\n\nStrengths\n\nThe overall approach of leveraging user activity patterns to learn user embeddings in a self-supervised way is well motivated and uses well-established methods like BERT to achieve this goal.\nThe discretization/tokenization and multi-label classification idea is well thought out.\n\nWeaknesses\n\nLimited discussion of previous work in this space of user representation learning (the user modeling section lacks any citations)\nStraightforward BERT application with somewhat mixed results on some tasks (case 1 on user targeting and “has_car” attribute prediction)\nIt is hard to determine what improvements are meaningful since the datasets used are standard to the best of my knowledge and there aren’t any significance testing.\n\nQuestions & Comments:\n\nWhat is the baseline accuracy of predicting the same genre as the previous genre purchased in the next genre prediction task?\n\nThe limited supervision setting was unclear to me (especially in Figures 4 & 5). Please correct me if I’ve interpreted the results incorrectly. Figures 4 & 5 investigate *supervised* pre-training on labeled user data. The Transformer + MTL model when pre-trained on 12 classification tasks that require supervised data 1) gets better with more supervised pre-training data 2) does better than UserBERT with the same pre-training on supervised tasks. Figure 5 shows that UserBERT begins to overfit.\n\nA motivation mentioned was that self-supervised pre-training allows for strong semi-supervised learning and learning with limited supervision however most performance curves report performance as a function of the number of epochs rather than the amount of labeled data available for downstream tasks like user targeting and attribute prediction. Strong performance with limited labeled data would make this work better.\n\nThe paragraph that starts with “Inspired by the BERT model and its variations” contains a sentence like “understanding users in a similar way to how language is understood” and “syntax and semantics of sentence are comparable with behavioral patterns and characteristics of a user” both (especially the latter) are pretty bold claims and are largely unnecessary to motivate the proposed model. It should suffice to say that the model is inspired by BERT which has been immensely useful across a host of NLP tasks.\n\nThe section on user modeling has no references to previous work that looks at building user representations. The recommendation systems literature contains several such pieces of work and these should be discussed. Along these lines, how would the learned representations compare to those obtained from say SVD of the user-purchase/user-click/user-search matrix?\n\nThis work appears to have some potential for misuse, especially when trying to infer protected user attributes from user embeddings. This deserves to be discussed in some detail.\n\nIt's not clear to me why reporting area under the ROC curve as a function of the number of epochs is meaningful. Why aren’t models trained with early stopping?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ms4CLZz5kTb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Clarity:\n\nI find the paper lacks details on multiple aspects (please see my comments below).\n\nOriginality:\n\nThe idea introduced in the paper is interesting. Most of the originality comes from the fact that the author is converting a user modeling problem into a pre-training problem defined on a special \"vocabulary\", aka action types, attribute IDs.\n\nSignificance:\n\nI am not convinced that the result is significant. Mostly because the experiment is hard to be reproduced due to lack of description of modeling details and the dataset being used. \n\n\nSome other comments:\n\n* It would be more helpful to point out the difference in more details that how your work is different from other related work in section 2.1 and 2.2.\n\n* In section 3.2, \"Tokenization of user behavior sequences\", if I understand correctly, (at least some subset of) the attribute type of an user action is largely dependent on an external topic model or topic classifier (for example: to identity the main topic of a webpage, \"sport news\", or to recognize the relevant entity from a shop, \"nike shoes\" ). This part is not clearly explained in the paper. I would recommend using an appendix section to explain it further. Otherwise it is impossible for other readers to reproduce the results mentioned in this paper.\n\n* Can you rephrase this sentence \"The token representation is computed by the concatenation and mean calculation of the word embeddings of the attribute IDs in each action\" (Under Figure 1.) ? I am confused how this is done. Is it a mean word embedding for each attribute ID? Then we concatenate all of the mean embedding vectors?\n\n* In section 3.2 \"Input representations\", could you give an example of \"token\" (I see you said \"age\" is a token, is there any other examples? is \"token\" something only related to user demography) and \"action\"? what is the difference between them? This give me confusion when reading the second paragraph under \"Input representations\".\n\n* Under \"Pretraining tasks\", \"The final loss for one input sequence is the weighted sum of the losses of all masked tokens.\". Can you elaborate on what the \"weighted sum\" refers to? What is the weight? Are you assigning some weights to different type of user behavior, e.g., long-term, short-term and user profiles? How is that weight being decided?\n\n* In the experiment section 4.1, the dataset used in the paper does not seem public-available. Or do I missing something? This also blocks reproducibility of  the paper results. Is there anything you can use in the public world? Otherwise can you at least give more details (besides those already in 4.1) on the underline dataset in  the appendix?\n\n\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mCtQ5UTOou0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "UserBERT: Self-supervised User Representation Learning \n \n######################################################################\n\nSummary: \n  \nThe paper provides an extension of BERT to user data for pre-training user representation in a self-supervised manner. In particular, it analogise the user behaviour sequence to words in a sentence and leverages the Masked Language Model (MLM) approach typically used in NLP to train the user embedding. To facilitate such extension, the paper also proposes a discretisation approach and a unified input structuring to include long-term, short-term and demographic information.  \n  \n######################################################################\n\nPros:  \n  \n1. Even though, the idea of extending the self-supervised pre-training for user representations is not new, it is still an interesting area of research \n \n2. The discretisation of user behaviour signals over long-term and short-term to form “behavioural words” is quite reasonable\n  \n##################################################################### \nConcerns:  \n \nThe key concerns about the contributions of the paper are as follows: \n\nOverall, the novelty of this work is very limited. To elaborate:\nThe major contribution of this work is two fold:\n- Discretisation of raw user behaviour sequences (for long-term and short-term) and\n- Using the discretised aggregated “behaviour words” as inputs to the BERT architecture as is. \n\nThe other claimed contributions such as having a unified architecture and experiments to validate the approach do not seem substantial.\n\nWhen it comes to “discretisation”, though the idea seems appropriate, two crucial questions regarding this step are not validated:\n\n1) There is no empirical evidence presented in the paper which shows “discretisation” improves UserBERT’s accuracy. Since it is a major contribution, I request the authors to design and implement an ablation study to address this point.\n\n2) Seemingly, the authors have come up with a hand crafted/heuristic-driven  approach for discretisation. Why can’t it be data-driven too? Meaning, can clustering of actions be done in a data-driven manner? If so, what is the difference in accuracy between the proposed heuristic-driven and the data-driven alternative.\n\nWhen it comes to the second contribution, it is certainly not novel i.e., the paper does not propose any architectural change to BERT. Though the paper claims that the presented model is a unified model to learn long term, short term and demographics based user profile, the unification is brought upon as a by-product of feeding multimodal inputs to vanilla BERT. \n\nHence, the overall novelty of the paper is very limited.\n\n\nThe following comments are my major concerns in each section of the paper:\n\nSection 2:\n\nWhile the authors have reviewed some literature in transfer learning and self supervised learning and have cited some relevant work, they have not cited even one reference in section 2.3 which on “User Modeling” i.e., the main theme of the paper. I request the author to make a thorough survey and cite related work in section 2.3 and also highlight how UserBERT is different from them.\n\nSection 3:\n\nOverall, this section (and section 4) lack cohesion and can be written clearer with the figures, tables, algorithms and descriptions. This could help the reader better understand the approach. For instance, the following main points in the approach are not explained well:\n\n1) The paper states “the final loss for one input sequence is the weighted sum of the losses of all masked tokens”. There is no detail what the weights are, whether they are assigned based on heuristic or learnt.\n\n2) The approach considers “ordinal attributes” such as expense and age similar to “categorical attributes” (e.g., each age has a unique embedding). This seems counterintuitive and there is no empirical evidence to show that this counterintuitive design works well.\n\n3) It is not clear how to use the hidden representation to predict attributes from the transformed masked tokens. More precisely, it is possible that many attributes belonging to different actions are masked and then converted into one token embedding. So what attributes are to be predicted in the final fully connected layer, in this case?\n\n4) Minor concern: what does E stand for in equation 1? \n\nSection 4:\n\nOverall, in this section, the experimental design is not comprehensive, and the results are not convincing for the following reasons:\n \n- Along with the Wide&Deep, LSTM, Transformers as baseline, it would have been better to also include vanilla BERT to the baselines against which the UserBERT can be compared. In fact, Vanilla BERT would be the closest and most appropriate baseline for comparison. Hence, I request the authors to include it.\n\n- All the experiments are conducted on custom datasets. Since user profiling is an extremely useful and ubiquitous activity that benefits multiple domains, I request the authors to experiment UserBERT on well-known open source e-commerce (and other user profiling) datasets (ref: [1] and [2]). In fact, the profiles could be tested on downstream tasks like “next genre prediction” with these datasets. This will help the reader to trust the UserBERT model better.\n\n- Input representation, being one of the major contributions of the paper, it would give more insights if an ablation study is made on the user behaviour data (long-term features, short-term features, demographic features) to compare and contrast the contribution and lift by each of the behaviour categories \n\n- In the attribute prediction task, within the two attributes experimented, the performance of the proposed model is quite unconvincing. Would benefit if more experiments are performed.\n\n- For the “next genre prediction”, though there are more than 10k genres, each users’ typically have a very small subset of interest. Therefore, it would be more informative if can compare the model’s mAP@10 with the user-level mode’s mAP@10.\n\n- The discussions of results are very vague and could be a lot deeper and precise.\n\n\n##################################################################### \n\nQuestions during rebuttal period:  \n  \nPlease address and clarify the concerns above  \n  \n##################################################################### \n\nReference\n\n[1] Sun, Fei, et al. \"BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer.\" Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2019.\n\n[2] Kang, Wang-Cheng, and Julian McAuley. \"Self-attentive sequential recommendation.\" 2018 IEEE International Conference on Data Mining (ICDM). IEEE, 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "WZKoggSn-fA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##\nI have read the author response and I still think the paper is limited in terms of novelty, significance and experiments. I would like to keep my current score. \n##\n\nThis work proposed a self-supervised pre-training approach to model users from user behavior time-series data. Given time-series data of user actions on ecommerce websites, web browsing, etc.,  a self-supervised prediction task similar to masked language modeling is introduced where a transformer model predicts masked actions from the context. Paper claims to show that the proposed pre-training approach performs better than baselines pre-trained using multitasking on user behavior prediction tasks such as next purchase. \n\nPros\n* Good user models can be useful in recommendation systems and paper tackles an important problem\n* Proposed pre-training approach based on time-series data is interesting and looks reasonable\n* Paper is generally well written and easy to follow\n\nCons\n* Limited novelty and significance\n* Weak experiments\n* Related work on user modeling may not be comprehensive enough\n\nRelated work not comprehensive: The paper doesn’t do a good job of positioning the work in the context of prior work. I don’t see any papers discussed in the user modeling section of the related work, which raises concerns about the significance of this work. \n\nPoor baselines: The model is compared against baselines constructed by the authors, some of which are vaguely described (Eg. I don’t understand the Wide&Deep baseline). I am not sure if these are strong baselines. It is also unclear how good the multi-task learning baselines are and there isn’t any significant discussion about the tasks/hyperparameters used to train these baselines. \n\nLimited novelty/contributions: The approach can be considered an application of masked language modeling. I’m not sure if the details about tokenization bear much significance. If the authors claim this to be a contribution, the same should be verified by comparing against other ways of tokenizing the data. Modulo this detail, the approach is a straightforward application of BERT-style pre-training. \n\nI don’t think the analogy between language and user behavior needs to be mentioned/adds any value. Masked language model pre-training can be applied to any time-series data.\n\nAlthough the paper claims to model users, I couldn’t find details in the paper about how exactly a user-model is constructed from the pre-trained model and how these models are fine-tuned on the target task. \n\nWhile the paper attempts to address an important problem, it has serious issues in terms of limited novelty/significance of contributions and weak experiments.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper discusses an extension of BERT for learning user representations based on activity patterns in a self-supervised setting. All reviewers have concerns about the validity of the claims and the significance of the experimental results. Overall, I agree with the reviewers that the paper needs more work to be published at ICLR. I recommend rejection.
Decision: Reject