Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents an approach to learning user representations based on activity patterns on e-commerce websites and a user profile. The method turns activity patterns into a sequence of discrete tokens based on the action type and attributes that correspond to a certain action. A self-supervised transformer is trained on this data with a masked language modeling (MLM) objective. Data is compartmentalized as long-term patterns such as a purchase or the use of reward points or short-term such as clickthrough data or user profile information such as user age, gender, or location. Separate segment and position embeddings are used within each compartment. Since each masked token is a high-level action type that may have many attributes, predicting a masked-token is cast as a multi-label classification problem over attributes.

The trained model is evaluated on downstream user targeting and user attribute prediction benchmarks.

Overall, the paper is a straightforward application of BERT pre-training (with MLM only) to learn user representations. The main contribution of this work is the tokenization/discretization strategy and multi-label classification to enable masked language modeling.

Strengths

The overall approach of leveraging user activity patterns to learn user embeddings in a self-supervised way is well motivated and uses well-established methods like BERT to achieve this goal.
The discretization/tokenization and multi-label classification idea is well thought out.

Weaknesses

Limited discussion of previous work in this space of user representation learning (the user modeling section lacks any citations)
Straightforward BERT application with somewhat mixed results on some tasks (case 1 on user targeting and “has_car” attribute prediction)
It is hard to determine what improvements are meaningful since the datasets used are standard to the best of my knowledge and there aren’t any significance testing.

Questions & Comments:

What is the baseline accuracy of predicting the same genre as the previous genre purchased in the next genre prediction task?

The limited supervision setting was unclear to me (especially in Figures 4 & 5). Please correct me if I’ve interpreted the results incorrectly. Figures 4 & 5 investigate *supervised* pre-training on labeled user data. The Transformer + MTL model when pre-trained on 12 classification tasks that require supervised data 1) gets better with more supervised pre-training data 2) does better than UserBERT with the same pre-training on supervised tasks. Figure 5 shows that UserBERT begins to overfit.

A motivation mentioned was that self-supervised pre-training allows for strong semi-supervised learning and learning with limited supervision however most performance curves report performance as a function of the number of epochs rather than the amount of labeled data available for downstream tasks like user targeting and attribute prediction. Strong performance with limited labeled data would make this work better.

The paragraph that starts with “Inspired by the BERT model and its variations” contains a sentence like “understanding users in a similar way to how language is understood” and “syntax and semantics of sentence are comparable with behavioral patterns and characteristics of a user” both (especially the latter) are pretty bold claims and are largely unnecessary to motivate the proposed model. It should suffice to say that the model is inspired by BERT which has been immensely useful across a host of NLP tasks.

The section on user modeling has no references to previous work that looks at building user representations. The recommendation systems literature contains several such pieces of work and these should be discussed. Along these lines, how would the learned representations compare to those obtained from say SVD of the user-purchase/user-click/user-search matrix?

This work appears to have some potential for misuse, especially when trying to infer protected user attributes from user embeddings. This deserves to be discussed in some detail.

It's not clear to me why reporting area under the ROC curve as a function of the number of epochs is meaningful. Why aren’t models trained with early stopping?

review_title: Official Review - AnonReviewer2

final_score: rating: 3: Clear rejection
review_content: UserBERT: Self-supervised User Representation Learning 
 
######################################################################

Summary: 
  
The paper provides an extension of BERT to user data for pre-training user representation in a self-supervised manner. In particular, it analogise the user behaviour sequence to words in a sentence and leverages the Masked Language Model (MLM) approach typically used in NLP to train the user embedding. To facilitate such extension, the paper also proposes a discretisation approach and a unified input structuring to include long-term, short-term and demographic information.  
  
######################################################################

Pros:  
  
1. Even though, the idea of extending the self-supervised pre-training for user representations is not new, it is still an interesting area of research 
 
2. The discretisation of user behaviour signals over long-term and short-term to form “behavioural words” is quite reasonable
  
##################################################################### 
Concerns:  
 
The key concerns about the contributions of the paper are as follows: 

Overall, the novelty of this work is very limited. To elaborate:
The major contribution of this work is two fold:
- Discretisation of raw user behaviour sequences (for long-term and short-term) and
- Using the discretised aggregated “behaviour words” as inputs to the BERT architecture as is. 

The other claimed contributions such as having a unified architecture and experiments to validate the approach do not seem substantial.

When it comes to “discretisation”, though the idea seems appropriate, two crucial questions regarding this step are not validated:

1) There is no empirical evidence presented in the paper which shows “discretisation” improves UserBERT’s accuracy. Since it is a major contribution, I request the authors to design and implement an ablation study to address this point.

2) Seemingly, the authors have come up with a hand crafted/heuristic-driven  approach for discretisation. Why can’t it be data-driven too? Meaning, can clustering of actions be done in a data-driven manner? If so, what is the difference in accuracy between the proposed heuristic-driven and the data-driven alternative.

When it comes to the second contribution, it is certainly not novel i.e., the paper does not propose any architectural change to BERT. Though the paper claims that the presented model is a unified model to learn long term, short term and demographics based user profile, the unification is brought upon as a by-product of feeding multimodal inputs to vanilla BERT. 

Hence, the overall novelty of the paper is very limited.


The following comments are my major concerns in each section of the paper:

Section 2:

While the authors have reviewed some literature in transfer learning and self supervised learning and have cited some relevant work, they have not cited even one reference in section 2.3 which on “User Modeling” i.e., the main theme of the paper. I request the author to make a thorough survey and cite related work in section 2.3 and also highlight how UserBERT is different from them.

Section 3:

Overall, this section (and section 4) lack cohesion and can be written clearer with the figures, tables, algorithms and descriptions. This could help the reader better understand the approach. For instance, the following main points in the approach are not explained well:

1) The paper states “the final loss for one input sequence is the weighted sum of the losses of all masked tokens”. There is no detail what the weights are, whether they are assigned based on heuristic or learnt.

2) The approach considers “ordinal attributes” such as expense and age similar to “categorical attributes” (e.g., each age has a unique embedding). This seems counterintuitive and there is no empirical evidence to show that this counterintuitive design works well.

3) It is not clear how to use the hidden representation to predict attributes from the transformed masked tokens. More precisely, it is possible that many attributes belonging to different actions are masked and then converted into one token embedding. So what attributes are to be predicted in the final fully connected layer, in this case?

4) Minor concern: what does E stand for in equation 1? 

Section 4:

Overall, in this section, the experimental design is not comprehensive, and the results are not convincing for the following reasons:
 
- Along with the Wide&Deep, LSTM, Transformers as baseline, it would have been better to also include vanilla BERT to the baselines against which the UserBERT can be compared. In fact, Vanilla BERT would be the closest and most appropriate baseline for comparison. Hence, I request the authors to include it.

- All the experiments are conducted on custom datasets. Since user profiling is an extremely useful and ubiquitous activity that benefits multiple domains, I request the authors to experiment UserBERT on well-known open source e-commerce (and other user profiling) datasets (ref: [1] and [2]). In fact, the profiles could be tested on downstream tasks like “next genre prediction” with these datasets. This will help the reader to trust the UserBERT model better.

- Input representation, being one of the major contributions of the paper, it would give more insights if an ablation study is made on the user behaviour data (long-term features, short-term features, demographic features) to compare and contrast the contribution and lift by each of the behaviour categories 

- In the attribute prediction task, within the two attributes experimented, the performance of the proposed model is quite unconvincing. Would benefit if more experiments are performed.

- For the “next genre prediction”, though there are more than 10k genres, each users’ typically have a very small subset of interest. Therefore, it would be more informative if can compare the model’s mAP@10 with the user-level mode’s mAP@10.

- The discussions of results are very vague and could be a lot deeper and precise.


##################################################################### 

Questions during rebuttal period:  
  
Please address and clarify the concerns above  
  
##################################################################### 

Reference

[1] Sun, Fei, et al. "BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer." Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2019.

[2] Kang, Wang-Cheng, and Julian McAuley. "Self-attentive sequential recommendation." 2018 IEEE International Conference on Data Mining (ICDM). IEEE, 2018.
review_title: Official Blind Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##
I have read the author response and I still think the paper is limited in terms of novelty, significance and experiments. I would like to keep my current score. 
##

This work proposed a self-supervised pre-training approach to model users from user behavior time-series data. Given time-series data of user actions on ecommerce websites, web browsing, etc.,  a self-supervised prediction task similar to masked language modeling is introduced where a transformer model predicts masked actions from the context. Paper claims to show that the proposed pre-training approach performs better than baselines pre-trained using multitasking on user behavior prediction tasks such as next purchase. 

Pros
* Good user models can be useful in recommendation systems and paper tackles an important problem
* Proposed pre-training approach based on time-series data is interesting and looks reasonable
* Paper is generally well written and easy to follow

Cons
* Limited novelty and significance
* Weak experiments
* Related work on user modeling may not be comprehensive enough

Related work not comprehensive: The paper doesn’t do a good job of positioning the work in the context of prior work. I don’t see any papers discussed in the user modeling section of the related work, which raises concerns about the significance of this work. 

Poor baselines: The model is compared against baselines constructed by the authors, some of which are vaguely described (Eg. I don’t understand the Wide&Deep baseline). I am not sure if these are strong baselines. It is also unclear how good the multi-task learning baselines are and there isn’t any significant discussion about the tasks/hyperparameters used to train these baselines. 

Limited novelty/contributions: The approach can be considered an application of masked language modeling. I’m not sure if the details about tokenization bear much significance. If the authors claim this to be a contribution, the same should be verified by comparing against other ways of tokenizing the data. Modulo this detail, the approach is a straightforward application of BERT-style pre-training. 

I don’t think the analogy between language and user behavior needs to be mentioned/adds any value. Masked language model pre-training can be applied to any time-series data.

Although the paper claims to model users, I couldn’t find details in the paper about how exactly a user-model is constructed from the pre-trained model and how these models are fine-tuned on the target task. 

While the paper attempts to address an important problem, it has serious issues in terms of limited novelty/significance of contributions and weak experiments.

review_title: Limited significance and weak experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: Clarity:

I find the paper lacks details on multiple aspects (please see my comments below).

Originality:

The idea introduced in the paper is interesting. Most of the originality comes from the fact that the author is converting a user modeling problem into a pre-training problem defined on a special "vocabulary", aka action types, attribute IDs.

Significance:

I am not convinced that the result is significant. Mostly because the experiment is hard to be reproduced due to lack of description of modeling details and the dataset being used. 


Some other comments:

* It would be more helpful to point out the difference in more details that how your work is different from other related work in section 2.1 and 2.2.

* In section 3.2, "Tokenization of user behavior sequences", if I understand correctly, (at least some subset of) the attribute type of an user action is largely dependent on an external topic model or topic classifier (for example: to identity the main topic of a webpage, "sport news", or to recognize the relevant entity from a shop, "nike shoes" ). This part is not clearly explained in the paper. I would recommend using an appendix section to explain it further. Otherwise it is impossible for other readers to reproduce the results mentioned in this paper.

* Can you rephrase this sentence "The token representation is computed by the concatenation and mean calculation of the word embeddings of the attribute IDs in each action" (Under Figure 1.) ? I am confused how this is done. Is it a mean word embedding for each attribute ID? Then we concatenate all of the mean embedding vectors?

* In section 3.2 "Input representations", could you give an example of "token" (I see you said "age" is a token, is there any other examples? is "token" something only related to user demography) and "action"? what is the difference between them? This give me confusion when reading the second paragraph under "Input representations".

* Under "Pretraining tasks", "The final loss for one input sequence is the weighted sum of the losses of all masked tokens.". Can you elaborate on what the "weighted sum" refers to? What is the weight? Are you assigning some weights to different type of user behavior, e.g., long-term, short-term and user profiles? How is that weight being decided?

* In the experiment section 4.1, the dataset used in the paper does not seem public-available. Or do I missing something? This also blocks reproducibility of  the paper results. Is there anything you can use in the public world? Otherwise can you at least give more details (besides those already in 4.1) on the underline dataset in  the appendix?









review_title: Interesting approach to model user behavior but lacking of modeling details

Meta-review: The paper discusses an extension of BERT for learning user representations based on activity patterns in a self-supervised setting. All reviewers have concerns about the validity of the claims and the significance of the experimental results. Overall, I agree with the reviewers that the paper needs more work to be published at ICLR. I recommend rejection.
Decision: Reject