Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper deals with the problem of obtaining user representations from behavior sequences and proposes a method introducing the idea of a "bag of features" and multi-head attention. More specifically, a behavior sequence is expressed by a sequence of multi-dimensional vectors and a fixed-length segment of the sequence is converted to a histogram, where each bin corresponds to a cluster of vectors. Each is then fed to a multi-head attention module to obtain a segment representation and all the segment representations are aggregated into a single vector through a time aggregation module. Experimental evaluations for several downstream tasks demonstrate the effectiveness of the proposed method against several simple baselines.
main_review: [Advantages]
- Modeling users and user behaviors are essential, especially for web-based applications. The problem dealt with in this paper will be a promising application of representation learning based on neural networks.
- The structure of the proposed method is reasonable, especially for e-commerce applications.

[Drawbacks]
- I could not understand the novelty of the proposed method. The bag-of-features representation seems to be standard in several research fields such as computer vision, natural language processing, and data mining. The "multi-anchor module" seems to be almost the same as the multi-head attention module that is famous in the transformer. The time aggregation module does not contain a special trick. According to the introduction of this paper, the novelty of the proposed method is the ability for modeling life-long behaviors. However, I could not find any special techniques for handling life-long sequences. Please clearly state the technical novelty of the proposed method.
- The proposed method for modeling user behaviors dismisses both microscopic and macroscopic time-series characteristics, which would be significant for tracing user interest. More specifically, bag-of-interest representations remove microscopic characteristics, and the time aggregation module erases macroscopic features. If the authors believe that I have some misunderstanding, please justify it.
- When obtaining stable user representations, the knowledge of life-long learning will be required, since (1) users might have several completely different interests in their minds, (2) each interest lasts only a short term, especially in e-commerce applications, and (3) interests containing old behavior sequences might come back suddenly. This implies that almost all the information containing behavior sequences should be kept in user representations and this problem is exactly in the context of life-long learning. Please perform bibliographic surveys for life-long learning, describe the critical problem for the previous work, and present the novelty and advantage(s)  of the proposed method against the previous work.
- The downstream tasks in the experimental evaluations seem to be too easy and not suitable for evaluating the ability to handle long behavior sequences.
summary_of_the_review: I could not find any positive aspects for accepting this paper to the premier conference in the field of representation learning. Both the methodology and documentation need to be improved for acceptance.

[Post-rebuttal comments]
After reading through the paper and author rebuttals, one of the reasons for misunderstanding among us is a gap in the target time duration of "microscopic" and "macroscopic". My understanding is
- "Microscopic" user behaviors correspond to short sequences of individual actions during single sessions, which last at most a day.
- From this viewpoint, the proposed BoI representation removes "microscopic" information that will be useful for the next user action.
- Monthly and yearly user behaviors might be "microscopic" since a month or a year is shorter than their lifetime.
- However, a month is sufficiently long to capture whole the interests of users, which implies that monthly behaviors are "macroscopic".

I understand that the authors try to model "users", not "user behaviors", as implied by the downstream tasks. I also understand that "microscopic" (by my definition) sequence modeling is not required and only "macroscopic" (again by my definition) user behaviors are only of interest.

I have just revised my evaluations according to the above discussion.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a universal user representation learning approach based on self-supervised learning on long-term user behaviors. The authors first represent user behaviors into sparse vectors, and then encode them into multiple vectors to represent users in different aspects. The authors further study using different time aggregation strategies to tradeoff accuracy and computational cost. Experiments on two datasets show the effectiveness of the proposed approach.
main_review: Strengths:

1. This paper is easy to understand.
2. The problem studied in this paper is important.
3. The proposed multi-scale aggregation method is interesting.

Weaknesses:

1. The contribution of this paper can be somewhat incremental. First, the multi-anchor attention network is also called "multi-aspect attention", which has been widely used in many scenarios [1,2,3]. A very similar contrastive model pertaining approach is also studied in [4]. Thus, the contributions of this paper are somewhat incremental.
2. Discussions and comparisons on universal user modeling methods are insufficient. The authors only include PTUM as the baseline but do not discuss it in Section 2.1. There are also many other methods like [4,5]. They are also not cited nor compared.
3. The design of the item embedding method is problematic. Different from texts that there are rich relations between adjacent words, even adjacent behaviors can be rather random. Thus, it may not be suitable to learn item embeddings by regarding items as words and user behavior sequences as sentences. There are many methods for item embedding learning such as PinSage [6]. 
4. Compared baselines are also insufficient. As mentioned in the second point, there are many user model pretraining methods that are not compared. In addition, the compared basic user modeling techniques are also rather weak (e.g., CNN, HAN, Doc2Vec and TF-IDF).
5. Some details are unclear. PTUM and PeterRec require fine-tuning the user model in downstream tasks. However, in this work it is not clear whether the pretrained user model is finetuned.
6. The selection of some hyperparameters seems to be arbitrary, e.g., the number of anchors and the window size $\beta$. The authors do not provide experiments to show the influence of several key hyperparameters.
7. The generality of the proposed method is not verified. It is not clear whether the proposed method can be generalized to other types of user models.


Minor:

There are some typos and grammatical issues. For example, "committed to learn" and "Comparisonn".

[1] Event detection using hierarchical multi-aspect attention. WWW 2019.

[2] ARP: Aspect-aware neural review rating prediction. CIKM 2019.

[3] Hybrid graph convolutional networks with multi-head attention for location recommendation. World Wide Web 23.6 (2020): 3125-3151.

[4] Contrastive Pre-training for Sequential Recommendation. arXiv e-prints (2020): arXiv-2010.

[5] Parameter-efficient transfer from sequential behaviors for user modeling and recommendation. SIGIR. 2020.

[6] Graph convolutional neural networks for web-scale recommender systems. KDD 2018.
summary_of_the_review: This work has some merits but it suffers from several issues that need to be addressed. Thus, my recommendation is a weak rejection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a universal Lifelong User Representation Model to encode user behaviors in any period with a sparse vector with super-high dimension and map these user features to user representations with almost lossless dimensionality reduction. In this way, the model could easily learn different aspects of user preferences in extremely long sequences and generalize to other downstream tasks.
main_review: Strengths:

--Previous work cannot model universal user representation based on lifelong sequences of user behavior since registration and challenging to make full use of highly long user behavior sequences accumulated in Internet service platforms.

-- This work proposes a universal Lifelong User Representation Model to encode user behaviors in any period with a sparse vector with super-high dimension and map these user features to user representations with almost lossless dimensionality reduction. In this way, the model could easily learn different aspects of user preferences in extremely long sequences and generalize to other downstream tasks.

Weaknesses:

-- The structure of related works is not clear enough. At the end of universal user modeling, this work focuses on the conclusion that previous work dedicated to learning universal user representations cannot leverage information from lifelong user behaviors instead of their generalization ability. The same problem also appeared at the end of lifelong user modeling.

-- In a multi-scale aggregation module, this work captures diverse patterns from user behavior by aggregating several representations generated at different time granularities. However, this method could only use information intra-corresponding representation of every time granularity and cannot explore the information inter different time granularities.

-- The explanation of the considerable gap between BoI and LURM on the two datasets is unconvincing. On the industrial dataset, the performance of BOI is better than LURM, which proved that the design of SMEN cannot bring benefits for BOI.

-- In different downstream tasks, the benefits on category preference identification tasks are tiny, even using behaviors of 5 years on this task. There is no explanation about this phenomenon, and it proves BOI cannot learn much preference information about user behaviors on different categories.

-- This work did not design experiments about multi-scale aggregation modules to prove the effectiveness of process representations of different time granularities.

-- There are some format problems in the paper, such as no period at the end of the first paragraph of the introduction.
summary_of_the_review: This work proposes a universal Lifelong User Representation Model to learn different aspects of user preferences in extremely long sequences and easily generalize to other downstream tasks by the self-supervised method.

However, the experimental design of this work needs to be improved, and more explicability needs to be provided about crucial modules and abnormal experimental phenomena, which have more enlightening significance for the community.
review_title: null

Meta-review: Ultimately somewhat below the threshold based on the scores. The reviewers raise issues of the overall contribution, as well as issues with the design/structure of the model/paper and issues with the experiments. While there are some positive aspects, collectively the issues put the paper below the bar for acceptance.
Decision: Reject