Learning Universal User Representations via Self-Supervised Lifelong Behaviors Modeling
Authors: Bei Yang
Ke Liu
Xiaoxiao Xu
Renjun Xu
Hong Liu
huan xu
Venue: ICLR 2022 Submitted (2022)
Keywords: universal user representation
extremely long sequence modeling
self-supervised learning

Abstract: Universal user representation is an important research topic in industry, and is widely used in diverse downstream user analysis tasks, such as user profiling and user preference prediction. With the rapid development of Internet service platforms, extremely long user behavior sequences have been accumulated. However, existing researches have little ability to model universal user representation based on lifelong behavior sequences since user registration. In this study, we propose a novel framework called Lifelong User Representation Model (LURM) to tackle this challenge. Specifically, LURM consists of two cascaded sub-models: (i) Bag of Interests (BoI) encodes user behaviors in any time period into a sparse vector with super-high dimension (eg. 10^5); (ii) Self-supervised Multi-anchor Encoder Network (SMEN) maps sequences of BoI features to multiple low-dimensional user representations by contrastive learning. SMEN achieves almost lossless dimensionality reduction with the main help of a novel multi-anchor module which can learn different aspects of user preferences. Experiments on several benchmark datasets show that our approach can outperform state-of-the-art unsupervised representation methods in downstream tasks.

Under review as a conference paper at ICLR 2022
LEARNING UNIVERSAL USER REPRESENTATIONS VIA SELF- SULE-IUPERVISED LIFELONG BEHAVIORS MODELING
Anonymous authors Paper under double-bling review
ABSTRACT
Universal user representation is an important research topic in industry, and is widely used in diverse downstream user analysis tasks, such as user profiling and user preference prediction. With the rapid development of Internet service plat- forms, extremely long user behavior sequences have been accumulated. However, existing researches have little ability to model universal user representation based on lifelong sequences of user behavior since registration. this study, we propose novel framework called Lifelong User Representation Model (LURM) to tackle this challenge. Specifically, LURM consists of two cascaded sub-models: (i) Bag of Interests (Bol) encodes user behaviors in any time period into sparse vector H with super-high dimension (e.g.. ) (ii) Self-supervised Multi-anchor Encoder Network (SMEN) maps sequences of Bol features to multiple low-dimensional user representations by contrastive learning. SMEN achieves almost lossless di- mensionality reduction, benefiting from novel multi-anchor module which can learn different aspects of user preferences. Experiments on several benchmark datasets show that our approach outperforms stte-offthe-aa unsupervised repre- sentation methods in downstream tasks.
INTRODUCTION
Inferring user intents through user behavior data has been extensively studied in industrial appli- cations, such as recommendation systems, search engines, and online advertising Dupret & Pi- wowarski 2008); He et al. (2014); [Elkahky et al. (2015): ; Yu et al. (2016). One key aspect in these systems is user modeling, which describes the process of building up and modifying a conceptual understanding of the user Fischer (2001). Essentially, user modeling is to learn user representation that helps to capture the user's interests and preferences to improve the performance on downstream
tasks.
In the literature, there have been many studies that focused on task-specific user modeling, such as user response prediction and personalized recommendation Ren al. (2019); Yu et al. (2019); cet al. (2020). However, the user representation learned by a specific task can hardly be generalized other tasks. As result, specific representation models need to be trained in each downstream task, which requires massive labeled data, training time, computing and storage resources. Given these limitations, universal user representations that can serve variety of downstream tasks are preferred.
Due the sequential form of user behavior data, recurrent neural networks (RNNs) are usually used to encode the temporal dynamics of user behavior sequences Wu et al. (0017) Devooght & Bersini (211);; Zhu Zhu al. (2017):
(2019). Unfortunately, these approaches can only process user behavior sequences with a length of tens or hundreds, while the length can reach hundreds of thousands many social networks and e-commerce services. Moreover, we have verified by experiments that the performance will become better as the user behavior becomes richer in most downstream tasks (Fig. 3). To solve this problem, many methods drown from natural language processing (NLP) Kumar et al. (2016); Dai et al. (2019); Yang et al. (2019) are proposed. They model long sequential data based on hierarchical architectures and memory networks YYng et al.) (0018 Pi al. ) Ren et al. (2019). However, it is still hard for them to encode lifelong user behavior sequences when the length scales up to 1000. And representations they learned through a specific task result in poor generalization capabilities.

---

Under review as a conference paper at ICLR 2022
In this work, we propose novel framework called Lifelong User Representation Model (LURM) to model user behaviors since registration. To meet the needs of extremely long sequence modeling, first introduce model named Bag of Interests (Bol) to summarize items in behavior sequences similar to Bag of Visual Words. In this way, we can use a sparse vector with super-high dimen- sion to represent user behavior in any time period. Then, a Self-supervised Multi-anchor Encoder Network (SMEN) that maps sequences of Bol features to multiple low-dimensional user representa- tions proposed. SMEN consists of three modules: a multi-anchor module which can learn different aspects of user preferences, a time aggregation module which can model evolution of user behav- iors, and multi-scale aggregation module which can learn and aggregate Bol features in different scales. Considering the consistency between user behaviors in different time periods, we introduce a contrastive loss function to the self-supervised training of SMEN. With the designs above, SMEN achieves almost lossless dimensionality reduction.
The main contribution of our work can be summarized as follows:
In this work, novel framework named LURM proposed to model lifelong user behaviors of any length. To the best of our knowledge, it is the first method that has the ability to model lifelong behaviors in the field of universal user representation learning.
We introduce a sub-model named Bol which can encode behaviors in any time period, so that lifelong behavior data can be represented by a sequence of sparse vectors.
We can obtain compressed user representations with little information loss with the help of designed sub-model named SMEN.
Extensive experiments are performed on several real-world datasets. The results demon- strate the effectiveness and generalization ability of the learned user representation.
RELATED WORKS
2.1 UNIVERSAL USER MODELING
Compared with task-specific user modeling that requires more resources, universal user represen- tations are preferred to serve different downstream tasks. In recent years, some works dedicated to learning universal user representations have been proposed. Ni et al. (0188) proposed a repre- sentation learning method based multi-task learning, which enabled the network to generalize universal user representations. Extensive experiments showed the generality and transferability of the user representation. However the effectiveness of this method may still suffer due to the selec- tion of tasks and the need of labels. To release the burden of labeling, (Andrwws & Bishop (2019) proposed novel procedure to learn user embedding by using metric learning. They learned map- ping from short episodes of user behaviors to a vector space in which the distance between points captures the similarity of the corresponding users' invariant features. Gu et al.] (2020) proposed network named self-supervised user modeling network (SUMN) to encode user behavior data into universal representation. They introduced a behavior consistency loss, which guided the model to fully identify and preserve valuable user information under a self-supervised learning framework. Wu et al. (2020) proposed pre-trained user models (PTUM), which can learn universal user models based on two self-supervision tasks for pre-training. The first one was masked behavior prediction, which can model the relatedness between historical behaviors. The second one was next behavior prediction, which can model the relatedness between past and future behaviors. Unfortunately. these methods can only process user behavior sequences with a length of hundreds, and cannot leverage the rich information brought by lifelong user behaviors.
2.2 LIFELONG USER MODELING
Previous works have shown that considering long-term historical behavior sequences for user mod- eling can significantly improve the performance of different tasks Ren et al. (2019; Pi et al. (2019) 2020). Ren et al. proposed a hierarchical periodic memory network for lifelong sequential mod- eling. They built personalized memorization for each user, which remembers both intrinsic user tastes and multi-facet user interests with the learned while compressed memory. Pi et al. decoupled the user modeling from the whole CTR prediction system tackle the challenge of the storage cost
2

---

Under review as a conference paper at ICLR 2022
register Lifelang Uisrr Behaviors
User Represantation
for
Profiling User Preference
(e)Bag Inle
Self Selt-suparvised Multi-anchor Encoder Network(5MEN)
"Iterrest clusters
Muitiiscale Aggregation
Multi-scale Aggregation
large-scale dluteninn
Time Agtrapation
Time Apgregatioon
Medule
Medsle
Mustaarchrr
Mult-ancher
Module
Medule
132.015-..11
hesture Sequencea..
Bel Pesture Sequerco
Bun
Item embedding
Bol Sequences
Figure 1: Illustration of our Lifelong User Representation Model (LURM) for user understanding. LURM consists of two sub-model: (a) Bag Interests (Bol) is used to aggregate user behavior data 'interest' granularity, and is composed of an item embedding module and large scale clustering module. We apply the Bol model on lifelong behavior data to generate multi-scale Bol feature sequences, (b) Self-supervised Multi-anchor Encoder Network (SMEN) is used to learn compressed user representations from Bol features sequences, and is composed of multi- aatti- module, time aggregation module, multi-scale aggregation module and contrastive learning module.
and the system latency. Specifically, they proposed a user interest center module for real-time infer- and memory-based network that can be implemented incrementally. Pi al. also designed search-basee based interest model (SIM) with cascaded two-stage search paradigm to capture the diverse long-tern teng-term interest with target item. Unfortunately, the length of the user behavior sequence that these models can handle is still limited. Moreover, these models are all trained on specific tasks, which limits the generalization ability.
METHODOLOGY
In this work, committed to learning universal user representation through truly lifelong behavior sequences with arbitrary length. For this purpose, we propose framework named Life- long User Representation Model (LURM) which consists of two cascaded sub-models: Bag of In- terests (Bol) and Self-supervised Multi-anchor Encoder Network (SMEN). The overall architecture of LURM is shown in Fig.
3.1 BAG OF INTERESTS
In order to model extremely long lifelong user behavior sequence, we propose to aggregate the con- tent of items under user purchases, clicks, or other behaviors at certain granularity. Inspired by Bag of Visual Words (BoVW) Fei-Fei & Perona (2005), item is the natural granularity for aggregation. But there are billions of items on the entire e-commerce platform, which means that the item vo- cabulary is extremely large. making it infeasible in practice. Therefore, propose model called Bag of Interests (Bol) to aggregate user behavior data at 'interest' granularity. Every "interest" is cluster of similar items and represents certain kind of preference. The size of 'interest' vocabulary is often selected at level of about 10 for retaining enough details. As shown in Fig. [ (a), Bol consists of an item embedding module and large-scale clustering module. For convenience, we focus on text modality only in this work. It should be noted that our method can be extended to multi-modal data easily.
3.1.1
ITEM EMBEDDING MODULE
Like the BoVW model, 'interest' vocabulary supposed be built in our Bol model. There- fore, the embedding of each item is required, so that similar items with close distance in the em- bedding space can be clustered together. Recently, in natural language and image processing, dis-
3

---

Under review as a conference paper at ICLR 2022
criminative approaches based contrastive learning in the latent space have shown great success in the field of representation learning, achieving state-of-thh-an results. Inspired by these works, we design contrastive learning task based on the relation between items drawn from user to learn item embedding similar to item2vec Barkan & Koenigstein (2016).
Given set of users U {u1, u2,..., (err), each user U corresponds to behavior sequence
S [11, T2........ )| where H; E S denotes the i-th item. [U] and |S| denote the number of users and the length of u's behaviors respectively. Generally, the content of an item can be expressed as {wi, u2; ..., W/]]) where w, denotes word from a vocabulary V, and |ar denotes the number of words in the content of æ. Firstly, an encoder with average operation is used to generate item embedding E:
Ex encoder(w1 W2, ..., w|=n) proj(
Wi),
(I)
where W; E Rd is the embedding of word w, and will be learned during training, proj(-) includes two residual blocks, and L2 normalization layer. To construct the contrastive learning task, we sample positive pairs from behavior sequences users randomly. Specifically, two items (æt, De) similar, i.e. a positive pair, if they drawn from the same user behavior sequence and the time interval between the occurrence these two items is less than 3, where is the window size controlling the interval of the two user behaviors. Without loss of generality, the sampled mini-batch with batch size n can be denoted as A = {ar²; positive pair drawn from the behavior sequence St of the i-th user in batch. Then, the contrastive prediction task is defined to identify y² in A (xx²) for a given x², and all other items in A \ ((x y²}
:.r"yy' y² x²2 y² y²,..., x y"), where (æ:", y²) construct a
are negatives. The loss for the positive pair (x² y²) ) is written as
ea(m",y/)/
1(x* y²) llg EVEAN M)/T M)/T'
(2)
cy where g(x, 3) Tea ey | e Ey denotes the cosine similarity between the embedding eæ and the embedding By and the temperature parameter. The final objective the average loss of all positive pairs in the mini-batch which can be written as
1 Loss
E((a'' y²) x²)).
(3)
2n
Noting that, other methods which can be used generate item embedding are feasible.
3.1.2 LARGE-SCALE CLUSTERING MODULE
After the encoder above has been trained, an item embedding set E = {eshigr is obtained, where the complete collection of items the billion level. In order to retain details many as possible, the size of the 'interest' vocabulary is set to be at 10a ~ 10 level. In other words, all items should be clustered into D (e.g.. 10") categories. Considering the large-scale item set, a subset E' E at the million level is sampled, and an efficient clustering algorithm named HDSC Yi et al. (2014) on E' is employed to cluster similar items into the same 'interest'.
After clustering, the cluster centers C' make up an 'interest' vocabulary. Therefore each item can be attributed to one/multiple 'interest(s)" by hard/soft cluster assignment. Take hard cluster assignment as example, each user obtain his/her sparsely high-dimensional Bol feature b, E R2 in time period & according to his/her behavior sequence S, = 11, x2, ..., x|S.]}
b = [log(1 + 2/ In, Ec, ), log 2 5 Eer
(4)
..., log + ) [lr
where a; Cj means item i is assigned the cluster Cj C, I is indicator function.
3.2 SELF-SUPERVISER MULTI-ANCHOR ENCODER NETWORK
As mentioned above, a Bol feature b, for each user is obtained, through the Bol model given behav- ior data in any time period t. Most directly, user representation with super-high dimension by can
4

---

Under review as a conference paper at ICLR 2022
be obtained by applying the Bol model to the whole lifelong time T. However, there two main disadvantages: 1) is not friendly to the downstream tasks since the dimension of representation too high, and 2) it is too crude aggregate the information in the whole lifelong time without considering variations over time.
Therefore, propose get Bol feature sequence by applying the Bol model at each time period of the whole lifelong time. Then, a Self-supervised Multi-anchor Encoder Network (SMEN) is designed to learn compressed user representations from the sequence Bol features.
The lifelong time T' is divided into N parts at a fixed time interval, í.e. T = (41 l2, IN and 4 denotes the i-th time period. In experiments, the time interval is usually set to be montiyy/saasonlyyyearyy granularity. In this way, a sequence of Bol features B = {br, .t..... bty) can be obtained, where br, denotes the Bol feature corresponding to the i-th time period fg. After that, SMEN is used to map B to low-dimensional user representations. As shown in Fig. (b), SMEN consists of a multi-anchor module, a time aggregation module, a multi- scale aggregation module and contrastive learning module. Details of the model will be described the following subsections.
Time Aggregation Module
y
M
anchoor Module
Gpk
+1
+
Sharpon
Sottenax Sharpen
Multi- anchof Attention
Mult anchor Attention
Multi anchor Attention
Anclyr
Andhor
Des
b
8ol Feature Sequence B
Sparse Bol Feature
P.
Figure 2: Illustration of the multi-anchor module in SMEN. The module outputs a group of diverse representations by assigning different portions of each 'interest' to different anchors. The de- notes the dot product operation, @ denotes the scalar multiply operation, and denotes vector sum operation.
3.2.1 MULTI-ANCHOR MODULE
The data of user behavior, which is highly unstructured and complex, implies different preferences of the user. To capture diverse aspects of user preferences, a novel multi-anchor module is proposed. Specifically, suppose there are M anchors, and each of them indicates a certain preference of users. Let b be Bol feature, the module converts b to M low-dimensional representations as shown in Fig. Each representation is computed as
2 ReLU
ReLU Σ aus (cg ReLU Σ aedby Wf),
(5)
where f(by) by We is the 'interest' embedding function, by is the j-th element b, We (Wf, WS.... WG) 3 is the embedding matrix. And ai is the attention weight between the i-th anchor and the j-th 'interest', which measures the portion assigned to the i-th preference from the j-th behavior 'interest'. The weight an defined as
cxp(W WW ai ΣS exp (W/k,)
(6)
where W is the anchor vector corresponding to the i-th anchor, Wa = (Wf, Wa. the anchor matrix, and ky RP is the interest vector corresponding to the j-th 'interest'. We E D Wa E M and WP E H are learned parameters. r² can be computed efficiently since is sparse vector. Due the different anchor vectors, different attention weights can be generated for each 'interest'. Finally, a group of different aggregated representations R (rr1 22 1. pM can be obtained (M indicates the total number of anchors). In this way,
W T
5

---

Under review as a conference paper at ICLR 2022
can learn different aspects of user preferences. Specially, experiments prove that SMEN can achieve almost lossless dimensionality reduction mainly due to the multi-anchor module.
3.2.2 TIME AGGREGATION MODULE
Through the multi-anchor module, a sequence of representation groups R (Rt, Rty. Re for each user is obtained, where Rt, {rr, r f...... r., r?:....,r??,
M.M) is representation group generated by multi-anchor module corresponding to the Bol feature dz, in time period 4, In the time aggregation module, each sequence Rt = {ri, ri....., rix) is aggregated separately to yield new represen- tation F TR... Thus the variable-size representation sequence R be transformed into M fixed-size representations R - (7 F.
There are various methods which can be used to aggregate variable-size sequences, such as aver- age/max pooling and RNNs. Compared average and max pooling, RNNs more appropriate to capture variations over time. Among all RNN-based models, long short-term memory (LSTM) and gated recurrent units (GRU) are most commonly used. Considering that GRU has fewer parameters and is less computationally intensive, GRU is adopted to the time aggregation module in this work.
3.2.3 MULTI-SCALE AGGREGATION MODULE
Under a time division T = {t1 ( we have obtained M representations R {F1, 72 ...... FM] through the Bol, multi-anchor module and time aggregation module as introduced above. It's worth noting that if the time interval is too small, the input sequence of SMEN, i.c. B (be, bt.....,bby ] will become extremely long, which contains extensive details but causes modeling difficulties due catastrophic forgetting. On the other hand, details may be lost if the time interval is too large.
To address this trade-off, multi-scale aggregation module is designed. The module captures diverse patterns from user behavior by aggregating several representations generated at differ- ent granularities. Generally, user behavior in this work is aggregated monthly and yearly, i.e. T = {t1, t,.... tN) [ti tg.... ... (e.g., if the length of time period is years, N is 60 for monthly granularity, and N' is 5 for yearly granularity). Thus two sequences of Bol feature B (br, bty } and B' beg brg..... ....,,by/
byyy } at different scales correspondingly are ob- tained. Then, two groups of representations R = (F F FM) and R' = (--- { are obtained after the multi-anchor module and time aggregation module with parameter sharing. Finally we aggregate R and R' with self self-attention follow:
gs S² S(F FDD @ F* +1 st s' (FFF))))
where O is the Hadamard product, [:] is a operation concatenating vectors along the last dimension, and s.) is the switch function which is implemented as a fully-connected layer followed by a sigmoid function. The switch function controls the fusion between Bol feature sequences of different scales. The output of this module R [+l, 22 ..., M is served as the final user representation.
3.2.4 CONTRASTIVE LEARNING MODULE
As mentioned in 3.1.1 contrastive loss is also used to learn representation in this network. In order to obtain a unified vector for contrastive learning, nonlinear projection head is added, ie. h(-), used in SimCLR Chen et al. (2020). The function h(-) is computed
v ReLU (ww w 22, p²,... Who MM)
(8)
(((v) = whe ReLU(p miz(v, whs).
The function mix(v, Wha) is allow interaction between different preferences, and is imple- mented as
mix(v, Wha) haa (v)),
where r2(:) reshapes to a matrix of size M H, and r2(+) reshapes ( to vector. Whu E H Wha MHMH and Why TR MM are trainable parameters.

---

Under review as a conference paper at ICLR 2022
Two behavior sequences drawn from the same treated positive pair, and behavior quences from other users are negatives. All sequences are continuous sub-sequences randomly sampled from the whole life time T. Finally, the contrastive loss is defined the same as equation (
EXPERIMENTS
In this section, we present our experiments in detail.
4.1 DATASETS
Model comparisons were conducted on public datatset and an industrial dataset. Table Thooss the statistics of two datasets. The public Amazon dataset et al. (2019) contains product reviews and metadata like product titles and categories from Amazon. For each user, the reviewed product titles constitute sequence review behaviors. The dataset for self-supervised user modeling on Amazon DA was constructed by collecting users who have behaviors between 1997-01 and 2017-12, while taking behaviors occurred between 2018-01 and 2018- to form datasets {D2, (t)} (t refers to tasks) for downstream tasks. Industrial dataset was collected from visit/order logs on popular e- commerce platform. We constructed a training dataset D' for self-supervised user modeling by randomly sampling hundred million of users who have behaviors between 2016-06 and 2021-05, while taking behaviors occurred between 2021-07 and 2021-08 to form datasets (dd
Table 1: Statistics of the datasets. Tr (.) indicates the truncation threshold.
Dataset
U
Maz ([S))
Tr(a))
[V]
Amazon
43,531,850
13,122
35
103,581
Industrial
214,317,285
1,952,546
24
178,422
4.2 DOWNSTREAM TASKS
Two kinds of downstream tasks were used to evaluate our method: category preference identification and user profiling prediction. In the experiments, we randomly selected 80% of the samples from DA (t)D', (t) for task to train downstream models and the rest for performance validation.
Category preference identification refers to the task of predicting whether users have short-term preferences in the target category. For Amazon dataset,four categories were included: BBooss of Literature', 'Games of Sports', 'Outdoor- 'Outdoor-hunting' and 'Musical Instrument'. For the industrial dataset, we considered three categories including clothing, shoe and coffee for evaluation.
User profiling prediction aims to identify experiments on industrial dataset. Two tasks were involved: (1) user age classification task (age is divided into 6 classes) which predicts the age ranges of users, and (2) baby age classification task (7 class). In both tasks, the ground-truth labels came from an online questionnaire.
aspects such gender and age. We only conducted
4.3 COMPETITORS
We compared the proposed LURM model against rich set of user representation learning methods, including TF-IDF, Doc2Vec, TexICNN, HAN, PTUM and SUMN. TF-IDF and Doc2Vec view the whole user behavior sequence as a document. TF-IDF generates a sparse high-dimensional vector, while Doc2Vec learns to represent the document by dense vector. We also compared LURM with two methods named TexICNN and HAN which learn task-specific user representations and classifiers in supervised manner. Finally, we compared LURM with two newly self-supervised methods for user modeling mentioned before, PTUM and SUMN.
4.4 TRAINING DETAILS
For item embedding learning, the number of words in each item æ was truncated. The principle of setting truncation threshold is that 95% data values be covered by the threshold. In experiments, word2vec and average pooling were used to obtain item embedding on Amazon dataset, while the method described in 3.1.1 was used the industrial dataset. The entire behavior sequences were

---

Under review as a conference paper at ICLR 2022
input into SMEN because the length of input sequence only depends the time interval. On Amazon dataset, TF-IDF and Vec2 also took the entire behavior sequence as input. Meanwhile, the length of input of TexICNN, HAN, Doc2Vec, SUMN and PUTM limited to 50 due to memory limitation. On industrial dataset, all competitors used data of two months.
We set the latent dimensions of word/item embedding and all hidden layers to 128 in LURM. In item embedding module, the window size / was set to 5 days, and the temperature 7 was 0.1. In order to retain details as many as possible, the size of the 'interest' vocabulary was set 10 and the number of anchors M was set to 10 in our experiments. So the final user embedding dimension was 1280. In addition, the input of month-gramun granularity and year-granularig were used. For all user representation models, the Adam optimizer with a learning rate of 0.001, and a batch size of 256 was used for training. The hyper-parameters of the supervised competitors were tuned on the validation set. For downstream tasks, a simple MLP classifiers was applied after the derived user representation. Note that LURM, SUMN and PUTT were all not fine-tuned for downstream tasks in our experiments.
4.5 RESULTS
Table 2 shows the comparison of category preference identification on the public Amazon dataset. The last two rows show the results of our models under different settings. The first one uses the high- dimensional output of Bol module as input the downstream tasks, while the second uses the output of LURM. It can be seen that LURM consistently outperforms other unsupervised methods, e.g., about 3. 113%%/.1499 average improvements than SUMN and 4.41% 96/2.0% average improvements than PUTM in terms of AUC and ACC respectively. Even compared with supervised methods, LURM outperforms TexICNN and HAN about 8.26% and 3.82% 2.4%% respectively. In particular, it can be observed that LURM has significant improvement over Bol.
Table 3 shows the comparison of two user profiling prediction tasks and three category preference identification tasks the industrial dataset. In this table, Bol and LURM using 2-month input compared. When using the same input, LURMM(sort) achieved better results than other methods on both user profiling prediction tasks and category preference identification tasks. When using longer input, we can obtain more improvements, e.g., about 15.55% average improvement on user profiling prediction tasks, and 4. 1.199/2.919 % average improvement on category preference identification tasks compared with SUMN.
Table 2: Comparison in terms AUC(% (UCCS)) on the Amazon dataset. Several category pref-
erence identification tasks are evaluated. The best score is bold.
Method Books of Literature Games of Sports
Outdoor-hunting
Musical Instrument
TexICNN 73.84/72.83
61.00/69.92
66.06/72.76
68.281 68.28/71.81
HAN 79.12/77.8
66.62/70.01
71.19/73.42
70.01/72.12
TF-IDF
78.62/77.21
65.69768.69
71.90/72.19
69.16/68.62
Doc2Vec
71.21/68.26
66.75/66.88 85
71.58/66.36
70.29/67.37
PTUM
78.66/77.91
66.06/68.97
70.34/73.01
69.50/72.92
SUMN
79.57/77.73
66.83/70.63
72.27/73.44
70.61/73.43
Bol 76.29/75.75 LURM 82.55/79.J3
59.34/67.58
67.80/71.28
66.36/69.88
68.02/72.49
77.59/75.15
74.04/74.43
4.6 ABLATION STUDIES AND DISCUSSION
we have observed that LURM has achieved better performance com- pared other supervieed and supervised methods. Its success can attributed to lifelong user behavior modeling and lossless dimensionality reduction of SMEN.
From Table and Table 3
Fig. 3 shows the performance of LURM on several downstream tasks of industrial dataset when using inputs of different lengths. It can be seen that the richer the behavior, the better the perfor- mance on user profiling prediction tasks. Compared to using only behaviors of two months, an average improvement of 9.75% can be achieved when using behaviors of 5 years these tasks. While the benefits of using richer behavioral data become small for category preference iden- tification tasks. This is because these tasks are more focused the short-term interests of users, and it can be observed that the best performance can be obtained when input with a length one year is used these tasks. We also verified the necessity of multi-scale aggregation module. The
8

---

Under review as a conference paper at ICLR 2022
third and fourth rows of Table 4 show the results of LURM with input of monthly/yearly granularity only, respectively. It can be seen that using multi-scale aggregation module can achieve significant improvements on user profiling prediction tasks, while the benefits disappear category preference tasks. The reason the same as explained in Fig. 3 that is, there no need to model representa- tion with extremely long user behaviors for category preference identification tasks. The last two rows of Show the results of LURM with I anchor LURM(lanchor 128)), and anchor with 1280-dimensional output (eRRM(Ianccor--o
Taking Bol as a benchmark, can be seen that LURM achieves almost lossless dimensionality reduction compared to LURM(1 (1nchor- and (URM((1nchor--) 1280), which verifies the effectiveness of the multi-anchor module.
Table 3: Comparison terms of AUC/ACC on Industrial dataset. Two profiling prediction tasks and three category preference identification tasks evaluated. The best score is bold.
Method
Age
Baby Age
Clothing
Shoe 84.33/80.24
Coffee
TexCCNN
86.40/61.02
72.19/66.01
76.64/76.13
80.49/79.46
TF-IDF
87.73/61.75
73.08/67.26
77.61/78.01
85.04/81.15
81.75/80.78
SUMN
85.35/60.61
74.20/67.63
73.86/74.16
83.47/79.63
79.89/79.42
Bol(short)
89.57/63.99
82.23/70.22
78.93/78.85
85.53/81.68
82.61/81.4
LURM(short)
88.48/61.99
82.71/69.22
78.37/78.57
85.15/81.18
81.94/80.78
Bol
96.48/
06.48/80.11 /80.19 19
93.29/83.57
80.83/79 /7../79..3
86.69 //2. 16
21.69/82.11
83.78 1188/81.2 .74
LURM
96.09/78.43
94.59/84.991
94.59 /84.91
80.37/79.10
86.30/81.66
83.11/81...
Age
Clothing
Coffee
Shoe
Baby
Clothing
Coffee
95%
85%
80%
90%
85%
70%
80%
65%
75%
60%
70%
months
yeur
2 years
years
4 years
years
months
year
2 years
yeurs
years
years
Figure 3: Comparison of LURM with inputs of different lengths on several downstream tasks. The two figures show the results in terms of AUC and ACC respectively.
Table 4: Comparison of LURM with different configurations on industrial dataset.
Method
Age
Baby Age
Clothing
Shoe
Coffee
Bol
96.48/80.19
93.29/83.57
80.83/79.33
86.69/82.16 16
83.78/81.77
LURM
96.09/78.43
94.59/84.91
80.37/79.10
86.30/81.66
83.11/81.18
LURM(monthly)
95.82/77.54
93.19/82.17
80.30/79.02
86.26/81.61
23.08/81.14
LURM(yearly) 128)
95.61/77.19
92.80/80.93
80.24/78.88
16.17/81.36
82.66/80.78
93.98/72.78
86.13/71.22
78.93/78.27
85,80/81.1
81.98/80.39
LURM( Lanchor-
94.35/73.71
89.21/75.35
79.38/78.54
86.07/81.32
82.34/80.59
In addition, we also observed that the gap between Bol and LURM is very different the two datasets. It may be that the item embedding we learned is better on the industrial dataset and then LURM achieved similar results compared with Bol, indicating the ability of SMEN to compress representation losslessly. On the Amazon dataset, the item embedding is worse. And It can be seen that the representation was improved through the further learning of SMEN with self-supervised tasks and the ability to encode temporal variation.
5 CONCLUSION
In this work, a novel framework named LURM proposed to model lifelong user behaviors with any length. With the ability to model lifelong user behaviors, method shows promising results on different downstream tasks and datasets. Although our method has made some progress, there is still space for improvement. In the future research work, will consider more types of tasks; input data in different modalities, such as images, video, and audio; more dedicated network architecture and

---

Under review as a conference paper at ICLR 2022
REFERENCES
Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. Neural news rec- ommendation with long-and short-term user representations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 336-345, 2019.
Nicholas Andrews and Marcus Bishop. Learning invariant representations of social media users. arXiv preprint arXiv: 191.vv1910004999. 2019.
Oren Barkan and Noam Koenigstein. Item2vec: neural item embedding for collaborative filtering- In 2016 IEEE 26th International Workshop Machine Learning for Signal Processing (MLSP),
pp. 1-6. IEEE, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv:1901.02660 2019.
Robin Devooght and Hugues Bersini. Long and short-term recommendations with recurrent neural networks. In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization, pp. 13-21, 2017.
Georges Dupret and Benjamin Piwowarski. A user browsing model to predict search engine click data from past observations. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pp. 331-338, 2008
Ali Mamdouh Elkahky, Yang Song, and Xiaodong He. A multi-view deep learning approach for cross domain user modeling in recommendation systems. In Proceedings of the 24th international conference on world wide web, pp. 278-288, 2015.
Li Fei-Fei and Pietro Perona. A bayesian hierarchical model for learning natural scene cate- gories. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) volume 2, pp. 524-531. IEEE, 2005.
Gerhard Fischer. User modeling in human- computer interaction. User modeling and user-adapted interaction. 11(1):65-86, 2001.
Jie Gu, Feng Wang, Qinghui Sun, Zhiquan Ye, Xiaoxiao Xu, Jingmin Chen, and Jun Zhang, Exploit- ing behavioral consistence for universal user representation. arXiv preprint arXiv:2012.06 146, 2020.
Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, al. Practical lessons from predicting clicks on ads at facebook. In Proceedings of the Eighth International Workshop on Data Mining for Online Advertising, pp- 1-9, 2014.
Wendi Ji, Yinglong Sun, Tingwei Chen, and Xiaoling Wang. Two-stage sequential recommendation via bidirectional attentive behavior embedding and long/short-term integration. In 2020 IEEE International Conference on Knowledge Graph (ICKG), pp. 449-457. IEEE, 2020.
Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. Ask me anything: Dynamic memory networks for natural language processing. In International conference on machine learning, pp. 1378-13877 PMLR, 2016.
Jianmo Ni. Jiacheng Li, and Julian McAuley. Justifying recommendations using distantlyllabeles reviews and fine- aee-graindd aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNL.--JCNIP)) pp. 188-197, 2019.
10

---

Under review as a conference paper at ICLR 2022
Yabo Ni, Dan Ou, Shichen Liu, Xiang Li, Wenwu Ou, Anxiang Zeng, and Luo Si. Perceive your users in depth: Learning universal user representations from multiple e-commerce tasks. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp- 596-605, 2018.
Qi Pi, Weijie Bian, Guorui Zhou, Xiaoqiang Zhu, and Kun Gai. Practice on long sequential user behavior modeling for click- click-through rate prediction. In Proceedings the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2671-26777 2019.
Qi Pi, Guorui Zhou, Yujing Zhang, Zhe Wang, Lejian Ren, Ying Fan, Xiaoqiang Zhu, and Kun Gai. Search-based user interest modeling with lifelong sequential behavior data for click-through rate prediction. In Proceedings of the 29th ACM International Conference Information & Knowledge Management, pp. 2685-2692, 2020.
Kan Ren, Jiarui Qin, Yuchen Fang, Weinan Zhang, Lei Zheng, Weijie Bian, Guorui Zhou, Jian Xu, Yong Yu, Xiaoqiang Zhu, al. Lifelong sequential modeling with personalized memorization for user response prediction. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 565-574, 2019.
Chao-Yuan Wu, Amr Ahmed, Alex Beutel, Alexander J Smola, and How Jing. Recurrent recom- mender networks. In Proceedings of the tenth ACM international conference on web search and data mining. pp. 495-503, 2017.
Chuhan Wu, Fangzhao Wu, Tao Qi, Jianxun Lian, Yongfeng Huang, and Xing Xie. Ptum: Pre-training user model from unlabeled user behaviors via self-supervision. arXiv preprint arXiv:2010.01494, 2020.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. 2019.
Jinfeng Yi, Lijun Zhang, Jun Wang, Rong Jin, and Anil Jain. A single-pass algorithm for effi- ciently recovering sparse cluster centers of high-dimensional data. In International Conference on Machine Learning, pp. 658 666. PMLR, 2014.
Haochao Ying, Fuzhen Zhuang, Fuzheng Zhang, Yanchi Liu, Guandong Xu, Xing Xie, Hui Xiong, and Jian Wu. Sequential recommender system based on hierarchical attention network. In IJCA International Joint Conference on Artificial Intelligence, 2018.
Feng Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. A dynamic recurrent model for next basket recommendation. In Proceedings of the 39th International ACM SIGIR conference Research and Development in Information Retrieval, pp. 729-732, 2016.
Zeping Yu, Jianxun Lian, Ahmad Mahmoody, Gongshen Liu, and Xing Xie. Adaptive user modeling with long and short-term preferences for personalized recommendation. In UICAL, pp. 4213-4219, 2019.
Yu Zhu, Hao Li, Yikang Liao, Beidou Wang, Ziyu Guan, Haifeng Liu, and Deng Cai. What to do next: Modeling user behaviors by time-Istm. IJCAI, volume 17, pp. -600-3608, 2017.


Reviews:
[{"review_id": "R6LyQrh3VIl", "reviewer": "Reviewer_FiJs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Advantages]\n- Modeling users and user behaviors are essential, especially for web-based applications. The problem dealt with in this paper will be a promising application of representation learning based on neural networks.\n- The structure of the proposed method is reasonable, especially for e-commerce applications.\n\n[Drawbacks]\n- I could not understand the novelty of the proposed method. The bag-of-features representation seems to be standard in several research fields such as computer vision, natural language processing, and data mining. The \"multi-anchor module\" seems to be almost the same as the multi-head attention module that is famous in the transformer. The time aggregation module does not contain a special trick. According to the introduction of this paper, the novelty of the proposed method is the ability for modeling life-long behaviors. However, I could not find any special techniques for handling life-long sequences. Please clearly state the technical novelty of the proposed method.\n- The proposed method for modeling user behaviors dismisses both microscopic and macroscopic time-series characteristics, which would be significant for tracing user interest. More specifically, bag-of-interest representations remove microscopic characteristics, and the time aggregation module erases macroscopic features. If the authors believe that I have some misunderstanding, please justify it.\n- When obtaining stable user representations, the knowledge of life-long learning will be required, since (1) users might have several completely different interests in their minds, (2) each interest lasts only a short term, especially in e-commerce applications, and (3) interests containing old behavior sequences might come back suddenly. This implies that almost all the information containing behavior sequences should be kept in user representations and this problem is exactly in the context of life-long learning. Please perform bibliographic surveys for life-long learning, describe the critical problem for the previous work, and present the novelty and advantage(s)  of the proposed method against the previous work.\n- The downstream tasks in the experimental evaluations seem to be too easy and not suitable for evaluating the ability to handle long behavior sequences.", "summary_of_the_paper": "This paper deals with the problem of obtaining user representations from behavior sequences and proposes a method introducing the idea of a \"bag of features\" and multi-head attention. More specifically, a behavior sequence is expressed by a sequence of multi-dimensional vectors and a fixed-length segment of the sequence is converted to a histogram, where each bin corresponds to a cluster of vectors. Each is then fed to a multi-head attention module to obtain a segment representation and all the segment representations are aggregated into a single vector through a time aggregation module. Experimental evaluations for several downstream tasks demonstrate the effectiveness of the proposed method against several simple baselines.", "summary_of_the_review": "I could not find any positive aspects for accepting this paper to the premier conference in the field of representation learning. Both the methodology and documentation need to be improved for acceptance.\n\n[Post-rebuttal comments]\nAfter reading through the paper and author rebuttals, one of the reasons for misunderstanding among us is a gap in the target time duration of \"microscopic\" and \"macroscopic\". My understanding is\n- \"Microscopic\" user behaviors correspond to short sequences of individual actions during single sessions, which last at most a day.\n- From this viewpoint, the proposed BoI representation removes \"microscopic\" information that will be useful for the next user action.\n- Monthly and yearly user behaviors might be \"microscopic\" since a month or a year is shorter than their lifetime.\n- However, a month is sufficiently long to capture whole the interests of users, which implies that monthly behaviors are \"macroscopic\".\n\nI understand that the authors try to model \"users\", not \"user behaviors\", as implied by the downstream tasks. I also understand that \"microscopic\" (by my definition) sequence modeling is not required and only \"macroscopic\" (again by my definition) user behaviors are only of interest.\n\nI have just revised my evaluations according to the above discussion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QPVrYRlBvW", "reviewer": "Reviewer_Rubm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n--Previous work cannot model universal user representation based on lifelong sequences of user behavior since registration and challenging to make full use of highly long user behavior sequences accumulated in Internet service platforms.\n\n-- This work proposes a universal Lifelong User Representation Model to encode user behaviors in any period with a sparse vector with super-high dimension and map these user features to user representations with almost lossless dimensionality reduction. In this way, the model could easily learn different aspects of user preferences in extremely long sequences and generalize to other downstream tasks.\n\nWeaknesses:\n\n-- The structure of related works is not clear enough. At the end of universal user modeling, this work focuses on the conclusion that previous work dedicated to learning universal user representations cannot leverage information from lifelong user behaviors instead of their generalization ability. The same problem also appeared at the end of lifelong user modeling.\n\n-- In a multi-scale aggregation module, this work captures diverse patterns from user behavior by aggregating several representations generated at different time granularities. However, this method could only use information intra-corresponding representation of every time granularity and cannot explore the information inter different time granularities.\n\n-- The explanation of the considerable gap between BoI and LURM on the two datasets is unconvincing. On the industrial dataset, the performance of BOI is better than LURM, which proved that the design of SMEN cannot bring benefits for BOI.\n\n-- In different downstream tasks, the benefits on category preference identification tasks are tiny, even using behaviors of 5 years on this task. There is no explanation about this phenomenon, and it proves BOI cannot learn much preference information about user behaviors on different categories.\n\n-- This work did not design experiments about multi-scale aggregation modules to prove the effectiveness of process representations of different time granularities.\n\n-- There are some format problems in the paper, such as no period at the end of the first paragraph of the introduction.", "summary_of_the_paper": "This work proposes a universal Lifelong User Representation Model to encode user behaviors in any period with a sparse vector with super-high dimension and map these user features to user representations with almost lossless dimensionality reduction. In this way, the model could easily learn different aspects of user preferences in extremely long sequences and generalize to other downstream tasks.", "summary_of_the_review": "This work proposes a universal Lifelong User Representation Model to learn different aspects of user preferences in extremely long sequences and easily generalize to other downstream tasks by the self-supervised method.\n\nHowever, the experimental design of this work needs to be improved, and more explicability needs to be provided about crucial modules and abnormal experimental phenomena, which have more enlightening significance for the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5QlChtfl9EC", "reviewer": "Reviewer_2k8K", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. This paper is easy to understand.\n2. The problem studied in this paper is important.\n3. The proposed multi-scale aggregation method is interesting.\n\nWeaknesses:\n\n1. The contribution of this paper can be somewhat incremental. First, the multi-anchor attention network is also called \"multi-aspect attention\", which has been widely used in many scenarios [1,2,3]. A very similar contrastive model pertaining approach is also studied in [4]. Thus, the contributions of this paper are somewhat incremental.\n2. Discussions and comparisons on universal user modeling methods are insufficient. The authors only include PTUM as the baseline but do not discuss it in Section 2.1. There are also many other methods like [4,5]. They are also not cited nor compared.\n3. The design of the item embedding method is problematic. Different from texts that there are rich relations between adjacent words, even adjacent behaviors can be rather random. Thus, it may not be suitable to learn item embeddings by regarding items as words and user behavior sequences as sentences. There are many methods for item embedding learning such as PinSage [6]. \n4. Compared baselines are also insufficient. As mentioned in the second point, there are many user model pretraining methods that are not compared. In addition, the compared basic user modeling techniques are also rather weak (e.g., CNN, HAN, Doc2Vec and TF-IDF).\n5. Some details are unclear. PTUM and PeterRec require fine-tuning the user model in downstream tasks. However, in this work it is not clear whether the pretrained user model is finetuned.\n6. The selection of some hyperparameters seems to be arbitrary, e.g., the number of anchors and the window size $\\beta$. The authors do not provide experiments to show the influence of several key hyperparameters.\n7. The generality of the proposed method is not verified. It is not clear whether the proposed method can be generalized to other types of user models.\n\n\nMinor:\n\nThere are some typos and grammatical issues. For example, \"committed to learn\" and \"Comparisonn\".\n\n[1] Event detection using hierarchical multi-aspect attention. WWW 2019.\n\n[2] ARP: Aspect-aware neural review rating prediction. CIKM 2019.\n\n[3] Hybrid graph convolutional networks with multi-head attention for location recommendation. World Wide Web 23.6 (2020): 3125-3151.\n\n[4] Contrastive Pre-training for Sequential Recommendation. arXiv e-prints (2020): arXiv-2010.\n\n[5] Parameter-efficient transfer from sequential behaviors for user modeling and recommendation. SIGIR. 2020.\n\n[6] Graph convolutional neural networks for web-scale recommender systems. KDD 2018.", "summary_of_the_paper": "This paper introduces a universal user representation learning approach based on self-supervised learning on long-term user behaviors. The authors first represent user behaviors into sparse vectors, and then encode them into multiple vectors to represent users in different aspects. The authors further study using different time aggregation strategies to tradeoff accuracy and computational cost. Experiments on two datasets show the effectiveness of the proposed approach.", "summary_of_the_review": "This work has some merits but it suffers from several issues that need to be addressed. Thus, my recommendation is a weak rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Ultimately somewhat below the threshold based on the scores. The reviewers raise issues of the overall contribution, as well as issues with the design/structure of the model/paper and issues with the experiments. While there are some positive aspects, collectively the issues put the paper below the bar for acceptance.
Decision: Reject