SELF-SUPERVISED LIFELONG BEHAVIORS MODELING
Year: 2021

Abstract:
Universal user representation is an important research topic in industry, and is widely used in diverse downstream user analysis tasks, such as user profiling and user preference prediction. With the rapid development of Internet service platforms, extremely long user behavior sequences have been accumulated. However, existing researches have little ability to model universal user representation based on lifelong sequences of user behavior since registration. In this study, we propose a novel framework called Lifelong User Representation Model (LURM) to tackle this challenge. Specifically, LURM consists of two cascaded sub-models: (i) Bag of Interests (BoI) encodes user behaviors in any time period into a sparse vector with super-high dimension (e.g., 10); (ii) Self-supervised Multi-anchor Encoder Network (SMEN) maps sequences of BoI features to multiple low-dimensional user representations by contrastive learning. SMEN achieves almost lossless dimensionality reduction, benefiting from a novel multi-anchor module which can learn different aspects of user preferences. Experiments on several benchmark datasets show that our approach outperforms state-of-the-art unsupervised representation methods in downstream tasks.

heading: 1 INTRODUCTION
text: Inferring user intents through user behavior data has been extensively studied in industrial applications, such as recommendation systems, search engines, and online advertising Dupret & Piwowarski (2008); He et al. (2014); Elkahky et al. (2015); Yu et al. (2016). One key aspect in these systems is user modeling, which describes the process of building up and modifying a conceptual understanding of the user Fischer (2001). Essentially, user modeling is to learn a user representation that helps to capture the user’s interests and preferences to improve the performance on downstream tasks.
In the literature, there have been many studies that focused on task-specific user modeling, such as user response prediction and personalized recommendation Ren et al. (2019); Yu et al. (2019); Ji et al. (2020). However, the user representation learned by a specific task can hardly be generalized to other tasks. As a result, specific user representation models need to be trained in each downstream task, which requires massive labeled data, training time, computing and storage resources. Given these limitations, universal user representations that can serve a variety of downstream tasks are preferred.
Due to the sequential form of user behavior data, recurrent neural networks (RNNs) are usually used to encode the temporal dynamics of user behavior sequences Wu et al. (2017); Devooght & Bersini (2017); Zhu et al. (2017); An et al. (2019). Unfortunately, these approaches can only process user behavior sequences with a length of tens or hundreds, while the length can reach hundreds of thousands in many social networks and e-commerce services. Moreover, we have verified by experiments that the performance will become better as the user behavior becomes richer in most downstream tasks (Fig. 3). To solve this problem, many methods drown from natural language processing (NLP) Kumar et al. (2016); Dai et al. (2019); Yang et al. (2019) are proposed. They model long sequential data based on hierarchical architectures and memory networks Ying et al. (2018); Pi et al. (2019); Ren et al. (2019). However, it is still hard for them to encode lifelong user behavior sequences when the length scales up to 1000. And representations they learned through a specific task result in poor generalization capabilities.
In this work, we propose a novel framework called Lifelong User Representation Model (LURM) to model user behaviors since registration. To meet the needs of extremely long sequence modeling, we first introduce a model named Bag of Interests (BoI) to summarize items in behavior sequences similar to Bag of Visual Words. In this way, we can use a sparse vector with super-high dimension to represent user behavior in any time period. Then, a Self-supervised Multi-anchor Encoder Network (SMEN) that maps sequences of BoI features to multiple low-dimensional user representations is proposed. SMEN consists of three modules: a multi-anchor module which can learn different aspects of user preferences, a time aggregation module which can model evolution of user behaviors, and a multi-scale aggregation module which can learn and aggregate BoI features in different scales. Considering the consistency between user behaviors in different time periods, we introduce a contrastive loss function to the self-supervised training of SMEN. With the designs above, SMEN achieves almost lossless dimensionality reduction.
The main contribution of our work can be summarized as follows:
• In this work, a novel framework named LURM is proposed to model lifelong user behaviors of any length. To the best of our knowledge, it is the first method that has the ability to model lifelong behaviors in the field of universal user representation learning.
• We introduce a sub-model named BoI which can encode behaviors in any time period, so that lifelong behavior data can be represented by a sequence of sparse vectors.
• We can obtain compressed user representations with little information loss with the help of a designed sub-model named SMEN.
• Extensive experiments are performed on several real-world datasets. The results demonstrate the effectiveness and generalization ability of the learned user representation.

heading: 2 RELATED WORKS

heading: 2.1 UNIVERSAL USER MODELING
text: Compared with task-specific user modeling that requires more resources, universal user representations are preferred to serve different downstream tasks. In recent years, some works dedicated to learning universal user representations have been proposed. Ni et al. (2018) proposed a representation learning method based on multi-task learning, which enabled the network to generalize universal user representations. Extensive experiments showed the generality and transferability of the user representation. However the effectiveness of this method may still suffer due to the selection of tasks and the need of labels. To release the burden of labeling, Andrews & Bishop (2019) proposed a novel procedure to learn user embedding by using metric learning. They learned a mapping from short episodes of user behaviors to a vector space in which the distance between points captures the similarity of the corresponding users’ invariant features. Gu et al. (2020) proposed a network named self-supervised user modeling network (SUMN) to encode user behavior data into universal representation. They introduced a behavior consistency loss, which guided the model to fully identify and preserve valuable user information under a self-supervised learning framework. Wu et al. (2020) proposed pre-trained user models (PTUM), which can learn universal user models based on two self-supervision tasks for pre-training. The first one was masked behavior prediction, which can model the relatedness between historical behaviors. The second one was next K behavior prediction, which can model the relatedness between past and future behaviors. Unfortunately, these methods can only process user behavior sequences with a length of hundreds, and cannot leverage the rich information brought by lifelong user behaviors.

heading: 2.2 LIFELONG USER MODELING
text: Previous works have shown that considering long-term historical behavior sequences for user modeling can significantly improve the performance of different tasks Ren et al. (2019); Pi et al. (2019; 2020). Ren et al. proposed a hierarchical periodic memory network for lifelong sequential modeling. They built a personalized memorization for each user, which remembers both intrinsic user tastes and multi-facet user interests with the learned while compressed memory. Pi et al. decoupled the user modeling from the whole CTR prediction system to tackle the challenge of the storage cost
and the system latency. Specifically, they proposed a user interest center module for real-time inference and a memory-based network that can be implemented incrementally. Pi et al. also designed a search-based interest model (SIM) with a cascaded two-stage search paradigm to capture the diverse long-term interest with target item. Unfortunately, the length of the user behavior sequence that these models can handle is still limited. Moreover, these models are all trained on specific tasks, which limits the generalization ability.

heading: 3 METHODOLOGY
text: In this work, we are committed to learning universal user representation through truly lifelong user behavior sequences with arbitrary length. For this purpose, we propose a framework named Lifelong User Representation Model (LURM) which consists of two cascaded sub-models: Bag of Interests (BoI) and Self-supervised Multi-anchor Encoder Network (SMEN). The overall architecture of LURM is shown in Fig. 1.

heading: 3.1 BAG OF INTERESTS
text: In order to model extremely long lifelong user behavior sequence, we propose to aggregate the content of items under user purchases, clicks, or other behaviors at a certain granularity. Inspired by Bag of Visual Words (BoVW) Fei-Fei & Perona (2005), item is the natural granularity for aggregation. But there are billions of items on the entire e-commerce platform, which means that the item vocabulary is extremely large, making it infeasible in practice. Therefore, we propose a model called Bag of Interests (BoI) to aggregate user behavior data at ‘interest’ granularity. Every ‘interest’ is a cluster of similar items and represents a certain kind of preference. The size of ‘interest’ vocabulary is often selected at a level of about 105 for retaining enough details. As shown in Fig. 1 (a), BoI consists of an item embedding module and a large-scale clustering module. For convenience, we focus on text modality only in this work. It should be noted that our method can be extended to multi-modal data easily.

heading: 3.1.1 ITEM EMBEDDING MODULE
text: Like the BoVW model, an ‘interest’ vocabulary are supposed to be built in our BoI model. Therefore, the embedding of each item is required, so that similar items with close distance in the embedding space can be clustered together. Recently, in natural language and image processing, dis-
criminative approaches based on contrastive learning in the latent space have shown great success in the field of representation learning, achieving state-of-the-art results. Inspired by these works, we design a contrastive learning task based on the relation between items drawn from a user to learn item embedding similar to item2vec Barkan & Koenigstein (2016).
Given a set of users U = {u1, u2, ..., u|U |}, each user u ∈ U corresponds to a behavior sequence S = { x1,x2, ...,x|S| } , where xi ∈ S denotes the i-th item. |U | and |S| denote the number of users and the length of u’s behaviors respectively. Generally, the content of an item x can be expressed as {w1, w2, ..., w|x|}, where wi denotes a word from a vocabulary V , and |x| denotes the number of words in the content of x. Firstly, an encoder with average operation is used to generate item embedding e:
ex = encoder(w1, w2, ..., w|x|) = proj( 1
|x| |x|∑ i=1 Wi), (1)
where Wi ∈ Rd is the embedding of word wi and will be learned during training, proj(·) includes two residual blocks, and a L2 normalization layer. To construct the contrastive learning task, we sample positive pairs from behavior sequences of users randomly. Specifically, two items (xi,yi) are similar, i.e. a positive pair, if they are drawn from the same user behavior sequence and the time interval between the occurrence of these two items is less than β, where β is the window size controlling the interval of the two user behaviors. Without loss of generality, the sampled mini-batch with batch size n can be denoted as ∆ = {x1,y1,x2,y2, ...,xn,yn}, where (xi,yi) construct a positive pair drawn from the behavior sequence Sit of the i-th user in batch. Then, the contrastive prediction task is defined to identify yi in ∆\{xi} for a given xi, and all other items in ∆\{xi,yi} are negatives. The loss for the positive pair (xi,yi) is written as
l(xi,yi) = − log e g(xi,yi)/τ∑
ν∈∆,ν 6=xi e g(xi,ν)/τ
, (2)
where g(x,y) = e T xey
‖ex‖‖ey‖ = e T xey denotes the cosine similarity between the embedding ex and
the embedding ey , and τ is the temperature parameter. The final objective is the average loss of all positive pairs in the mini-batch, which can be written as
Loss = 1
2n ∑ i (l(xi,yi) + l(yi,xi)). (3)
Noting that, other methods which can be used to generate item embedding are feasible.

heading: 3.1.2 LARGE-SCALE CLUSTERING MODULE
text: After the encoder above has been trained, an item embedding setE = {ei}i∈I is obtained, where I is the complete collection of items at the billion level. In order to retain details as many as possible, the size of the ‘interest’ vocabulary is set to be at 104 ∼ 105 level. In other words, all items should be clustered into D (e.g., 105) categories. Considering the large-scale item set, a subset E′ ⊂ E at the million level is sampled, and an efficient clustering algorithm named HDSC Yi et al. (2014) on E′ is employed to cluster similar items into the same ‘interest’.
After clustering, the cluster centersC make up an ‘interest’ vocabulary. Therefore each item can be attributed to one/multiple ‘interest(s)’ by hard/soft cluster assignment. Take hard cluster assignment as an example, each user can obtain his/her sparsely high-dimensional BoI feature bt ∈ RD in time period t according to his/her behavior sequence St = { x1,x2, ...,x|St| } :
bt = [log(1 + Σ |St| i=1 Ixi∈c1), log(1 + Σ |St| i=1 Ixi∈c2),
..., log(1 + Σ |St| i=1 Ixi∈cD )]
(4)
where xi ∈ cj means item i is assigned to the cluster cj ∈ C, I is an indicator function.

heading: 3.2 SELF-SUPERVISED MULTI-ANCHOR ENCODER NETWORK
text: As mentioned above, a BoI feature bt for each user is obtained, through the BoI model given behavior data in any time period t. Most directly, a user representation with super-high dimension bT can
be obtained by applying the BoI model to the whole lifelong time T . However, there are two main disadvantages: 1) it is not friendly to the downstream tasks since the dimension of representation is too high, and 2) it is too crude to aggregate the information in the whole lifelong time without considering variations over time.
Therefore, we propose to get a BoI feature sequence by applying the BoI model at each time period of the whole lifelong time. Then, a Self-supervised Multi-anchor Encoder Network (SMEN) is designed to learn compressed user representations from the sequence of BoI features.
The lifelong time T is divided into N parts at a fixed time interval, i.e. T = {t1, t2, ..., tN} and ti denotes the i-th time period. In our experiments, the time interval is usually set to be monthly/seasonly/yearly granularity. In this way, a sequence of BoI features B = {bt1 , bt2 , ..., btN } can be obtained, where bti denotes the BoI feature corresponding to the i-th time period ti. After that, SMEN is used to map B to low-dimensional user representations. As shown in Fig. 1 (b), SMEN consists of a multi-anchor module, a time aggregation module, a multiscale aggregation module and a contrastive learning module. Details of the model will be described in the following subsections.

heading: 3.2.1 MULTI-ANCHOR MODULE
text: The data of user behavior, which is highly unstructured and complex, implies different preferences of the user. To capture diverse aspects of user preferences, a novel multi-anchor module is proposed. Specifically, suppose there are M anchors, and each of them indicates a certain preference of users. Let b be a BoI feature, the module converts b to M low-dimensional representations as shown in Fig. 2. Each representation is computed as
ri = ReLU(αTi f(b)) = ReLU( D∑ j αijf(bj)) = ReLU( D∑ j αijbjW e j ), (5)
where f(bj) = bjW ej is the ‘interest’ embedding function, bj is the j-th element of b, W e = (W e1 ,W e 2 , ...,W e D)
T is the embedding matrix. And αij is the attention weight between the i-th anchor and the j-th ‘interest’, which measures the portion assigned to the i-th preference from the j-th behavior ‘interest’. The weight αij is defined as
αij = exp(W ai kj)∑ l exp(W a l kj) , (6)
where W ai is the anchor vector corresponding to the i-th anchor, W a = (W a1 ,W a 2 , ...,W a M )
T
is the anchor matrix, and kj = W pReLU(W ej ) is the interest vector corresponding to the j-th ‘interest’. W e ∈ RD×H , W a ∈ RM×H , and W p ∈ RH×H are learned parameters. ri can be computed efficiently since b is a sparse vector. Due to the different anchor vectors, different attention weights can be generated for each ‘interest’. Finally, a group of different aggregated representations R = {r1, r2, ..., rM} can be obtained (M indicates the total number of anchors). In this way, we
can learn different aspects of user preferences. Specially, experiments prove that SMEN can achieve almost lossless dimensionality reduction mainly due to the multi-anchor module.

heading: 3.2.2 TIME AGGREGATION MODULE
text: Through the multi-anchor module, a sequence of representation groups R = {Rt1 ,Rt2 , ...,RtN } for each user is obtained, where Rti = {r1ti , r 2 ti , ..., r M ti } is a representation group generated by multi-anchor module corresponding to the BoI feature bti in time period ti. In the time aggregation module, each sequence Ri = {rit1 , r i t2 , ..., r i tN } is aggregated separately to yield a new representation r̃i ∈ RH . Thus the variable-size representation sequence R can be transformed into M fixed-size representations R̃ = { r̃1, r̃2, ..., r̃M } .
There are various methods which can be used to aggregate variable-size sequences, such as average/max pooling and RNNs. Compared to average and max pooling, RNNs are more appropriate to capture variations over time. Among all RNN-based models, long short-term memory (LSTM) and gated recurrent units (GRU) are most commonly used. Considering that GRU has fewer parameters and is less computationally intensive, GRU is adopted to the time aggregation module in this work.

heading: 3.2.3 MULTI-SCALE AGGREGATION MODULE
text: Under a time division T = {t1, t2, ..., tN}, we have obtained M representations R̃ ={ r̃1, r̃2, ..., r̃M } through the BoI, multi-anchor module and time aggregation module as introduced above. It’s worth noting that if the time interval is too small, the input sequence of SMEN, i.e. B = {bt1 , bt2 , ..., btN } will become extremely long, which contains extensive details but causes modeling difficulties due to catastrophic forgetting. On the other hand, details may be lost if the time interval is too large.
To address this trade-off, multi-scale aggregation module is designed. The module captures diverse patterns from user behavior by aggregating several representations generated at different granularities. Generally, user behavior in this work is aggregated monthly and yearly, i.e. T = {t1, t2, ..., tN} = { t′1, t ′ 2, ..., t ′ N ′ } (e.g., if the length of time period is 5 years, N is 60 for monthly granularity, and N ′ is 5 for yearly granularity). Thus two sequences of BoI feature B = {bt1 , bt2 , ..., btN } and B′ = { bt′1 , bt ′ 2 , ..., bt′ N′ } at different scales correspondingly are ob-
tained. Then, two groups of representations R̃ = { r̃1, r̃2, ..., r̃M } and R̃′ = { r̃′ 1 , r̃′ 2 , ..., r̃′ M }
are obtained after the multi-anchor module and time aggregation module with parameter sharing. Finally we aggregate R̃ and R̃′ with self-attention as follow:
r̂i = si([r̃i, r̃′ i ]) r̃i + (1− si([r̃i, r̃′i])) r̃′i, (7)
where is the Hadamard product, [·] is a operation concatenating vectors along the last dimension, and si(·) is the switch function which is implemented as a fully-connected layer followed by a sigmoid function. The switch function controls the fusion between BoI feature sequences of different scales. The output of this module R̂ = [r̂1, r̂2, ..., r̂M ] is served as the final user representation.

heading: 3.2.4 CONTRASTIVE LEARNING MODULE
text: As mentioned in 3.1.1, contrastive loss is also used to learn user representation in this network. In order to obtain a unified vector for contrastive learning, a nonlinear projection head is added, i.e. h(·), as used in SimCLR Chen et al. (2020). The function h(·) is computed as
v = ReLU([W h11 r̂ 1,W h12 r̂ 2, ...,W h1M r̂ M ])
h(v) = W h2ReLU(v +mix(v,W h3). (8)
The function mix(v,W h3) is to allow interaction between different preferences, and is implemented as
mix(v,W h3) = r2(W h3r1(v)), (9)
where r1(·) reshapes v to a matrix of size M × H , and r2(·) reshapes W h3r1(v) to a vector. W h1i ∈ RH×H ,W h2 ∈ RMH×MH , andW h3 ∈ RM×M are trainable parameters.
Two behavior sequences drawn from the same user are treated as a positive pair, and behavior sequences from other users are negatives. All sequences are continuous sub-sequences randomly sampled from the whole life time T . Finally, the contrastive loss is defined the same as equation (3).

heading: 4 EXPERIMENTS
text: In this section, we present our experiments in detail.

heading: 4.1 DATASETS
text: Model comparisons were conducted on a public datatset and an industrial dataset. Table 1 shows the statistics of two datasets. The public Amazon dataset Ni et al. (2019) contains product reviews and metadata like product titles and categories from Amazon. For each user, the reviewed product titles constitute a sequence of review behaviors. The dataset for self-supervised user modeling on Amazon DApt was constructed by collecting users who have behaviors between 1997-01 and 2017-12, while taking behaviors occurred between 2018-01 and 2018-10 to form datasets {DAds(t)}t (t refers to tasks) for downstream tasks. Industrial dataset was collected from visit/order logs on a popular ecommerce platform. We constructed a training dataset DIpt for self-supervised user modeling by randomly sampling hundred million of users who have behaviors between 2016-06 and 2021-05, while taking behaviors occurred between 2021-07 and 2021-08 to form datasets {DIds(t)}t.

heading: 4.2 DOWNSTREAM TASKS
text: Two kinds of downstream tasks were used to evaluate our method: category preference identification and user profiling prediction. In the experiments, we randomly selected 80% of the samples from DAds(t)/D I ds(t) for task t to train downstream models and the rest for performance validation.
Category preference identification refers to the task of predicting whether users have short-term preferences in the target category. For Amazon dataset,four categories were included:‘Books of Literature’, ‘Games of Sports’, ‘Outdoor-hunting’, and ‘Musical Instrument’. For the industrial dataset, we considered three categories including clothing, shoe and coffee for evaluation.
User profiling prediction aims to identify user aspects such as gender and age. We only conducted experiments on industrial dataset. Two tasks were involved: (1) user age classification task (age is divided into 6 classes) which predicts the age ranges of users, and (2) baby age classification task (7 class). In both tasks, the ground-truth labels came from an online questionnaire.

heading: 4.3 COMPETITORS
text: We compared the proposed LURM model against a rich set of user representation learning methods, including TF-IDF, Doc2Vec, TextCNN, HAN, PTUM and SUMN. TF-IDF and Doc2Vec view the whole user behavior sequence as a document. TF-IDF generates a sparse high-dimensional vector, while Doc2Vec learns to represent the document by a dense vector. We also compared LURM with two methods named TextCNN and HAN which learn task-specific user representations and classifiers in a supervised manner. Finally, we compared LURM with two newly self-supervised methods for user modeling mentioned before, PTUM and SUMN.

heading: 4.4 TRAINING DETAILS
text: For item embedding learning, the number of words in each item x was truncated. The principle of setting truncation threshold is that 95% data values can be covered by the threshold. In experiments, word2vec and average pooling were used to obtain item embedding on Amazon dataset, while the method described in 3.1.1 was used on the industrial dataset. The entire behavior sequences were
input into SMEN because the length of input sequence only depends on the time interval. On Amazon dataset, TF-IDF and Doc2Vec also took the entire behavior sequence as input. Meanwhile, the length of input of TextCNN, HAN, Doc2Vec, SUMN and PUTM was limited to 50 due to memory limitation. On industrial dataset, all competitors used data of two months.
We set the latent dimensions of word/item embedding and all hidden layers to 128 in LURM. In item embedding module, the window size β was set to 5 days, and the temperature τ was 0.1. In order to retain details as many as possible, the size of the ‘interest’ vocabulary was set to 105, and the number of anchorsM was set to 10 in our experiments. So the final user embedding dimension was 1280. In addition, the input of month-granularity and year-granularity were used. For all user representation models, the Adam optimizer with a learning rate of 0.001, and a batch size of 256 was used for training. The hyper-parameters of the supervised competitors were tuned on the validation set. For downstream tasks, a simple MLP classifiers was applied after the derived user representation. Note that LURM, SUMN and PUTM were all not fine-tuned for downstream tasks in our experiments.

heading: 4.5 RESULTS
text: Table 2 shows the comparison of category preference identification on the public Amazon dataset. The last two rows show the results of our models under different settings. The first one uses the highdimensional output of BoI module as input in the downstream tasks, while the second uses the output of LURM. It can be seen that LURM consistently outperforms other unsupervised methods, e.g., about 3.23%/1.49% average improvements than SUMN and 4.41%/2.09% average improvements than PUTM in terms of AUC and ACC respectively. Even compared with supervised methods, LURM outperforms TextCNN and HAN about 8.26%/3.47% and 3.82%/2.04% respectively. In particular, it can be observed that LURM has a significant improvement over BoI.
Table 3 shows the comparison of two user profiling prediction tasks and three category preference identification tasks on the industrial dataset. In this table, BoI and LURM using 2-month input are compared. When using the same input, LURM(short) achieved better results than other methods on both user profiling prediction tasks and category preference identification tasks. When using longer input, we can obtain more improvements, e.g., about 15.55%/17.55% average improvement on user profiling prediction tasks, and 4.19%/2.91% average improvement on category preference identification tasks compared with SUMN.

heading: 4.6 ABLATION STUDIES AND DISCUSSION
text: From Table 2 and Table 3, we have observed that LURM has achieved better performance compared to other self-supervised and supervised methods. Its success can be attributed to lifelong user behavior modeling and lossless dimensionality reduction of SMEN.
Fig. 3 shows the performance of LURM on several downstream tasks of industrial dataset when using inputs of different lengths. It can be seen that the richer the behavior, the better the performance on user profiling prediction tasks. Compared to using only behaviors of two months, an average improvement of 9.75%/16.03% can be achieved when using behaviors of 5 years on these tasks. While the benefits of using richer behavioral data become small for category preference identification tasks. This is because these tasks are more focused on the short-term interests of users, and it can be observed that the best performance can be obtained when input with a length of one year is used on these tasks. We also verified the necessity of multi-scale aggregation module. The
third and fourth rows of Table 4 show the results of LURM with input of monthly/yearly granularity only, respectively. It can be seen that using multi-scale aggregation module can achieve significant improvements on user profiling prediction tasks, while the benefits disappear on category preference tasks. The reason is the same as explained in Fig. 3, that is, there is no need to model representation with extremely long user behaviors for category preference identification tasks. The last two rows of Table 4 show the results of LURM with 1 anchor (LURM(1anchor-128)), and 1 anchor with 1280-dimensional output (LURM(1anchor-1280)). Taking BoI as a benchmark, it can be seen that LURM achieves almost lossless dimensionality reduction compared to LURM(1anchor-128) and LURM(1anchor-1280), which verifies the effectiveness of the multi-anchor module.
In addition, we also observed that the gap between BoI and LURM is very different on the two datasets. It may be that the item embedding we learned is better on the industrial dataset and then LURM achieved similar results compared with BoI, indicating the ability of SMEN to compress representation losslessly. On the Amazon dataset, the item embedding is worse. And It can be seen that the representation was improved through the further learning of SMEN with self-supervised tasks and the ability to encode temporal variation.

heading: 5 CONCLUSION
text: In this work, a novel framework named LURM is proposed to model lifelong user behaviors with any length. With the ability to model lifelong user behaviors, our method shows promising results on different downstream tasks and datasets. Although our method has made some progress, there is still space for improvement. In the future research work, we will consider more types of tasks; input data in different modalities, such as images, video, and audio; more dedicated network architecture and so on.

Review summaries:
This paper deals with the problem of obtaining user representations from behavior sequences and proposes a method introducing the idea of a "bag of features" and multi-head attention. More specifically, a behavior sequence is expressed by a sequence of multi-dimensional vectors and a fixed-length segment of the sequence is converted to a histogram, where each bin corresponds to a cluster of vectors. Each is then fed to a multi-head attention module to obtain a segment representation and all the segment representations are aggregated into a single vector through a time aggregation module. Experimental evaluations for several downstream tasks demonstrate the effectiveness of the proposed method against several simple baselines.

This paper introduces a universal user representation learning approach based on self-supervised learning on long-term user behaviors. The authors first represent user behaviors into sparse vectors, and then encode them into multiple vectors to represent users in different aspects. The authors further study using different time aggregation strategies to tradeoff accuracy and computational cost. Experiments on two datasets show the effectiveness of the proposed approach.