Learning Universal User Representations via Self-Supervised Lifelong Behaviors Modeling

1 INTRODUCTION . Inferring user intents through user behavior data has been extensively studied in industrial applications , such as recommendation systems , search engines , and online advertising Dupret & Piwowarski ( 2008 ) ; He et al . ( 2014 ) ; Elkahky et al . ( 2015 ) ; Yu et al . ( 2016 ) . One key aspect in these systems is user modeling , which describes the process of building up and modifying a conceptual understanding of the user Fischer ( 2001 ) . Essentially , user modeling is to learn a user representation that helps to capture the user ’ s interests and preferences to improve the performance on downstream tasks . In the literature , there have been many studies that focused on task-specific user modeling , such as user response prediction and personalized recommendation Ren et al . ( 2019 ) ; Yu et al . ( 2019 ) ; Ji et al . ( 2020 ) . However , the user representation learned by a specific task can hardly be generalized to other tasks . As a result , specific user representation models need to be trained in each downstream task , which requires massive labeled data , training time , computing and storage resources . Given these limitations , universal user representations that can serve a variety of downstream tasks are preferred . Due to the sequential form of user behavior data , recurrent neural networks ( RNNs ) are usually used to encode the temporal dynamics of user behavior sequences Wu et al . ( 2017 ) ; Devooght & Bersini ( 2017 ) ; Zhu et al . ( 2017 ) ; An et al . ( 2019 ) . Unfortunately , these approaches can only process user behavior sequences with a length of tens or hundreds , while the length can reach hundreds of thousands in many social networks and e-commerce services . Moreover , we have verified by experiments that the performance will become better as the user behavior becomes richer in most downstream tasks ( Fig . 3 ) . To solve this problem , many methods drown from natural language processing ( NLP ) Kumar et al . ( 2016 ) ; Dai et al . ( 2019 ) ; Yang et al . ( 2019 ) are proposed . They model long sequential data based on hierarchical architectures and memory networks Ying et al . ( 2018 ) ; Pi et al . ( 2019 ) ; Ren et al . ( 2019 ) . However , it is still hard for them to encode lifelong user behavior sequences when the length scales up to 1000 . And representations they learned through a specific task result in poor generalization capabilities . In this work , we propose a novel framework called Lifelong User Representation Model ( LURM ) to model user behaviors since registration . To meet the needs of extremely long sequence modeling , we first introduce a model named Bag of Interests ( BoI ) to summarize items in behavior sequences similar to Bag of Visual Words . In this way , we can use a sparse vector with super-high dimension to represent user behavior in any time period . Then , a Self-supervised Multi-anchor Encoder Network ( SMEN ) that maps sequences of BoI features to multiple low-dimensional user representations is proposed . SMEN consists of three modules : a multi-anchor module which can learn different aspects of user preferences , a time aggregation module which can model evolution of user behaviors , and a multi-scale aggregation module which can learn and aggregate BoI features in different scales . Considering the consistency between user behaviors in different time periods , we introduce a contrastive loss function to the self-supervised training of SMEN . With the designs above , SMEN achieves almost lossless dimensionality reduction . The main contribution of our work can be summarized as follows : • In this work , a novel framework named LURM is proposed to model lifelong user behaviors of any length . To the best of our knowledge , it is the first method that has the ability to model lifelong behaviors in the field of universal user representation learning . • We introduce a sub-model named BoI which can encode behaviors in any time period , so that lifelong behavior data can be represented by a sequence of sparse vectors . • We can obtain compressed user representations with little information loss with the help of a designed sub-model named SMEN . • Extensive experiments are performed on several real-world datasets . The results demonstrate the effectiveness and generalization ability of the learned user representation . 2 RELATED WORKS . 2.1 UNIVERSAL USER MODELING . Compared with task-specific user modeling that requires more resources , universal user representations are preferred to serve different downstream tasks . In recent years , some works dedicated to learning universal user representations have been proposed . Ni et al . ( 2018 ) proposed a representation learning method based on multi-task learning , which enabled the network to generalize universal user representations . Extensive experiments showed the generality and transferability of the user representation . However the effectiveness of this method may still suffer due to the selection of tasks and the need of labels . To release the burden of labeling , Andrews & Bishop ( 2019 ) proposed a novel procedure to learn user embedding by using metric learning . They learned a mapping from short episodes of user behaviors to a vector space in which the distance between points captures the similarity of the corresponding users ’ invariant features . Gu et al . ( 2020 ) proposed a network named self-supervised user modeling network ( SUMN ) to encode user behavior data into universal representation . They introduced a behavior consistency loss , which guided the model to fully identify and preserve valuable user information under a self-supervised learning framework . Wu et al . ( 2020 ) proposed pre-trained user models ( PTUM ) , which can learn universal user models based on two self-supervision tasks for pre-training . The first one was masked behavior prediction , which can model the relatedness between historical behaviors . The second one was next K behavior prediction , which can model the relatedness between past and future behaviors . Unfortunately , these methods can only process user behavior sequences with a length of hundreds , and can not leverage the rich information brought by lifelong user behaviors . 2.2 LIFELONG USER MODELING . Previous works have shown that considering long-term historical behavior sequences for user modeling can significantly improve the performance of different tasks Ren et al . ( 2019 ) ; Pi et al . ( 2019 ; 2020 ) . Ren et al . proposed a hierarchical periodic memory network for lifelong sequential modeling . They built a personalized memorization for each user , which remembers both intrinsic user tastes and multi-facet user interests with the learned while compressed memory . Pi et al . decoupled the user modeling from the whole CTR prediction system to tackle the challenge of the storage cost and the system latency . Specifically , they proposed a user interest center module for real-time inference and a memory-based network that can be implemented incrementally . Pi et al . also designed a search-based interest model ( SIM ) with a cascaded two-stage search paradigm to capture the diverse long-term interest with target item . Unfortunately , the length of the user behavior sequence that these models can handle is still limited . Moreover , these models are all trained on specific tasks , which limits the generalization ability . 3 METHODOLOGY . In this work , we are committed to learning universal user representation through truly lifelong user behavior sequences with arbitrary length . For this purpose , we propose a framework named Lifelong User Representation Model ( LURM ) which consists of two cascaded sub-models : Bag of Interests ( BoI ) and Self-supervised Multi-anchor Encoder Network ( SMEN ) . The overall architecture of LURM is shown in Fig . 1 . 3.1 BAG OF INTERESTS . In order to model extremely long lifelong user behavior sequence , we propose to aggregate the content of items under user purchases , clicks , or other behaviors at a certain granularity . Inspired by Bag of Visual Words ( BoVW ) Fei-Fei & Perona ( 2005 ) , item is the natural granularity for aggregation . But there are billions of items on the entire e-commerce platform , which means that the item vocabulary is extremely large , making it infeasible in practice . Therefore , we propose a model called Bag of Interests ( BoI ) to aggregate user behavior data at ‘ interest ’ granularity . Every ‘ interest ’ is a cluster of similar items and represents a certain kind of preference . The size of ‘ interest ’ vocabulary is often selected at a level of about 105 for retaining enough details . As shown in Fig . 1 ( a ) , BoI consists of an item embedding module and a large-scale clustering module . For convenience , we focus on text modality only in this work . It should be noted that our method can be extended to multi-modal data easily . 3.1.1 ITEM EMBEDDING MODULE . Like the BoVW model , an ‘ interest ’ vocabulary are supposed to be built in our BoI model . Therefore , the embedding of each item is required , so that similar items with close distance in the embedding space can be clustered together . Recently , in natural language and image processing , dis- criminative approaches based on contrastive learning in the latent space have shown great success in the field of representation learning , achieving state-of-the-art results . Inspired by these works , we design a contrastive learning task based on the relation between items drawn from a user to learn item embedding similar to item2vec Barkan & Koenigstein ( 2016 ) . Given a set of users U = { u1 , u2 , ... , u|U | } , each user u ∈ U corresponds to a behavior sequence S = { x1 , x2 , ... , x|S| } , where xi ∈ S denotes the i-th item . |U | and |S| denote the number of users and the length of u ’ s behaviors respectively . Generally , the content of an item x can be expressed as { w1 , w2 , ... , w|x| } , where wi denotes a word from a vocabulary V , and |x| denotes the number of words in the content of x. Firstly , an encoder with average operation is used to generate item embedding e : ex = encoder ( w1 , w2 , ... , w|x| ) = proj ( 1 |x| |x|∑ i=1 Wi ) , ( 1 ) where Wi ∈ Rd is the embedding of word wi and will be learned during training , proj ( · ) includes two residual blocks , and a L2 normalization layer . To construct the contrastive learning task , we sample positive pairs from behavior sequences of users randomly . Specifically , two items ( xi , yi ) are similar , i.e . a positive pair , if they are drawn from the same user behavior sequence and the time interval between the occurrence of these two items is less than β , where β is the window size controlling the interval of the two user behaviors . Without loss of generality , the sampled mini-batch with batch size n can be denoted as ∆ = { x1 , y1 , x2 , y2 , ... , xn , yn } , where ( xi , yi ) construct a positive pair drawn from the behavior sequence Sit of the i-th user in batch . Then , the contrastive prediction task is defined to identify yi in ∆\ { xi } for a given xi , and all other items in ∆\ { xi , yi } are negatives . The loss for the positive pair ( xi , yi ) is written as l ( xi , yi ) = − log e g ( xi , yi ) /τ∑ ν∈∆ , ν 6=xi e g ( xi , ν ) /τ , ( 2 ) where g ( x , y ) = e T xey ‖ex‖‖ey‖ = e T xey denotes the cosine similarity between the embedding ex and the embedding ey , and τ is the temperature parameter . The final objective is the average loss of all positive pairs in the mini-batch , which can be written as Loss = 1 2n ∑ i ( l ( xi , yi ) + l ( yi , xi ) ) . ( 3 ) Noting that , other methods which can be used to generate item embedding are feasible .

Summary: This paper introduces a universal user representation learning approach based on self-supervised learning on long-term user behaviors. The authors first represent user behaviors into sparse vectors, and then encode them into multiple vectors to represent users in different aspects. The authors further study using different time aggregation strategies to tradeoff accuracy and computational cost. Experiments on two datasets show the effectiveness of the proposed approach.