Learning Universal User Representations via Self-Supervised Lifelong Behaviors Modeling

1 INTRODUCTION . Inferring user intents through user behavior data has been extensively studied in industrial applications , such as recommendation systems , search engines , and online advertising Dupret & Piwowarski ( 2008 ) ; He et al . ( 2014 ) ; Elkahky et al . ( 2015 ) ; Yu et al . ( 2016 ) . One key aspect in these systems is user modeling , which describes the process of building up and modifying a conceptual understanding of the user Fischer ( 2001 ) . Essentially , user modeling is to learn a user representation that helps to capture the user ’ s interests and preferences to improve the performance on downstream tasks . In the literature , there have been many studies that focused on task-specific user modeling , such as user response prediction and personalized recommendation Ren et al . ( 2019 ) ; Yu et al . ( 2019 ) ; Ji et al . ( 2020 ) . However , the user representation learned by a specific task can hardly be generalized to other tasks . As a result , specific user representation models need to be trained in each downstream task , which requires massive labeled data , training time , computing and storage resources . Given these limitations , universal user representations that can serve a variety of downstream tasks are preferred . Due to the sequential form of user behavior data , recurrent neural networks ( RNNs ) are usually used to encode the temporal dynamics of user behavior sequences Wu et al . ( 2017 ) ; Devooght & Bersini ( 2017 ) ; Zhu et al . ( 2017 ) ; An et al . ( 2019 ) . Unfortunately , these approaches can only process user behavior sequences with a length of tens or hundreds , while the length can reach hundreds of thousands in many social networks and e-commerce services . Moreover , we have verified by experiments that the performance will become better as the user behavior becomes richer in most downstream tasks ( Fig . 3 ) . To solve this problem , many methods drown from natural language processing ( NLP ) Kumar et al . ( 2016 ) ; Dai et al . ( 2019 ) ; Yang et al . ( 2019 ) are proposed . They model long sequential data based on hierarchical architectures and memory networks Ying et al . ( 2018 ) ; Pi et al . ( 2019 ) ; Ren et al . ( 2019 ) . However , it is still hard for them to encode lifelong user behavior sequences when the length scales up to 1000 . And representations they learned through a specific task result in poor generalization capabilities . In this work , we propose a novel framework called Lifelong User Representation Model ( LURM ) to model user behaviors since registration . To meet the needs of extremely long sequence modeling , we first introduce a model named Bag of Interests ( BoI ) to summarize items in behavior sequences similar to Bag of Visual Words . In this way , we can use a sparse vector with super-high dimension to represent user behavior in any time period . Then , a Self-supervised Multi-anchor Encoder Network ( SMEN ) that maps sequences of BoI features to multiple low-dimensional user representations is proposed . SMEN consists of three modules : a multi-anchor module which can learn different aspects of user preferences , a time aggregation module which can model evolution of user behaviors , and a multi-scale aggregation module which can learn and aggregate BoI features in different scales . Considering the consistency between user behaviors in different time periods , we introduce a contrastive loss function to the self-supervised training of SMEN . With the designs above , SMEN achieves almost lossless dimensionality reduction . The main contribution of our work can be summarized as follows : • In this work , a novel framework named LURM is proposed to model lifelong user behaviors of any length . To the best of our knowledge , it is the first method that has the ability to model lifelong behaviors in the field of universal user representation learning . • We introduce a sub-model named BoI which can encode behaviors in any time period , so that lifelong behavior data can be represented by a sequence of sparse vectors . • We can obtain compressed user representations with little information loss with the help of a designed sub-model named SMEN . • Extensive experiments are performed on several real-world datasets . The results demonstrate the effectiveness and generalization ability of the learned user representation . 2 RELATED WORKS . 2.1 UNIVERSAL USER MODELING . Compared with task-specific user modeling that requires more resources , universal user representations are preferred to serve different downstream tasks . In recent years , some works dedicated to learning universal user representations have been proposed . Ni et al . ( 2018 ) proposed a representation learning method based on multi-task learning , which enabled the network to generalize universal user representations . Extensive experiments showed the generality and transferability of the user representation . However the effectiveness of this method may still suffer due to the selection of tasks and the need of labels . To release the burden of labeling , Andrews & Bishop ( 2019 ) proposed a novel procedure to learn user embedding by using metric learning . They learned a mapping from short episodes of user behaviors to a vector space in which the distance between points captures the similarity of the corresponding users ’ invariant features . Gu et al . ( 2020 ) proposed a network named self-supervised user modeling network ( SUMN ) to encode user behavior data into universal representation . They introduced a behavior consistency loss , which guided the model to fully identify and preserve valuable user information under a self-supervised learning framework . Wu et al . ( 2020 ) proposed pre-trained user models ( PTUM ) , which can learn universal user models based on two self-supervision tasks for pre-training . The first one was masked behavior prediction , which can model the relatedness between historical behaviors . The second one was next K behavior prediction , which can model the relatedness between past and future behaviors . Unfortunately , these methods can only process user behavior sequences with a length of hundreds , and can not leverage the rich information brought by lifelong user behaviors . 2.2 LIFELONG USER MODELING . Previous works have shown that considering long-term historical behavior sequences for user modeling can significantly improve the performance of different tasks Ren et al . ( 2019 ) ; Pi et al . ( 2019 ; 2020 ) . Ren et al . proposed a hierarchical periodic memory network for lifelong sequential modeling . They built a personalized memorization for each user , which remembers both intrinsic user tastes and multi-facet user interests with the learned while compressed memory . Pi et al . decoupled the user modeling from the whole CTR prediction system to tackle the challenge of the storage cost and the system latency . Specifically , they proposed a user interest center module for real-time inference and a memory-based network that can be implemented incrementally . Pi et al . also designed a search-based interest model ( SIM ) with a cascaded two-stage search paradigm to capture the diverse long-term interest with target item . Unfortunately , the length of the user behavior sequence that these models can handle is still limited . Moreover , these models are all trained on specific tasks , which limits the generalization ability . 3 METHODOLOGY . In this work , we are committed to learning universal user representation through truly lifelong user behavior sequences with arbitrary length . For this purpose , we propose a framework named Lifelong User Representation Model ( LURM ) which consists of two cascaded sub-models : Bag of Interests ( BoI ) and Self-supervised Multi-anchor Encoder Network ( SMEN ) . The overall architecture of LURM is shown in Fig . 1 . 3.1 BAG OF INTERESTS . In order to model extremely long lifelong user behavior sequence , we propose to aggregate the content of items under user purchases , clicks , or other behaviors at a certain granularity . Inspired by Bag of Visual Words ( BoVW ) Fei-Fei & Perona ( 2005 ) , item is the natural granularity for aggregation . But there are billions of items on the entire e-commerce platform , which means that the item vocabulary is extremely large , making it infeasible in practice . Therefore , we propose a model called Bag of Interests ( BoI ) to aggregate user behavior data at ‘ interest ’ granularity . Every ‘ interest ’ is a cluster of similar items and represents a certain kind of preference . The size of ‘ interest ’ vocabulary is often selected at a level of about 105 for retaining enough details . As shown in Fig . 1 ( a ) , BoI consists of an item embedding module and a large-scale clustering module . For convenience , we focus on text modality only in this work . It should be noted that our method can be extended to multi-modal data easily . 3.1.1 ITEM EMBEDDING MODULE . Like the BoVW model , an ‘ interest ’ vocabulary are supposed to be built in our BoI model . Therefore , the embedding of each item is required , so that similar items with close distance in the embedding space can be clustered together . Recently , in natural language and image processing , dis- criminative approaches based on contrastive learning in the latent space have shown great success in the field of representation learning , achieving state-of-the-art results . Inspired by these works , we design a contrastive learning task based on the relation between items drawn from a user to learn item embedding similar to item2vec Barkan & Koenigstein ( 2016 ) . Given a set of users U = { u1 , u2 , ... , u|U | } , each user u ∈ U corresponds to a behavior sequence S = { x1 , x2 , ... , x|S| } , where xi ∈ S denotes the i-th item . |U | and |S| denote the number of users and the length of u ’ s behaviors respectively . Generally , the content of an item x can be expressed as { w1 , w2 , ... , w|x| } , where wi denotes a word from a vocabulary V , and |x| denotes the number of words in the content of x. Firstly , an encoder with average operation is used to generate item embedding e : ex = encoder ( w1 , w2 , ... , w|x| ) = proj ( 1 |x| |x|∑ i=1 Wi ) , ( 1 ) where Wi ∈ Rd is the embedding of word wi and will be learned during training , proj ( · ) includes two residual blocks , and a L2 normalization layer . To construct the contrastive learning task , we sample positive pairs from behavior sequences of users randomly . Specifically , two items ( xi , yi ) are similar , i.e . a positive pair , if they are drawn from the same user behavior sequence and the time interval between the occurrence of these two items is less than β , where β is the window size controlling the interval of the two user behaviors . Without loss of generality , the sampled mini-batch with batch size n can be denoted as ∆ = { x1 , y1 , x2 , y2 , ... , xn , yn } , where ( xi , yi ) construct a positive pair drawn from the behavior sequence Sit of the i-th user in batch . Then , the contrastive prediction task is defined to identify yi in ∆\ { xi } for a given xi , and all other items in ∆\ { xi , yi } are negatives . The loss for the positive pair ( xi , yi ) is written as l ( xi , yi ) = − log e g ( xi , yi ) /τ∑ ν∈∆ , ν 6=xi e g ( xi , ν ) /τ , ( 2 ) where g ( x , y ) = e T xey ‖ex‖‖ey‖ = e T xey denotes the cosine similarity between the embedding ex and the embedding ey , and τ is the temperature parameter . The final objective is the average loss of all positive pairs in the mini-batch , which can be written as Loss = 1 2n ∑ i ( l ( xi , yi ) + l ( yi , xi ) ) . ( 3 ) Noting that , other methods which can be used to generate item embedding are feasible .

Summary: This paper deals with the problem of obtaining user representations from behavior sequences and proposes a method introducing the idea of a "bag of features" and multi-head attention. More specifically, a behavior sequence is expressed by a sequence of multi-dimensional vectors and a fixed-length segment of the sequence is converted to a histogram, where each bin corresponds to a cluster of vectors. Each is then fed to a multi-head attention module to obtain a segment representation and all the segment representations are aggregated into a single vector through a time aggregation module. Experimental evaluations for several downstream tasks demonstrate the effectiveness of the proposed method against several simple baselines.