Interest-based Item Representation Framework for Recommendation with Multi-Interests Capsule Network

1 INTRODUCTION . With the rapid development of deep learning , great achievements have been made in recommendation , such as news recommendation , video recommendation , e-commence and advertisement . For recommendation systems , user interaction behaviors imply single or multi interests of the user , not only items themselves in the sequences . In general , users may have multi interests , e.g. , a user interacts with products from several different categories , including clothes , sports and food . The interests lay below the interactive behaviors which increases the difficult of capturing them directly . For recommendation systems , how to target different users ’ interests is key object . Series of deep learning models on click-through rate ( CTR ) prediction have been proposed . Wide & Deep ( Cheng et al. , 2016 ) jointly trains wide linear models and deep neural networks to combine the benefits of memorization and generalization for recommender systems . PNN , Deep Crossing ( Shan et al. , 2016 ) and DeepFM ( Guo et al. , 2017 ) try to extract low-order and high-order feature extraction by adopting a product layer . DIN ( Zhou et al. , 2018 ) uses attention mechanism to increase the pooling weights of similar items . DIEN ( Zhou et al. , 2019 ) introduces sequential model to build the sequential character instead of using item embedding directly in DIN . DIEN extracts hidden states of GRU as attention input and uses AUGRU to take place of traditional attention model . Generally , deep neural networks depict user interests from previous user-item interactions by utilizing item embedding vectors . To settle the diffusion matter of interests , ( Sabour et al. , 2017 ) proposes dynamic routing capsule network and ( Edraki et al. , 2020 ) successfully achieves the better understanding of relationship between objects than CNNs . Based on dynamic routing of capsule network , ( Li et al. , 2019 ) proposes MIND for dealing with user ’ s diverse interests in retrieval systems by representing one user with multiple vectors encoding the different aspects of the user ’ s interests . However , existing representation learning methods mainly focus on optimizing item-based mechanism between user interactive behavior sequences and candidate item , therefore ignore the impacting of label diffusing into each item in user interactive behavior sequences which weaken the intensity of user interests in back propagation of training model . Besides , with the network layers getting deeper and the dimension of embedding layer becoming larger , new method always need to redesign the model architecture , bring in new dataset or other information . Hence , a method is necessary to enhance the model performance and could be generally used in practice without redesigning whole model architecture or bringing extra information . In this paper , we propose a framework to learn interest-based item representations directly by introducing user Multi Interests Capsule Network ( MICN ) . To make the framework model-agnostic , Multi Interests Capsule Network is designed as an auxiliary task ( ( Pi et al. , 2019 ) introduces a auxiliary task to enhance the model performance ) to jointly learn item-based item representations and interest-based item representations . Interest-based item representation generated by MICN shared with original model takes user diverse interest information in whole model . The contributions of this paper can be summarized as follows : • A new framework to learn interest-based item representations by introducing user Multi Interests Capsule Network ( MICN ) . MICN is designed as auxiliary task and easily integrated with original recommendation model . • The new item representation is generated by concatenating interest-based item representation produced by MICN and item-based item representation produced by original model . • An approach of joint learning method and hyper parameter optimization with MICN framework . Experimental results show the great improvement of different recommendation models on benchmark datasets . We do experiments on different public datasets and compare results between original ranking model ( such as Wide & Deep , DIN , DIEN ) and models with auxiliary MICN . The results demonstrate the framework we proposed has better performance than the original model . 2 MODEL ARCHITECTURE . In recommendation system , i denotes the item id in practice , Iu denotes a set of user interactive behavior sequences ( clicked or viewed item sequences ) by an user u. pu is basic user profile information . it is the candidate item id and rt is related information of candidate item from recommendation system . f ( it , rt ) is the feature function of candidate item information from candidate item it and item related information rt . Usually , user interests is represented to learn the function f ( Iu , pu ) , including user profile pu and user interactive behavior sequences Iu . Hence the user interests can be formulated as Vu = f ( Iu , pu ) ( 1 ) where V = ( v1u , v 2 u , . . . , v K u ) ∈ Rh×K is the representation vector learning from user u information Iu and pu , h is user interest vector dimension and K is number of user vector dimension . Actually , v1u is represent one of user multi interests vector and V is the collection of them . Additionally , candidate item representation is to learn the function f ( it , rt ) of item id and it ’ s information . We can obtain the candidate item embeddings by et = f ( iu , rt ) ( 2 ) where et ∈ Rh is the representation embedding vector learn from target item id it and it ’ s related information rt . In fact , et always is the Embeddings & Pooling layer vector taken from User Multi Interest Capsule Networks which present in next section . Recommendation system is push ‘ good ’ item for a visiting web or app user . Hence , the score of measuring the relationship between candidate item and user interests is necessary . Then the score is define as fscore ( Vu , et ) = e T t v k u ( 3 ) We can obtain a value fscore to measure the distance between user interests and candidate items . Finally , according to the collection of user multi interests , recommendation system will select the top ‘ good ’ items for user . 2.1 USER MULTI INTERESTS CAPSULE NETWORKS . to circumvent some limitations of CNNs , capsules replace scalars with vectors to encode appearance feature representation , allowing better preservation of spatial relationships between whole objects and its parts . They also introduced the dynamic routing mechanism , which allows to weight the contributions of parts to a whole object differently at each inference step . The multi interests of user usually hide in interactive behavior sequences and profile information . Capsules ( Sabour et al. , 2017 ) replace scalars with vectors to encode appearance feature representation by assemble a group of neurons . Dynamic routing of capsule network learn the weight of different capsule which capable of encoding the relationship between the part and the whole . Capsule has better understanding of relationship between objects than CNNs ( Edraki et al. , 2020 ) . In recommendation system , MIND ( Li et al. , 2019 ) automatically capture the high-level multi interests of user through dynamic routing of capsule and achieve good performance in retrieval system of e-commence . Consequently , we propose item representation based on user multi interest capsule networks ( MICN ) to help CTR prediction model promoting their performance . We briefly introduce dynamic ( Behavior to Interest ) routing of capsule for learning the representation of multi interests from user profile information and interactive behavior sequences . For the input of each capsule vj = ‖sj‖2 1 + ‖sj‖2 sj ‖sj‖ ( 4 ) where vj is output and sj is all of input of capsule j. sj = ∑ i cijx̂j|i = ∑ cijWijxj ( 5 ) cij = exp ( bij ) ∑ k exp ( bik ) ( 6 ) Then through the dynamic routing to capture the high-level abstract interests from raw features of user . cij is softmax function for input bij . The behavior to interest ( B2I ) ( Li et al. , 2019 ) adaptively aggregate user ’ s view sequences into multi interests representing vectors . According to the routing logits Equation 3 , the bij is defined as bij = u T j Sei , i ∈ I , j ∈ { 1 , 2 , . . . , K } ( 7 ) where ei ∈ Rh , and vei is one of item i embedding vector of user interactive behavior sequences . uj ∈ Rh , j ∈ { 1 , 2 , . . . , K } the capsule vector of user interest , K is hyper parameter which is the number of user ’ s interests . S ∈ Rh×h the bilinear mapping matrix , link the user ’ s capsule interests and viewed sequences . bij is connection on user ’ s interest and item and keep them in the same vector space mapping . Though capture multi interest capsule vectors from user interactive behavior sequences and profile information , MIND introduced label-aware attention based on scaled dot product to measure the relationship between user ’ s interest and item information . In label aware attention layer , candidate item is query and user interest capsule is key and value , candidate item embedding vector is represented in interest capsule space . Then the scaled dot product formulate as vu = Vusoftmax ( pow ( V T u ei , p ) ) ( 8 ) Consequently , we obtain the probability P ( ei|vu ) and use softmax activate function to select ‘ Good ’ one . The training loss is defined as Lmicn = ∑ u , i logP ( ei|vu ) ( 9 ) where Lmicn is the loss of user multi interests capsule network loss . As match between user ’ s interests and candidate item , the item embeddings vector and user ’ s interest capsule vector have the same vector space which based on the user ’ s interests representation , which is very important for interest-based item representation . 2.2 INTEREST BASED ITEM EMBEDDINGS REPRESENTATION . Embedding representation based on deep learning is of much concern in practice ( Wang et al. , 2020 ) . In recommendation system , each recommending model has its own generating embeddings method . Many works introduced in Section 1 extract multi interests represented by item embeddings through designing network structures . Further , the impact of label diffusing into each item in user interactive behavior sequences weaken the intensity of user interests in back propagation of training model . Though dynamic routing capsule network can partially settle the diffusion matter of interests , specifically integrated with recommending model needing to redesign the model architecture is difficult and not generally used . Inspired by DIEN , auxiliary task play significant role in improving model performance . In order to refrain from redesigning the complex main task model architecture , an auxiliary task is introduced for better item representation learning . Therefore , we propose a model framework of user Multi Interest Capsule Network ( MICN ) as auxiliary task and share interest based item embeddings with main recommendation model task which is Deep Interest Network ( DIN ) . According to the Equation 7 , the scaled dot product , the distance between interest and item , makes item embedding vector is indicated by user interest capsule vectors . Besides , auxiliary task brings item embeddings expressed by user interest capsule in main model by sharing the item embedding vector . In main model , we define the item embeddings compose of two parts : e = eorig ⊕ eaux ( 10 ) where ⊕ is concatenation operator , eorig and eaux are item embedding vector correspondingly in the main target recommending model task and auxiliary task designed as MICN . This framework not only expands original model item embeddings referring user interest capsule , but also keep the original model architecture and still have the original model property . Consequently , the framework can be applied in general recommending model . Because of the item embedding vector of main recommending model is combination of original model item embeddings and auxiliary model item embeddings , original item embeddings is not influenced by auxiliary task and auxiliary item embedding is influence by two task which can be controlled by a hyper parameter . Hence , the total loss of whole model is formulate as L = Lmain + λLmicn ( 11 ) where L is the total loss of the whole model , Lmain is the loss of main model and Lmicn is the loss of user interest capsule network . λ is the hyper parameters which adjust the balance of loss and auxiliary task loss . For the Lmicn , label aware attention layer need positive samples in constructing the loss function . Hence negative samples of label item are masked so that make the loss work when training the whole model . During model training process , the item embedding e will receive two parts of back propagating gradient : main task and auxiliary task . eorig only receive main model gradient∇emain and eaux receive auxiliary model ( MICN ) gradient ∇emain as well as main original model gradient ∇eaux main in order to make eaux fit for main model . The gradient of auxiliary task is update following ∇eaux = ( 1 − ϕ ) ∇eaux aux + ϕ∇eaux main where ϕ ∈ [ 0 , 1 ] . How to choose the suitable hyper parameter ϕ will be discussed in the experiment .

Summary: This paper introduces an interest-based item representation learning method with a multi-interest capsule network. The authors use an auxiliary task to learn interest-based item representations, which are further combined with item representations learned from their features for final recommendation. Experiments on two domains of the Amazon product dataset and an industrial dataset show that the proposed method outperforms several baseline methods.