Interest-based Item Representation Framework for Recommendation with Multi-Interests Capsule Network

1 INTRODUCTION . With the rapid development of deep learning , great achievements have been made in recommendation , such as news recommendation , video recommendation , e-commence and advertisement . For recommendation systems , user interaction behaviors imply single or multi interests of the user , not only items themselves in the sequences . In general , users may have multi interests , e.g. , a user interacts with products from several different categories , including clothes , sports and food . The interests lay below the interactive behaviors which increases the difficult of capturing them directly . For recommendation systems , how to target different users ’ interests is key object . Series of deep learning models on click-through rate ( CTR ) prediction have been proposed . Wide & Deep ( Cheng et al. , 2016 ) jointly trains wide linear models and deep neural networks to combine the benefits of memorization and generalization for recommender systems . PNN , Deep Crossing ( Shan et al. , 2016 ) and DeepFM ( Guo et al. , 2017 ) try to extract low-order and high-order feature extraction by adopting a product layer . DIN ( Zhou et al. , 2018 ) uses attention mechanism to increase the pooling weights of similar items . DIEN ( Zhou et al. , 2019 ) introduces sequential model to build the sequential character instead of using item embedding directly in DIN . DIEN extracts hidden states of GRU as attention input and uses AUGRU to take place of traditional attention model . Generally , deep neural networks depict user interests from previous user-item interactions by utilizing item embedding vectors . To settle the diffusion matter of interests , ( Sabour et al. , 2017 ) proposes dynamic routing capsule network and ( Edraki et al. , 2020 ) successfully achieves the better understanding of relationship between objects than CNNs . Based on dynamic routing of capsule network , ( Li et al. , 2019 ) proposes MIND for dealing with user ’ s diverse interests in retrieval systems by representing one user with multiple vectors encoding the different aspects of the user ’ s interests . However , existing representation learning methods mainly focus on optimizing item-based mechanism between user interactive behavior sequences and candidate item , therefore ignore the impacting of label diffusing into each item in user interactive behavior sequences which weaken the intensity of user interests in back propagation of training model . Besides , with the network layers getting deeper and the dimension of embedding layer becoming larger , new method always need to redesign the model architecture , bring in new dataset or other information . Hence , a method is necessary to enhance the model performance and could be generally used in practice without redesigning whole model architecture or bringing extra information . In this paper , we propose a framework to learn interest-based item representations directly by introducing user Multi Interests Capsule Network ( MICN ) . To make the framework model-agnostic , Multi Interests Capsule Network is designed as an auxiliary task ( ( Pi et al. , 2019 ) introduces a auxiliary task to enhance the model performance ) to jointly learn item-based item representations and interest-based item representations . Interest-based item representation generated by MICN shared with original model takes user diverse interest information in whole model . The contributions of this paper can be summarized as follows : • A new framework to learn interest-based item representations by introducing user Multi Interests Capsule Network ( MICN ) . MICN is designed as auxiliary task and easily integrated with original recommendation model . • The new item representation is generated by concatenating interest-based item representation produced by MICN and item-based item representation produced by original model . • An approach of joint learning method and hyper parameter optimization with MICN framework . Experimental results show the great improvement of different recommendation models on benchmark datasets . We do experiments on different public datasets and compare results between original ranking model ( such as Wide & Deep , DIN , DIEN ) and models with auxiliary MICN . The results demonstrate the framework we proposed has better performance than the original model . 2 MODEL ARCHITECTURE . In recommendation system , i denotes the item id in practice , Iu denotes a set of user interactive behavior sequences ( clicked or viewed item sequences ) by an user u. pu is basic user profile information . it is the candidate item id and rt is related information of candidate item from recommendation system . f ( it , rt ) is the feature function of candidate item information from candidate item it and item related information rt . Usually , user interests is represented to learn the function f ( Iu , pu ) , including user profile pu and user interactive behavior sequences Iu . Hence the user interests can be formulated as Vu = f ( Iu , pu ) ( 1 ) where V = ( v1u , v 2 u , . . . , v K u ) ∈ Rh×K is the representation vector learning from user u information Iu and pu , h is user interest vector dimension and K is number of user vector dimension . Actually , v1u is represent one of user multi interests vector and V is the collection of them . Additionally , candidate item representation is to learn the function f ( it , rt ) of item id and it ’ s information . We can obtain the candidate item embeddings by et = f ( iu , rt ) ( 2 ) where et ∈ Rh is the representation embedding vector learn from target item id it and it ’ s related information rt . In fact , et always is the Embeddings & Pooling layer vector taken from User Multi Interest Capsule Networks which present in next section . Recommendation system is push ‘ good ’ item for a visiting web or app user . Hence , the score of measuring the relationship between candidate item and user interests is necessary . Then the score is define as fscore ( Vu , et ) = e T t v k u ( 3 ) We can obtain a value fscore to measure the distance between user interests and candidate items . Finally , according to the collection of user multi interests , recommendation system will select the top ‘ good ’ items for user . 2.1 USER MULTI INTERESTS CAPSULE NETWORKS . to circumvent some limitations of CNNs , capsules replace scalars with vectors to encode appearance feature representation , allowing better preservation of spatial relationships between whole objects and its parts . They also introduced the dynamic routing mechanism , which allows to weight the contributions of parts to a whole object differently at each inference step . The multi interests of user usually hide in interactive behavior sequences and profile information . Capsules ( Sabour et al. , 2017 ) replace scalars with vectors to encode appearance feature representation by assemble a group of neurons . Dynamic routing of capsule network learn the weight of different capsule which capable of encoding the relationship between the part and the whole . Capsule has better understanding of relationship between objects than CNNs ( Edraki et al. , 2020 ) . In recommendation system , MIND ( Li et al. , 2019 ) automatically capture the high-level multi interests of user through dynamic routing of capsule and achieve good performance in retrieval system of e-commence . Consequently , we propose item representation based on user multi interest capsule networks ( MICN ) to help CTR prediction model promoting their performance . We briefly introduce dynamic ( Behavior to Interest ) routing of capsule for learning the representation of multi interests from user profile information and interactive behavior sequences . For the input of each capsule vj = ‖sj‖2 1 + ‖sj‖2 sj ‖sj‖ ( 4 ) where vj is output and sj is all of input of capsule j. sj = ∑ i cijx̂j|i = ∑ cijWijxj ( 5 ) cij = exp ( bij ) ∑ k exp ( bik ) ( 6 ) Then through the dynamic routing to capture the high-level abstract interests from raw features of user . cij is softmax function for input bij . The behavior to interest ( B2I ) ( Li et al. , 2019 ) adaptively aggregate user ’ s view sequences into multi interests representing vectors . According to the routing logits Equation 3 , the bij is defined as bij = u T j Sei , i ∈ I , j ∈ { 1 , 2 , . . . , K } ( 7 ) where ei ∈ Rh , and vei is one of item i embedding vector of user interactive behavior sequences . uj ∈ Rh , j ∈ { 1 , 2 , . . . , K } the capsule vector of user interest , K is hyper parameter which is the number of user ’ s interests . S ∈ Rh×h the bilinear mapping matrix , link the user ’ s capsule interests and viewed sequences . bij is connection on user ’ s interest and item and keep them in the same vector space mapping . Though capture multi interest capsule vectors from user interactive behavior sequences and profile information , MIND introduced label-aware attention based on scaled dot product to measure the relationship between user ’ s interest and item information . In label aware attention layer , candidate item is query and user interest capsule is key and value , candidate item embedding vector is represented in interest capsule space . Then the scaled dot product formulate as vu = Vusoftmax ( pow ( V T u ei , p ) ) ( 8 ) Consequently , we obtain the probability P ( ei|vu ) and use softmax activate function to select ‘ Good ’ one . The training loss is defined as Lmicn = ∑ u , i logP ( ei|vu ) ( 9 ) where Lmicn is the loss of user multi interests capsule network loss . As match between user ’ s interests and candidate item , the item embeddings vector and user ’ s interest capsule vector have the same vector space which based on the user ’ s interests representation , which is very important for interest-based item representation . 2.2 INTEREST BASED ITEM EMBEDDINGS REPRESENTATION . Embedding representation based on deep learning is of much concern in practice ( Wang et al. , 2020 ) . In recommendation system , each recommending model has its own generating embeddings method . Many works introduced in Section 1 extract multi interests represented by item embeddings through designing network structures . Further , the impact of label diffusing into each item in user interactive behavior sequences weaken the intensity of user interests in back propagation of training model . Though dynamic routing capsule network can partially settle the diffusion matter of interests , specifically integrated with recommending model needing to redesign the model architecture is difficult and not generally used . Inspired by DIEN , auxiliary task play significant role in improving model performance . In order to refrain from redesigning the complex main task model architecture , an auxiliary task is introduced for better item representation learning . Therefore , we propose a model framework of user Multi Interest Capsule Network ( MICN ) as auxiliary task and share interest based item embeddings with main recommendation model task which is Deep Interest Network ( DIN ) . According to the Equation 7 , the scaled dot product , the distance between interest and item , makes item embedding vector is indicated by user interest capsule vectors . Besides , auxiliary task brings item embeddings expressed by user interest capsule in main model by sharing the item embedding vector . In main model , we define the item embeddings compose of two parts : e = eorig ⊕ eaux ( 10 ) where ⊕ is concatenation operator , eorig and eaux are item embedding vector correspondingly in the main target recommending model task and auxiliary task designed as MICN . This framework not only expands original model item embeddings referring user interest capsule , but also keep the original model architecture and still have the original model property . Consequently , the framework can be applied in general recommending model . Because of the item embedding vector of main recommending model is combination of original model item embeddings and auxiliary model item embeddings , original item embeddings is not influenced by auxiliary task and auxiliary item embedding is influence by two task which can be controlled by a hyper parameter . Hence , the total loss of whole model is formulate as L = Lmain + λLmicn ( 11 ) where L is the total loss of the whole model , Lmain is the loss of main model and Lmicn is the loss of user interest capsule network . λ is the hyper parameters which adjust the balance of loss and auxiliary task loss . For the Lmicn , label aware attention layer need positive samples in constructing the loss function . Hence negative samples of label item are masked so that make the loss work when training the whole model . During model training process , the item embedding e will receive two parts of back propagating gradient : main task and auxiliary task . eorig only receive main model gradient∇emain and eaux receive auxiliary model ( MICN ) gradient ∇emain as well as main original model gradient ∇eaux main in order to make eaux fit for main model . The gradient of auxiliary task is update following ∇eaux = ( 1 − ϕ ) ∇eaux aux + ϕ∇eaux main where ϕ ∈ [ 0 , 1 ] . How to choose the suitable hyper parameter ϕ will be discussed in the experiment .

Summary: This paper proposes a user multi-interests capsule network (MICN) to improve better item representation. Because the proposed model is model-agnostic, it can be combined with existing models. The intuition of the proposed model is unclear. The proposed model does not consider the state-of-the-art CTR models. In this sense, I recommend that the authors clearly argue the key novelty of the proposed model and extensive experimental setup.