Year: 2021

Abstract:
Item representation plays an important role for recommendation, such as ecommerce, news, video, etc. It has been used by retrieval and ranking model to capture user-item relationship based on user behaviors. For recommendation systems, user interaction behaviors imply single or multi interests of the user, not only items themselves in the sequences. Existing representation learning methods mainly focus on optimizing item-based mechanism between user interaction sequences and candidate item(especially attention mechanism, sequential modeling). However, item representations learned by these methods lack modeling mechanism to reflect user interests. That is, the methods may be less effective and indirect to capture user interests. We propose a framework to learn interestbased item representations directly by introducing user Multi Interests Capsule Network(MICN). To make the framework model-agnostic, User Multi Interests Capsule Network is designed as an auxiliary task to jointly learn item-based item representations and interest-based item representations. Hence, the generic framework can be easily used to improve existing recommendation models without model redesign. The proposed approach is evaluated on multiple types of benchmarks. Furthermore, we investigate several situations on various deep neural networks, different length of behavior sequences and joint learning ratio of interestbased item representations. Experiment shows a great enhancement on performance of various recommendation models and has also validated our approach. We expect the framework could be widely used for recommendation systems.

heading: 1 INTRODUCTION
text: With the rapid development of deep learning, great achievements have been made in recommendation, such as news recommendation, video recommendation, e-commence and advertisement. For recommendation systems, user interaction behaviors imply single or multi interests of the user, not only items themselves in the sequences. In general, users may have multi interests, e.g., a user interacts with products from several different categories, including clothes, sports and food. The interests lay below the interactive behaviors which increases the difficult of capturing them directly.
For recommendation systems, how to target different users’ interests is key object. Series of deep learning models on click-through rate (CTR) prediction have been proposed. Wide & Deep (Cheng et al., 2016) jointly trains wide linear models and deep neural networks to combine the benefits of memorization and generalization for recommender systems. PNN, Deep Crossing (Shan et al., 2016) and DeepFM (Guo et al., 2017) try to extract low-order and high-order feature extraction by adopting a product layer. DIN(Zhou et al., 2018) uses attention mechanism to increase the pooling weights of similar items. DIEN (Zhou et al., 2019) introduces sequential model to build the sequential character instead of using item embedding directly in DIN. DIEN extracts hidden states of GRU as attention input and uses AUGRU to take place of traditional attention model. Generally, deep neural networks depict user interests from previous user-item interactions by utilizing item embedding vectors.
To settle the diffusion matter of interests, (Sabour et al., 2017) proposes dynamic routing capsule network and (Edraki et al., 2020) successfully achieves the better understanding of relationship between objects than CNNs. Based on dynamic routing of capsule network, (Li et al., 2019) proposes
MIND for dealing with user’s diverse interests in retrieval systems by representing one user with multiple vectors encoding the different aspects of the user’s interests.
However, existing representation learning methods mainly focus on optimizing item-based mechanism between user interactive behavior sequences and candidate item, therefore ignore the impacting of label diffusing into each item in user interactive behavior sequences which weaken the intensity of user interests in back propagation of training model. Besides, with the network layers getting deeper and the dimension of embedding layer becoming larger, new method always need to redesign the model architecture, bring in new dataset or other information. Hence, a method is necessary to enhance the model performance and could be generally used in practice without redesigning whole model architecture or bringing extra information.
In this paper, we propose a framework to learn interest-based item representations directly by introducing user Multi Interests Capsule Network(MICN). To make the framework model-agnostic, Multi Interests Capsule Network is designed as an auxiliary task((Pi et al., 2019) introduces a auxiliary task to enhance the model performance) to jointly learn item-based item representations and interest-based item representations. Interest-based item representation generated by MICN shared with original model takes user diverse interest information in whole model.
The contributions of this paper can be summarized as follows:
• A new framework to learn interest-based item representations by introducing user Multi Interests Capsule Network(MICN). MICN is designed as auxiliary task and easily integrated with original recommendation model.
• The new item representation is generated by concatenating interest-based item representation produced by MICN and item-based item representation produced by original model.
• An approach of joint learning method and hyper parameter optimization with MICN framework. Experimental results show the great improvement of different recommendation models on benchmark datasets.
We do experiments on different public datasets and compare results between original ranking model (such as Wide & Deep, DIN, DIEN) and models with auxiliary MICN. The results demonstrate the framework we proposed has better performance than the original model.

heading: 2 MODEL ARCHITECTURE
text: In recommendation system, i denotes the item id in practice, Iu denotes a set of user interactive behavior sequences (clicked or viewed item sequences) by an user u. pu is basic user profile information. it is the candidate item id and rt is related information of candidate item from recommendation system. f(it, rt) is the feature function of candidate item information from candidate item it and item related information rt.
Usually, user interests is represented to learn the function f(Iu,pu), including user profile pu and user interactive behavior sequences Iu. Hence the user interests can be formulated as
Vu = f(Iu,pu) (1) where V = (v1u, v 2 u, . . . , v K u ) ∈ Rh×K is the representation vector learning from user u information Iu and pu, h is user interest vector dimension and K is number of user vector dimension. Actually, v1u is represent one of user multi interests vector and V is the collection of them.
Additionally, candidate item representation is to learn the function f(it, rt) of item id and it’s information. We can obtain the candidate item embeddings by
et = f(iu, rt) (2) where et ∈ Rh is the representation embedding vector learn from target item id it and it’s related information rt. In fact, et always is the Embeddings & Pooling layer vector taken from User Multi Interest Capsule Networks which present in next section.
Recommendation system is push ‘good’ item for a visiting web or app user. Hence, the score of measuring the relationship between candidate item and user interests is necessary. Then the score is define as
fscore(Vu, et) = e T t v k u (3)
We can obtain a value fscore to measure the distance between user interests and candidate items. Finally, according to the collection of user multi interests, recommendation system will select the top ‘good’ items for user.

heading: 2.1 USER MULTI INTERESTS CAPSULE NETWORKS
text: to circumvent some limitations of CNNs, capsules replace scalars with vectors to encode appearance feature representation, allowing better preservation of spatial relationships between whole objects and its parts. They also introduced the dynamic routing mechanism, which allows to weight the contributions of parts to a whole object differently at each inference step.
The multi interests of user usually hide in interactive behavior sequences and profile information. Capsules(Sabour et al., 2017) replace scalars with vectors to encode appearance feature representation by assemble a group of neurons. Dynamic routing of capsule network learn the weight of different capsule which capable of encoding the relationship between the part and the whole. Capsule has better understanding of relationship between objects than CNNs (Edraki et al., 2020). In recommendation system, MIND (Li et al., 2019) automatically capture the high-level multi interests of user through dynamic routing of capsule and achieve good performance in retrieval system of e-commence. Consequently, we propose item representation based on user multi interest capsule networks(MICN) to help CTR prediction model promoting their performance.
We briefly introduce dynamic (Behavior to Interest) routing of capsule for learning the representation of multi interests from user profile information and interactive behavior sequences. For the input of each capsule
vj = ‖sj‖2 1 + ‖sj‖2 sj ‖sj‖
(4)
where vj is output and sj is all of input of capsule j. sj = ∑ i cijx̂j|i = ∑ cijWijxj (5)
cij = exp (bij)∑ k exp (bik)
(6)
Then through the dynamic routing to capture the high-level abstract interests from raw features of user. cij is softmax function for input bij . The behavior to interest(B2I) (Li et al., 2019) adaptively
aggregate user’s view sequences into multi interests representing vectors. According to the routing logits Equation 3, the bij is defined as
bij = u T j Sei, i ∈ I, j ∈ {1, 2, . . . ,K} (7)
where ei ∈ Rh, and vei is one of item i embedding vector of user interactive behavior sequences. uj ∈ Rh, j ∈ {1, 2, . . . ,K} the capsule vector of user interest, K is hyper parameter which is the number of user’s interests. S ∈ Rh×h the bilinear mapping matrix, link the user’s capsule interests and viewed sequences. bij is connection on user’s interest and item and keep them in the same vector space mapping.
Though capture multi interest capsule vectors from user interactive behavior sequences and profile information, MIND introduced label-aware attention based on scaled dot product to measure the relationship between user’s interest and item information. In label aware attention layer, candidate item is query and user interest capsule is key and value, candidate item embedding vector is represented in interest capsule space. Then the scaled dot product formulate as
vu = Vusoftmax(pow(V T u ei, p)) (8)
Consequently, we obtain the probability P (ei|vu) and use softmax activate function to select ‘Good’ one. The training loss is defined as
Lmicn = ∑ u,i logP (ei|vu) (9)
where Lmicn is the loss of user multi interests capsule network loss. As match between user’s interests and candidate item, the item embeddings vector and user’s interest capsule vector have the same vector space which based on the user’s interests representation, which is very important for interest-based item representation.

heading: 2.2 INTEREST BASED ITEM EMBEDDINGS REPRESENTATION
text: Embedding representation based on deep learning is of much concern in practice(Wang et al., 2020). In recommendation system, each recommending model has its own generating embeddings method. Many works introduced in Section 1 extract multi interests represented by item embeddings through designing network structures. Further, the impact of label diffusing into each item in user interactive behavior sequences weaken the intensity of user interests in back propagation of training model. Though dynamic routing capsule network can partially settle the diffusion matter of interests, specifically integrated with recommending model needing to redesign the model architecture is difficult and not generally used. Inspired by DIEN, auxiliary task play significant role in improving model performance. In order to refrain from redesigning the complex main task model architecture, an auxiliary task is introduced for better item representation learning. Therefore, we propose a model framework of user Multi Interest Capsule Network(MICN) as auxiliary task and share interest based item embeddings with main recommendation model task which is Deep Interest Network(DIN). According to the Equation 7, the scaled dot product, the distance between interest and item, makes item embedding vector is indicated by user interest capsule vectors. Besides, auxiliary task brings item embeddings expressed by user interest capsule in main model by sharing the item embedding vector. In main model, we define the item embeddings compose of two parts:
e = eorig ⊕ eaux (10) where ⊕ is concatenation operator, eorig and eaux are item embedding vector correspondingly in the main target recommending model task and auxiliary task designed as MICN. This framework not only expands original model item embeddings referring user interest capsule, but also keep the original model architecture and still have the original model property. Consequently, the framework can be applied in general recommending model.
Because of the item embedding vector of main recommending model is combination of original model item embeddings and auxiliary model item embeddings, original item embeddings is not influenced by auxiliary task and auxiliary item embedding is influence by two task which can be controlled by a hyper parameter. Hence, the total loss of whole model is formulate as
L = Lmain + λLmicn (11)
where L is the total loss of the whole model, Lmain is the loss of main model and Lmicn is the loss of user interest capsule network. λ is the hyper parameters which adjust the balance of loss and auxiliary task loss.
For the Lmicn, label aware attention layer need positive samples in constructing the loss function. Hence negative samples of label item are masked so that make the loss work when training the whole model. During model training process, the item embedding e will receive two parts of back propagating gradient: main task and auxiliary task. eorig only receive main model gradient∇emain and eaux receive auxiliary model(MICN) gradient ∇emain as well as main original model gradient ∇eaux main in order to make eaux fit for main model. The gradient of auxiliary task is update following ∇eaux = (1 − ϕ)∇eaux aux + ϕ∇eaux main where ϕ ∈ [0, 1]. How to choose the suitable hyper parameter ϕ will be discussed in the experiment.

heading: 3 EXPERIMENTS
text: In this section, Experiments using user multi interest capsule network(MICN) we proposed as auxiliary task conducted on Amazon real world dataset and the results will be showed and discussed.

heading: 3.1 DATASETS
text: We use real world dataset from Amazon Electronics and Books datasets (Ni et al., 2019) to verify our proposed network architecture. Table 1 showed the basic information including user, item and category information.
Amazon Datasets
Amazon Customer Reviews (a.k.a. Product Reviews) is one of Amazon’s iconic datasets. Millions of Amazon customers have committed hundreds million reviews of their opinions and shopping experiences regarding products on the Amazon.com website.
Electronics: Including 192403 consumer reviews for Amazon electronics products like HDMI cables, bluetooth speakers etc, 63001 products covering 801 categories and total 1,689,188 samples. The dataset includes basic product information, category for each product and viewed products list.
Books: List of 603668 consumer reviews for Amazon books, 367,982 books covering 1600 categories and total 603668 samples. The dataset includes basic product information, category and viewed products list.

heading: 3.2 EXPERIMENT AND RESULTS
text: We present experiment of our model framework on the above Amazon dataset Electronics and Books comparing with other widely used model in industrial field.
Wide & Deep(Cheng et al., 2016): combines DNN as Deep part and linear model as Wide part to achieve model generalization and memorization.
DIN(Zhou et al., 2018): uses attentional mechanism modeling relationship between user interactive behavior sequences and candidate item and obtain contribution of each item made in historical visiting sequences to the target item.
DIEN(Zhou et al., 2019): based on DIN model architectures, adds GRUs to modeling the evolutional interests of users with attention.
We implement those model described above on Amazon opened industrial dataset. Then we add an auxiliary task introduced in Section 2 to accomplish multi interests extraction of user and compare the improvement of performance. We set one layer of multi interests extraction layer contains K = [log2 |Iu|] dynamic interest capsules, make the interest-based item embedding vector from MICN shared with main recommendation model and gradient updated by a small ratio ϕ = 0.3. As total loss combines main task loss and auxiliary loss, the weight hyper parameter of loss λ is 1. Optimizer is Adam and learning rate is 1e-4.
Each experiment is repeated 5 times. The mean value and standard deviation of model performance AUC are calculated to measure the model performance. Table 2 showed the performance of different model architecture. In Amazon datasets, maximal length of consumer viewed electronic products list is 10 and maximal length of consumer viewed books is 20. Obviously, the performance of all the model architecture is enhanced by the framework which indeed demonstrates the advantages of MICN as an auxiliary task, meanwhile, the main recommendation model still keep original structure.
Furthermore, with the increase length of viewed items, the framework has better effect on longer interactive behavior sequences. Thus, we conduct the experiment comparing the performance between the framework we proposed and benchmark on different length of user viewed sequences. Table 4 showed the experimental result. DIEN with MICN have better AUC result than original DIEN model. Consequently, for long click sequences of user, user interests capsule network works more efficient and accuracy to extract diverse user interests which conducted different type of experiments to verify.

heading: 4 CONCLUSION AND FUTURE WORKS
text: In this paper, we focus on a generic framework of interest-based item embedding representation to enhance performance of recommendation model. A framework to learn interest-based item representations directly by introducing user multi-interests capsule network was proposed. User multi interest capsule network is designed as an auxiliary task to jointly learn item-based item representations and interest-based item representations. The framework can be easily generalized to recommendation model architecture without extra data or substantial change of model. Besides, the
framework has an effect on longer viewed sequences and more diverse interests. With experimental result on industrial datasets, the framework of user multi interest capsule network as auxiliary task outperforms in the terms of click-throughout rate prediction task. In the future, we will try to extend our work to explainable recommendation of users’ interests through multi interest capsule network.

Review summaries:
This paper studies the item recommendation task. It claims that existing studies ignore the correlation between user interests and candidate items and that they are not easily extendable. To deal with this,  an auxiliary task fulfilled by Multi Interests Capsule Network (MICN) is proposed to extend the existing frameworks. Experiments are conducted via the click-through rate (CTR) prediction. The MICN can improve very slightly on the Amazon books dataset.

This paper introduces an interest-based item representation learning method with a multi-interest capsule network. The authors use an auxiliary task to learn interest-based item representations, which are further combined with item representations learned from their features for final recommendation. Experiments on two domains of the Amazon product dataset and an industrial dataset show that the proposed method outperforms several baseline methods.

This paper proposes a user multi-interests capsule network (MICN) to improve better item representation. Because the proposed model is model-agnostic, it can be combined with existing models. The intuition of the proposed model is unclear. The proposed model does not consider the state-of-the-art CTR models. In this sense, I recommend that the authors clearly argue the key novelty of the proposed model and extensive experimental setup.