Interest-based Item Representation Framework for Recommendation with Multi-Interests Capsule Network
Authors: Yanpeng Xie
Tong Zhang
Heng Zhang
Zhendong Qu
Venue: ICLR 2022 Submitted (2022)
Keywords: Feature Representation
Recommendation System
Dynamic Routing of Capsule

Abstract: Item representation plays an important role for recommendation, such as e-commerce, news, video, etc. It has been used by retrieval and ranking model to capture user-item relationship based on user behaviors. For recommendation systems, user interaction behaviors imply single or multi interests of the user, not only items themselves in the sequences. Existing representation learning methods mainly focus on optimizing item-based mechanism between user interaction sequences and candidate item(especially attention mechanism, sequential modeling). However, item representations learned by these methods lack modeling mechanism to reflect user interests. That is, the methods may be less effective and indirect to capture user interests. We propose a framework to learn interest-based item representations directly by introducing user Multi Interests Capsule Network(MICN). To make the framework model-agnostic, user Multi Interests Capsule Network is designed as an auxiliary task to jointly learn item-based item representations and interest-based item representations. Hence, the generic framework can be easily used to improve existing recommendation models without model redesign. The proposed approach is evaluated on multiple types of benchmarks. Furthermore, we investigate several situations on various deep neural networks, different length of behavior sequences and joint learning ratio of interest-based item representations. Experiment shows a great enhancement on performance of various recommendation models and has also validated our approach. We expect the framework could be widely used for recommendation systems.

Under review as a conference paper at ICLR 2022
INTEREST-BASED ITEM REPRESENTATION FRAMEWORK FOR RECOMMENDATION WITH MULTI-INTERESTS CAPSULE NETWORK
Anonymous authors Paper under double-blind review
ABSTRACT
Item representation plays an important role for recommendation, such as e- commerce, news, video, etc. It has been used by retrieval and ranking model to capture user-item relationship based on user behaviors. For recommendation systems, user interaction behaviors imply single or multi interests of the user, not only items themselves in the sequences. Existing representation learning meth- ods mainly focus optimizing item-based mechanism between user interaction sequences and candidate item(especially attention mechanism, sequential mod- eling). However, item representations learned by these methods lack modeling mechanism to reflect user interests. That is, the methods may be less effective and indirect capture user interests. We propose framework learn interest- based item representations directly by introducing user Multi Interests Capsule (MICO),(MICN), To make the framework model-agnostic, User Multi Interests Capsule Network is designed as an auxiliary task to jointly Iearn item based item representations and interest-based item representations. Hence, the generic frame- work can be easily used to improve existing recommendation models without model redesign. The proposed approach is evaluated on multiple types of bench- marks. Furthermore, we investigate several situations various deep neural net- works, different length of behavior sequences and joint learning ratio of interest- based item representations. Experiment shows great enhancement mance of various recommendation models and has also validated our approach. We expect the framework could be widely used for recommendation systems.
perfor-
INTRODUCTION
With the rapid development of deep learning, great achievements have been made recommenda- tion, such news recommendation, video recommendation, e-commence and advertisement. For recommendation systems, user interaction behaviors imply single or multi interests of the user, not only items themselves in the sequences. In general, users may have multi interests, e.g., a user inter- acts with products from several different categories, including clothes, sports and food. The interests lay below the interactive behaviors which increases the difficult of capturing them directly.
For recommendation systems, how to target different users' interests is key object. Series of deep learning models click-through rate (CTR) prediction have been proposed. Wide & Deep (Cheng et al. 2016) jointly trains wide linear models and deep neural networks to combine the benefits of memorization and generalization for recommender systems. PNN, Deep Crossing (Shan al. 2016) and DeepFM (Guo et al. (017)) try to extract low-order and high-order feature extraction by adopting a product layer. DIN et al. 2018) uses attention mechanism increase the pooling weights of similar items. DIEN (Zhou et al. 2019) introduces sequential model to build the sequential character instead of using item embedding directly in DIN. DIEN extracts hidden states of GRU as attention input and uses AUGRU take place of traditional attention model. Generally, deep neural networks depict user interests from previous user-item interactions by utilizing item embedding vectors.
To settle the diffusion matter of interests, (Sabour et al. 2017) proposes dynamic routing capsule network and (Edraki et al. 2020) successfully achieves the better understanding of relationship be- tween objects than CNNs. Based dynamic routing of capsule network, (Li et al. 2019) proposes

---

Under review as a conference paper at ICLR 2022
MIND for dealing with user's diverse interests in retrieval systems by representing one user with multiple vectors encoding the different aspects of the user's interests.
However, existing representation learning methods mainly focus optimizing item-based mecha- nism between user interactive behavior sequences and candidate item, therefore ignore the impacting of label diffusing into each item in user interactive behavior sequences which weaken the intensity of user interests in back propagation of training model. Besides, with the network layers getting deeper and the dimension of embedding layer becoming larger, new method always need to redesign the model architecture, bring in new dataset or other information. Hence, a method is necessary to en- hance the model performance and could be generally used in practice without redesigning whole model architecture or bringing extra information.
In this paper, we propose framework learn interest-based item representations directly by in- troducing user Multi Interests Capsule Network To make the framework model-agnostic, Multi Interests Capsule Network designed as an auxiliary task( et al., 2019) introduces a aux- iliary task to enhance the model performance) to jointly learn item-based item representations and interest- based item representations. Interest-based item representation generated by MICN shared with original model takes user diverse interest information in whole model.
The contributions of this paper can be summarized as follows:
A new framework to learn interest-based item representations by introducing user Multi Interests Capsule MMICO),(MICN), MICN is designed as auxiliary task and easily integrated with original recommendation model.
The new item representation is generated by concatenating interest-basee based item representation pro- duced by MICN and item-baser item representation produced by original model.
An approach of joint learning method and hyper parameter optimization with MICN framework. Experimental results show the great improvement of different recommendation models on bench- mark datasets.
We do experiments different public datasets and compare results between original ranking model (such as Wide & Deep, DIN, DIEN) and models with auxiliary MICN. The results demonstrate the framework we proposed has better performance than the original model.
2 MODEL ARCHITECTURE
In recommendation system, denotes the item id in practice, Th denotes a set of user interactive behavior sequences (clicked or viewed item sequences) by u. Ps. is basic user profile infor- mation. i; is the candidate item id and r, is related information of candidate item from recommen- dation system. f(e,, rz) is the feature function of candidate item information from candidate item & and item related information Tt.
Usually, user interests is represented to learn the function fIu: Pu), including user profile Pa and user interactive behavior sequences Ig. Hence the user interests can be formulated as
V, (Iu. Pu)
where V (w u I, and Pu. h user interest vector dimension and K number of user vector dimension. Actually, vy is represent
<
TRAXK is the representation vector learning from user information
of user multi interests vector and V is the collection of them.
Additionally, candidate item representation to learn the function f(é,, rt) of item id and it's infor- mation. We can obtain the candidate item embeddings by
e, = f(u, r:)
where e, Rath is the representation embedding vector learn from target item id i; and it's related information ry. fact, e, always is the Embeddings Pooling layer vector taken from User Multi Interest Capsule Networks which present in next section.
Recommendation system is push 'good' item for a visiting web or app user. Hence, the score of measuring the relationship between candidate item and user interests is necessary. Then the score is define as
S Sscore CaaneeVin (vs, ee) e7 v
2

---

Under review as a conference paper at ICLR 2022
We can obtain value Secore to measure the distance between user interests and candidate items. Finally, according to the collection user multi interests, recommendation system will select the top "good' items for user.
User MUB vereel Copruk Notwert
Man Midto CaN
Figure 1: Model architecture user Multi Interest Capsule NetCork(MICNN) as auxiliary task and share item embeddings with main model task(DIN). MICN takes user interactive behavior sequences and user profile into dynamic routing capsule and extracts user interests as capsules. Scaled dot product between interest capsule and candidate item embeddings makes them in the sentation space. In label aware attention layer. The item embedding vector combined with item embeddings of original recommending model brings in main model. The main model is designed suitable for specific task, such as ranking model etc. DIN model structure showed as an example and item embeddings Corig has extended with interest- based item embeddings Eour- Then the input embeddings of main model is e in Equation TO The MICN helps main task promoting the performance without change the model original architecture or bring in other dataset or information.
repre-
diverse interest-based item representation
2.1 USER MULTI INTERESTS CAPSULE NETWORKS
to circumvent some limitations of CNNs, capsules replace scalars with vectors to encode appearance feature representation, allowing better preservation of spatial relationships between whole objects and its parts. They also introduced the dynamic routing mechanism, which allows to weight the contributions of parts to whole object differently at each inference step.
The multi interests of user usually hide in interactive behavior sequences and profile information. Capsules(Sabour al., 2017) replace scalars with vectors encode appearance feature represen- tation by assemble a group of neurons. Dynamic routing of capsule network learn the weight of different capsule which capable of encoding the relationship between the part and the whole. Cap- sule has better understanding of relationship between objects than CNNs (Edraki et al. 2020). In recommendation system, MIND (Li al., 2019) automatically capture the high-level multi interests of user through dynamic routing of capsule and achieve good performance in retrieval system of e-commence. Consequently, we propose item representation based on user multi interest capsule networks(MICN) to help CTR prediction model promoting their performance.
We briefly introduce dynamic (Behavior to Interest) routing of capsule for learning the represen- tation of multi interests from user profile information and interactive behavior sequences. For the input of each capsule
|8
8,
(4)
1+ |,,
|s,||
where vy is output and 8, is all of input of capsule j.
8j = Σ
ciju = Σ Cij Wiy
(5)
exp (bxs)
Gj Σn exp (bsz)
(6)
Then through the dynamic routing to capture the high-level abstract interests from raw features of user. Cij is softmax function for input byj. The behavior interest(B21) (Li et al. 2019) adaptively
3

---

Under review as a conference paper at ICLR 2022
aggregate user's view sequences into multi interests representing vectors. According to the routing logits Equation the by is defined as
bi = u Se,
i 21,
where e; E R2 and ve, is one of item embedding vector of user interactive behavior sequences. My E RR. j (1, K} the capsule vector of user interest, K is hyper parameter which is the number of user's interests. S E Raxh the bilinear mapping matrix, link the user's capsule interests and viewed sequences, bij is connection on user's interest and item and keep them in the same vector space mapping.
Though capture multi interest capsule vectors from user interactive behavior sequences and pro- file information, MIND) introduced label-aware attention based on scaled dot product measure the relationship between user's interest and item information. In label aware attention layer, candi- date item is query and user interest capsule key and value, candidate item embedding vector is represented in interest capsule space. Then the scaled dot product formulate as
vy = V.softmax(povu( p),
(8)
Consequently, we obtain the probability P(e;(e,) and use softmax activate function to select 'Good' one. The training loss is defined as
Lmnien
Σ M,
log P(e,lo..)
where Liiion is the loss of user multi interests capsule network loss. As match between user's interests and candidate item, the item embeddings vector and user's interest capsule vector have the same vector space which based on the user's interests representation, which is very important for interest- based item representation.
2.2 INTEREST BASED ITEM EMBEDDINGS REPRESENTATION
Embedding representation based on deep learning is of much concern in practice Wang et al. 2020). In recommendation system, each recommending model has its own generating embeddings method. Many works introduced in Section [ exxtract multi interests represented by item embeddings through designing network structures. Further, the impact of label diffusing into each item in user interac- tive behavior sequences weaken the intensity of user interests in back propagation of training model. Though dynamic routing capsule network can partially settle the diffusion matter of interests, specif- ically integrated with recommending model needing to redesign the model architecture is difficult and not generally used. Inspired by DIEN, auxiliary task play significant role in improving model performance. In order to refrain from redesigning the complex main task model architecture, an auxiliary task is introduced for better item representation learning. Therefore, we propose a model framework of user Multi Interest Capsule NeIC)rk(MIIN) as auxiliary task and share interest based item embeddings with main recommendation model task which is Deep Interest NetWork(DIN). Ac- cording to the Equation 7 the scaled dot product, the distance between interest and item, makes item embedding vector is indicated by user interest capsule vectors. Besides, auxiliary task brings item embeddings expressed by user interest capsule in main model by sharing the item embedding vector. In main model, we define the item embeddings compose of two parts:
e Corig Canz
(10)
where e is concatenation operator, Corig and Caur are item embedding vector correspondingly in the main target recommending model task and auxiliary task designed as MICN. This framework not only expands original model item embeddings referring user interest capsule, but also keep the original model architecture and still have the original model property. Consequently, the framework can be applied in general recommending model.
Because of the item embedding vector of main recommending model is combination of original model item embeddings and auxiliary model item embeddings, original item embeddings is not influenced by auxiliary task and auxiliary item embedding influence by two task which can be controlled by a hyper parameter. Hence, the total loss of whole model is formulate as
L Temain + ATmiin
(11)
4

---

Under review as a conference paper at ICLR 2022
Table I: Experimental Datasets
Datasets
Users
Items
Categories
Samples
Books
603,668
367,982
1,600
603,668
Electronics
192,403
63,001
801
192,403
Table 2: Experimental Results(AUC) on Industrial Datasets
Model
Electronics
Books
Deee&Deep
0.7461
0.0015
0.7860 + 0.0013
Wide& Deep-MICN
0.7502
0.0010
0.7928 + 0.0009
DIN
0.7569 = 0.0009
0.7970 = 0.0010
DIN-MICN
0.7606 = 0.0013
0.8002 = 0.0009
DIEN
0.7706 0.0021
0.8534
0.0018
DIEN-MICN
0.7723 = 0.0002
0.8633 + 0.0019
where is the total loss of the whole model, Lmaii is the loss of main model and Linien is the loss of user interest capsule network. À is the hyper parameters which adjust the balance of loss and auxiliary task loss.
For the Lmsion label aware attention layer need positive samples in constructing the loss function. Hence negative samples of label item are masked so that make the loss work when training the whole model. During model training process, the item embedding e will receive two parts of back propagating gradient: main task and auxiliary task. Corig only receive main model gradient Vemain and Cauz receive auxiliary model(MICN) gradient Vemain as well as main original model gradient Veauzzmoin in order to make Eauz fit for main model. The gradient of auxiliary task update following Veaur (1 Veur.aau + VVeausmman where [0, 1]. How to choose the suitable hyper parameter yo will be discussed in the experiment.
EXPERIMENTS
In this section, Experiments using user multi interest capsule netWork(MICN) we proposed as aux- iliary task conducted on Amazon real world dataset and the results will be showed and discussed.
3.1 DATASETS
We use real world dataset from Amazon Electronics and Books datasets (Ni et al. 2019) to verify our proposed network architecture. Table showed the basic information including user, item and category information.
Amazon Datasets
Amazon Customer Reviews (a.k.a. Product Reviews) one of Amazon's 's iconic datasets. Millions of Amazon customers have committed hundreds million reviews of their opinions and shopping experiences regarding products on the Amazon.com website.
Electronics: Including 192403 consumer reviews for Amazon electronics products like HDMI ca- bles, bluetooth speakers etc, 63001 products covering 801 categories and total 1,689, 188 samples. The dataset includes basic product information, category for each product and viewed products list.
Books: List of 603668 consumer reviews for Amazon books, 367,982 books covering 1600 cat- egories and total 603668 samples. The dataset includes basic product information, category and viewed products list.
3.2 EXPERIMENT AND RESULTS
We present experiment of our model framework on the above Amazon dataset Electronics and Books comparing with other widely used model in industrial field.
5

---

Under review as a conference paper at ICLR 2022
Table 3: Hyper parameter yp in DIEN with multi interest capsule network auxiliary task Ama- zon books dataset.
AUC
AUC
0.1
0.8620 + 0.0018
0.6
0.8606 + 0.0014
0.2
0.8636
0.0019
0.7
0.8625 + 0.0010
0.3
0.8638 = 0.0005
0.8
0.8624 + 0,0018
0.4
0.8636
0.0013
0.9
0.8621 + 0.0012
0.5
0.8606 + 0.0012
1.0
0.8602 + 0.0022
Table The AUC performance of MICN framework with DIEN ranking model on various length of user interactive behavior sequences on Amazon books dataset.
length DIEN
10
20
50
0.8543 = 0.0016
0,8534 + 0.0018
0.8505 + 0.0022
DIEN-MICN
0.8571 = 0.0018
0.8633 + 0.0019
0.8613 0.0014
Wide & Deep(Cheng et al. 2016): combines DNN Deep part and linear model as Wide part to achieve model generalization and memorization.
Zhou et al. 2018): uses attentional mechanism modeling relationship between user interac- tive behavior sequences and candidate item and obtain contribution of each item made in historical visiting sequences to the target item.
DZho et al. 2019): based on DIN model architectures, adds GRUs to modeling the evolutional interests of users with attention.
We implement those model described above on Amazon opened industrial dataset. Then we add an auxiliary task introduced in Section 2 to accomplish multi interests extraction of user and compare the improvement performance. We set one layer of multi interests extraction layer contains [log2 [r]]]] dynamic interest capsules, make the interest-based item embedding vector from MICN shared with main recommendation model and gradient updated by small ratio & 0.3. As total loss combines main task loss and auxiliary loss, the weight hyper parameter of loss is Optimizer is Adam and learning rate is lc-4.
Each experiment is repeated times. The mean value and standard deviation of model performance AUC are calculated to measure the model performance. Tableeshoowed the performance of different model architecture.
Amazon datasets, maximal length consumer viewed electronic products list is 10 and maximal length of consumer viewed books is 20. Obviously, the performance of all model architecture enhanced by the framework which indeed demonstrates the advantages of MICN as auxiliary task, meanwhile, the main recommendation model still keep original structure.
Furthermore, with the increase length of viewed items, the framework has better effect on longer interactive behavior sequences. Thus, conduct the experiment comparing the performance be- tween the framework we proposed and benchmark on different length of user viewed sequences. Table showed the experimental result. DIEN with MICN have better AUC result than original DIEN model. Consequently, for long click sequences of user, user interests capsule network works more efficient and accuracy extract diverse user interests which conducted different type of periments to verify.
CONCLUSION AND FUTURE WORKS
In this paper, we focus on a generic framework of interest-based item embedding representation to enhance performance of recommendation model. A framework to learn interest-based item repre- sentations directly by introducing user multi-interests capsule network was proposed. User multi interest capsule network is designed tations and sntrest-bbsed item representations. The framework can easily generalized to rec- ommendation model architecture without extra data substantial change of model. Besides, the
auxiliary task to jointly learn item-based item represen-

---

Under review as a conference paper at ICLR 2022
framework has an effect on longer viewed sequences and more diverse interests. With experimental result on industrial datasets, the framework of user multi interest capsule network as auxiliary task outperforms in the terms of click-throughout rate prediction task. In the future, we will try to extend our work to explainable recommendation of users' interests through multi interest capsule network.
REFERENCES
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, al. Wide & deep learning for recom- mender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, systems, pp. 7-10, 2016.
Marzieh Edraki, Nazanin Rahnavard, and Mubarak Shah. Subspace capsule network. Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 10745-10753, 2020.
Huifeng Guo, Ruiming Tang. Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfin: factorization- machine based neural network for ctr prediction. arXiv preprini arXiv: 1703.04277, 2017.
Chao Li, Zhiyuan Liu, Mengmeng Wu, Yuchi Xu, Huan Zhao, Pipei Huang, Guoliang Kang, Qiwei Chen, Wei Li, and Dik Lun Lee. Multi-interee network with dynamic routing for recommen- dation at tmall. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 2615-2623, 2019.
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labeled reviews and fine-grate fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 188-197, 2019.
Qi Pi, Weijie Bian, Guorui Zhou, Xiaoqiang Zhu, and Kun Gai. Practice long sequential user behavior modeling for click-through rate prediction. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2671-2679, 2019.
Sara Sabour, Nicholas Frosst, and Geoffrey Hinton. Dynamic routing between capsules. arXiv
preprint arXiv: 17X1v:1710.09829, 2017.
Ying Shan, T Ryan Hoens, Jian Jiao, Haijing Wang, Dong Yu, and JC Mao. Deep crossing: Web- scale modeling without manually crafted combinatorial features. In Proceedings of the 22nd ACM SIGKDD international conference knowledge discovery and data mining, pp. 255-262, 2016.
Hui Wang, Zichun Le, and Xuan Gong. Recommendation system based on heterogeneous feature: A survey. IEEE Access, 8: 1777799-177799, 2020. doi: 10. A091100SCCESS.2220..0 2000.302144,
Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp- 1059-1068, 2018
Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. Deep interest evolution network for thick-through prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 5941-5948, 2019.


Reviews:
[{"review_id": "rmKQ-voUjqt", "reviewer": "Reviewer_P2Vg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. Novel model to jointly learn of item-based and interest-based representation of the item for recommended systems\n2. Practical application in recommended systems with real datasets.\n\nWeaknesses:\n1. Combination of the state-of-the-art models i.e., Capsules(Sabour et al., 2017) and MIND (Li et al., 2019), etc.\n2. There is no theoretical justification to support the advancement of the proposed model compared to the previous techniques.\n3. There should be more empirical results to support the claims. It would be more interesting to see the real recommendation rather than the AUC (yet another :)). ", "summary_of_the_paper": "The paper proposes an interest-based item embedding representation to enhance the performance of recommendation model by jointly learning iitem-based item representations and interest-based item representations.", "summary_of_the_review": "As Above", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fdNBTmdwM_", "reviewer": "Reviewer_KquS", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThe authors conduct experiments on multiple datasets.\n\nWeaknesses:\n1. The motivation of this work is problematic or unconvincing. It is not clear why the auxiliary interest-based item representations are necessary because they can be inherently encoded by item-based user interest models. The authors do not provide evidence on this point. In addition, it is not clear why the training task is also the recommendation task. It seems that the proposed method is in fact a kind of ensemble.\n2. The contribution of this paper is also marginal. The authors directly adopt the off-the-shelf capsule network MIND for multi-interest modeling. There are also many other existing multi-interest modeling works [1-3]. The authors do not cite nor compare any other multi-interest modeling methods. \n3. The evaluation is rather flawed. The authors only compare the results of adding MICN to several methods. However, there are many methods that aim to enhance item embeddings in existing literature (like [1]). In addition, the authors do not evaluate the effectiveness of the MICN method. Thus, it is not clear whether the improvement is brought by the multi-interest architecture or simply brought by the increase of model complexity. \n4. Many important details are missing. For example, many hyperparameters in this work are not included, and the methods for constructing training and evaluation samples are not introduced.\n5. The coverage of related work is too limited. The authors should track the recent advances in the recommender system community.\n6. The writing and presentation also need improvement. The motivation part needs revisions most. Many sentences are not clear. The experimental discussions are uninformative. There are many typos and grammatical errors. The fonts in figure 1 are too small.\n\n\n[1] Pal, Aditya, et al. \"PinnerSage: multi-modal user embedding framework for recommendations at pinterest.\" Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020.\n[2] Qi, Tao, et al. \"HieRec: Hierarchical User Interest Modeling for Personalized News Recommendation.\" arXiv preprint arXiv:2106.04408 (2021).\n[3] Liu, Zheng, et al. \"Octopus: Comprehensive and Elastic User Representation for the Generation of Recommendation Candidates.\" Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020.", "summary_of_the_paper": "This paper introduces an interest-based item representation learning method with a multi-interest capsule network. The authors use an auxiliary task to learn interest-based item representations, which are further combined with item representations learned from their features for final recommendation. Experiments on two domains of the Amazon product dataset and an industrial dataset show that the proposed method outperforms several baseline methods.", "summary_of_the_review": "The motivation of this work is not clear. This paper lacks sufficient technical contribution and the evaluation is also not thorough. The coverage of related work is quite limited. The writing also needs improvement. Thus, my recommendation is a strong rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cDMefmXhZmG", "reviewer": "Reviewer_wjxV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Important task: This paper focuses on dealing with CTR prediction, which is a very practical task that highly relates to profit in the industry.\n- Well-organized: This paper contains basic elements that are essential to a research paper.\n\nWeaknesses：\n- contribution: The main framework MICN proposed in this paper serves as an extension to existing frameworks. However, MICN is mostly borrowed from a previous work called MIND (Li et al., 2019), and MIND can not be regarded as the main contribution of this submission.\n- Motivation is not clear: It is not clear the reason this paper uses capsule networks. Why are feedforward networks not able to serve the same purpose?\n- Weak baselines: The baselines included in the experiment part are not very strong. Since CTR prediction is a well-studied problem, I suggest the authors review more strong related works, such as [1] et al.\n- Case studies missing: It would be interesting for the reviewer to see more details in the experiments, such as sensitivity w.r.t. Gamma, some visualization of the learned correlation between user interests and candidate items, \nWriting can be improved: I find it hard to understand some sentences when reviewing the draft. I suggest the authors proofread the draft more and fix all grammar issues and typos.\n\nReferences: \n- [1] Fei, Hongliang, et al. \"GemNN: Gating-enhanced Multi-task Neural Networks with Feature Interaction Learning for CTR Prediction.\" Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2021.\n", "summary_of_the_paper": "This paper studies the item recommendation task. It claims that existing studies ignore the correlation between user interests and candidate items and that they are not easily extendable. To deal with this,  an auxiliary task fulfilled by Multi Interests Capsule Network (MICN) is proposed to extend the existing frameworks. Experiments are conducted via the click-through rate (CTR) prediction. The MICN can improve very slightly on the Amazon books dataset.", "summary_of_the_review": "This paper studies a practical yet well-studied problem by applying existing techniques. The reviewer finds the paper well-organized. However, this paper has limited contribution, insufficient experiments, and missing details. These major drawbacks prevent the reviewer from championing accepting the paper.  In summary, I suggest rejecting the paper at ICLR and that the authors make major revisions until their next submission.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3QWYWqlaZYg", "reviewer": "Reviewer_LCWN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points\n\nS2. This paper is easy to understand.\n\nS3. The model architecture is simple and easy to understand.\n\nWeak points\n\nW1. The baseline models are not state-of-the-art models. I recommend that the authors consider recent publications for the CTR problem as below.\n-\t[A] Yuan Cheng, Yanbo Xue, Looking at CTR Prediction Again: Is Attention All You Need? SIGIR 2021\n-\t[B] Weiyu Cheng, Yanyan Shen, Linpeng Huang, Adaptive Factorization Network: Learning Adaptive-Order Feature Interactions, AAAI 2020\n-\t[C] Ruoxi Wang, Rakesh Shivanna, Derek Z. Cheng, Sagar Jain, Dong Lin, Lichan Hong, Ed H. Chi, DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems, WWW 2021 \n\nW2. It should be more explanation for experimental results. The authors should analyze the meaning of experimental results.\n\nW3. The technical novelty is incremental and unclear.\n\nMinor issue.\n-\tFigure 1 is too small to recognize.\n", "summary_of_the_paper": "This paper proposes a user multi-interests capsule network (MICN) to improve better item representation. Because the proposed model is model-agnostic, it can be combined with existing models. The intuition of the proposed model is unclear. The proposed model does not consider the state-of-the-art CTR models. In this sense, I recommend that the authors clearly argue the key novelty of the proposed model and extensive experimental setup.", "summary_of_the_review": "In terms of technical novelty and experimental setup, the quality of this paper is not enough to be accepted to ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a joint learning approach which combines item-based representations and interest-based representations to improve recommender systems. Overall the scores are negative among all the reviewers. The reviewers acknowledge that the proposed approach provides a simple yet effective way to improve the existing item-based representations. However, all the reviewers pointed out concerns around the motivation and limited novelty (the proposed approach mostly combines a few existing approaches together without careful examination/exploration in the experiments). Furthermore, the baselines considered in the paper are on the relatively weak side. The authors didn't provide any response. Therefore, I vote for reject.
Decision: Reject