SOIL: Contrastive Second-Order Interest Learning for Multimodal Recommendation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 08:23:52

Abstract: Mainstream multimodal recommender systems are designed to learn user interest by analyzing user-item interaction graphs. However, what they learn about user interest needs to be completed because historical interactions only record items that best match user interest (i.e., the first-order interest), while suboptimal items are absent. To fully exploit user interest, we propose a Second-Order Interest Learning (SOIL) framework to retrieve second-order interest from unrecorded suboptimal items. In this framework, we build a user-item interaction graph augmented by second-order interest, an interest-aware item-item graph for the visual modality, and a similar graph for the textual modality. In our work, all three graphs are constructed from user-item interaction records and multimodal feature similarity. Similarly to other graph-based approaches, we apply graph convolutional networks to each of the three graphs to learn representations of users and items. To improve the exploitation of both first-order and second-order interest, we optimize the model by implementing contrastive learning modules for user and item representations at both the user-item and item-item levels. The proposed framework is evaluated on three real-world public datasets in online shopping scenarios. Experimental results verify that our method is able to significantly improve prediction performance. For instance, our method outperforms the previous state-of-the-art method MGCN by an average of $8.1\%$ in terms of Recall@10.

Main paper: 
 introduction: 
 In recent years, mainstream online service providers have offered online services that integrate multimedia information (e.g., images, texts, and videos) and have established multimodal recommender systems. A typical multimodal recommender system consists of a group of users, a series of items, and the corresponding multimedia data for each item. Service providers design various solutions to predict possible user-item pairs based on historical user-item interactions [23, 29] .Among these multimodal recommendation solutions, most of them are graph-based approaches that utilize graph neural networks to exploit the relationship between user-item interactions and multimodal item embeddings [4, 18, 20, 23, 24, 26, 28] . Graph-based approaches generally begin by constructing an interaction-based user-item graph and an item-item relationship graph, and then perform graph convolutional learning on these graphs separately. Finally, the outputs of two graph convolutional networks are fused to predict the user-item matching pairs. Experimental results in recent reports have demonstrated that learning on interaction-based graphs improves prediction accuracy significantly.Despite the great success of interaction-based graph learning methods, more attention must be paid to the fact that relying solely on historical user-item interaction records is insufficient to represent user interest. For better understanding, we illustrate a typical sequence of user behaviors in the online shopping scenario in Figure 1 . As illustrated in Figure 1 , a person who is looking for fashionable spring clothing will first choose the candidate items that attract her and then click or buy the item that best matches her interest (In this paper, we refer to this phenomenon as the Best-Match Trading Principle.). The user behavior sequence indicates that items in the candidate list are part of user interest and that the user is likely to select these items in the future. For example, when an item that we have previously purchased is out of stock, we tend to select one from the candidate list for purchase. The experimental results in Figure 6 also verify that users do purchase the items in the candidate list reconstructed in this paper. Recommender systems that only use items in the interaction list are agnostic about the items in the candidate list, thus degrading recommendation performance. Taking into account this phenomenon, a recommender system should simultaneously manage the items on the interaction and candidate lists to thoroughly understand user interest. However, most previous studies failed to utilize these candidate items because they were not recorded in interaction logs.To fully exploit user interest, we aim to identify potential candidate items for each user utilizing multimodal information. By investigating previous studies, we outline three critical issues in mainstream multimodal recommendation solutions:(1) Incomplete User-Item Graph: The absence of interaction information between users and candidate items in the useritem graph leads to insufficient learning of user interest and thus degrades the recommendation performance. (2) Interest-agnostic Item-Item Graph: The absence of user interest information in the item-item graph leads to a vulnerable relationship between users and multimodal information, deteriorating recommendation performance. (3) Limited Interest Consistency: Inconsistency in user interest between multimodal embeddings and user-item embeddings for the same interaction record will lead to low ranking weights and degrades recommendation performance. To challenge these issues, we propose a solution to retrieve potential candidate items for each user and optimize the recommendation model with contrastive learning techniques, named the Second-Order Interest Learning (SOIL) framework. In this paper, for each user, we denote all items on the interaction list as first-order interest and all items on the candidate list as second-order interest. In the proposed SOIL framework, we explicitly construct a second-order user-item interaction graph and interest-aware item-item graphs, and propose to enhance interest consistency with a user-item level contrastive learning module and an item-item level contrastive learning module. Technically, to handle the challenge of incomplete user-item graph, we first calculate the attraction score for each potential user-item pair based on the similarity of multimodal features, and then construct the augmented user-item interaction graph that incorporates both attraction scores and interaction logs. To handle the challenge of interest-agnostic item-item graph, we first compute interest representations for each user through multimodal features, and then construct an interest-aware item-item graph based on the similarity between multimodal features and interest representations.To handle the challenge of limited interest consistency, we first design a contrastive learning module at the user-item level to enhance user interest for each user in a self-supervised learning manner. Furthermore, we design an item-item level contrastive learning module to maintain the consistency of user interest between user-item embeddings and multimodal embeddings.In summary, the contributions of our work are as follows:(1) We identify that due to the best-match trading principle, useritem interactions contain only the items that best match the user interest, while suboptimal items are absent. To mitigate this problem, we propose a Second-Order Interest Learning (SOIL) framework to exploit the second-order interest from unrecorded suboptimal items.(2) We propose to augment the user-item graph with secondorder interest and construct interest-aware item-item graphs through user interest. With these graphs, we conduct contrastive learning at both the user-item and item-item levels to enhance user interest. (3) We conduct extensive experiments on three public datasets in online shopping scenarios. Experimental results demonstrate that our method is able to significantly outperform state-ofthe-art methods by an average of 6.8% in terms of Recall@10. 
related work 2.1 multimodal recommendation: 
 Multimodal recommendation models utilize multimedia information (e.g., images, texts, and videos) and user-item interactions to provide personalized recommendation services [2, 21, 24, 25] . Traditional multimodal recommendation methods mainly incorporate multimodal information into collaborative filtering [15] or matrix factorization [12] models to improve prediction performance [6, 11] . Among them, VBPR [6] is a method based on matrix factorization that employs pre-trained convolutional neural networks to extract visual features, which are then integrated into item embeddings. MAML [10] is designed to exploit user preferences with an attention network. This method first estimates the user attention for each item and then incorporates the attention information into matrix factorization models. Inspired by graph convolution networks (GCNs) [9] , researchers have proposed various graph-based methods to learn user preferences [18, 20, 21, 24, 29] . Among them, MMGCN [21] is a representative method that constructs a bipartite user-item graph for each modality to implicitly exploit the multimodal item relationship. To directly exploit the item-item relationship, Zhang et al. [24] specifically designed an item-item graph to utilize multimodal features. Recent studies [4, 23, 26, 28] have verified that explicitly constructing item-item graphs significantly improves recommendation performance. Among them, FREEDOM [28] demonstrates that dynamic item-item graphs will degrade recommendation performance and presents a denoising framework with a fixed item-item graph.Graph-based recommendation models treat the information in recommender systems as various graphs and leverage graph convolution networks to embed user-item interactions [22] . LightGCN [7] is a representative graph-based recommendation method that simplifies the vanilla graph convolution network to satisfy the requirements of recommender systems. In multimodal recommender systems, graph-based recommendation methods have demonstrated extensive ways to construct graphs. For instance, GRCN [20] constructs a dynamic interaction graph and adaptively refines the structure of the interaction graph based on the training status. DualGNN [18] constructs a bipartite user-item graph and a user co-occurrence graph to exploit dynamic user preferences. LAT-TICE [24] presents a modality-aware structure learning network to exploit the item relationship for each modality and constructs an item-item graph based on the learned item relationship. MGCN [23] constructs an item-item graph for each modality and performs graph convolution learning on these graphs separately. Similarly, DRAGON [26] first constructs an item-item graph for each modality and then fuses these modality graphs into a multimodal item-item graph.LGMRec [4] learns hypergraphs for each modality to exploit the dependency relationship between items.Our work aims to build graphs that incorporate interaction records and multimodal features to capture user interest in a comprehensive way. To this end, we construct a user-item graph that contains both first-order and second-order interest. Furthermore, we propose to incorporate user interest into the item-item graph to leverage user preferences across various modalities. The proposed framework consists of three distinct graphs: a user-item interaction graph augmented by second-order interest, an interest-aware item-item relationship graph for the visual modality, and a similar graph for the textual modality. In this framework, graph convolutional networks are initially deployed on each of the three graphs to learn representations of users and items. Subsequently, contrastive learning is applied at both the user-item and item-item levels to effectively capture user interest. 
proposed method 3.1 problem formulation: 
 A typical multimodal recommendation system generally consists of a group of users, a series of items, and corresponding multimedia data for each item. In our work, we denote the set of 𝑚 users by U = {𝑢 1 , 𝑢 2 , ..., 𝑢 𝑚 }, and the set of 𝑛 items by I = {𝑖 1 , 𝑖 2 , ..., 𝑖 𝑛 }. We denote the multimodal data of 𝑛 items by M = {𝑀 1 , 𝑀 2 , ..., 𝑀 𝑛 }, where 𝑀 𝑖 = {𝑀 𝑣 𝑖 , 𝑀 𝑡 𝑖 } refers to the visual and textual features of the 𝑖-th item. The interaction matrix R ∈ {0, 1} |U| × |I| is a collection of all user-item interactions, where 𝑟 𝑢,𝑖 ∈ {0, 1} indicates whether user 𝑢 interacts with item 𝑖. Note that R is also referred to as the first-order interaction matrix in this paper.For better understanding, we illustrate the proposed Second-Order Interest Learning (SOIL) framework in Figure 2 . In this framework, we construct an augmented user-item interaction graph that contains both first-order and second-order interest, an interestaware item-item relationship graph for the visual modality, and an interest-aware item-item graph for the textual modality. The three graphs are built using user-item interaction records and multimodal features, and the process of constructing these graphs will be detailed in the subsequent sections. With these three graphs, graph convolutional networks can be implemented to learn the representations of users and items. To enhance the exploitation of user interest and maintain consistency in user interest across the three graphs, we implement contrastive learning modules at both the user-item and item-item levels for user and item representations. Finally, we optimize the whole model with the pairwise Bayesian personalized ranking loss.As illustrated in Figure 1 , we identify that the items on the candidate list are not recorded. However, these candidates are crucial aspects of user interest that can improve recommendation performance. In this paper, we denote all candidate items of a user as the second-order interest of that user. To retrieve candidate items and exploit the second-order interest, we first compute the attractiveness score for each potential user-item pair and then append potential user-item pairs to the interaction graph by taking attractiveness scores as edge weights. The augmented interaction graph is depicted in Figure 3 , with dashed lines representing the newly added edges and the weights corresponding to the scores of the items that attract users. For instance, path "𝑢 2 --𝑖 3 " indicates that item 𝑖 3 has an attractiveness score of 0.85 for user 𝑢 2 .In our work, the attractiveness score is calculated by the similarity between an item and the first-order interest of a user. We first calculate the similarity of the features for each modality to obtain attractiveness scores. The similarity matrix 𝑆 ∈ R 𝑛×𝑛 consists of 𝑛 × 𝑛 similarity scores, where the element 𝑠 𝑖,𝑗 in row 𝑖 and column 𝑗 is calculated with a cosine similarity function as follows:EQUATIONwhere 𝑀 𝑖 and 𝑀 𝑗 refer to the multimodal features of item 𝑖 and item 𝑗. In our work, we calculate the similarities of visual and textual features separately and denote them as 𝑠 𝑣 𝑖,𝑗 and 𝑠 𝑡 𝑖,𝑗 . Based on Eq.( 1), we calculate feature similarity matrices 𝑆 𝑣 and 𝑆 𝑡 for visual and textual modalities. In our work, the attractiveness score matrix is calculated by the dot product of 𝑆 𝑣 and 𝑆 𝑡 , denoted as 𝑆 𝑎 = 𝑆 𝑣 • 𝑆 𝑡 . Each element 𝑠 𝑎 𝑖,𝑗 in the matrix 𝑆 𝑎 represents the similarity between the item 𝑖 and the item 𝑗. With matrix 𝑆 𝑎 , we select items similar to the first-order interest as the second-order interest for each user. The process of selecting second-order interest for user 𝑢 is formulated as follows: With Eq.( 2), we are able to build the second-order interaction matrix R 𝑎 as follows:EQUATIONEQUATIONThe second-order interaction matrix R 𝑎 shares the same shape with the first-order interaction matrix R, allowing them to be integrated into the user-item interaction graph to learn collaborative representations. In our work, we implement a light graph convolutional network (lightGCN) [7] to compute collaborative information on the user-item interaction graph. The computation of the 𝑙-th layer is formulated as follows:EQUATIONwhere E 𝑙 𝑢𝑖 refers to the embedding matrix of the 𝑙-th layer. For the first layer, the initial embedding matrix 𝐸 0 𝑢𝑖 = 𝐸 𝑢 ⊕ 𝐸 𝑖 is the concatenation of the user embedding matrix 𝐸 𝑢 and the item embedding matrix 𝐸 𝑖 . L refers to the sum of the Laplacian matrix for the first-order interaction graph and the Laplacian matrix for the second-order interaction graph. It is formulated as follows:EQUATIONwhere D 1 is the diagonal degree matrix of R, whose 𝑖-th element is the sum of the 𝑖-th row in R. Similarly, D 2 is the diagonal degree matrix of R a . Note that we compute the Laplacian matrices of the two graphs separately to maintain the original structures inside the two graphs.According to Section 2.2, previous multimodal recommendation approaches have constructed various item-item graphs based on feature similarity and 𝑘NN sparsification [1, 23, 24, 28] . However, these graphs are agnostic to interest and are not able to learn user interest to improve recommendation performance. To enhance the relationship between multimodal features and users, we propose to create interest-aware item-item graphs for each modality.In our work, we propose to construct interest-aware graphs with feature similarity matrices 𝑆 𝑣 and 𝑆 𝑡 , which are calculated with Eq.( 1). With 𝑆 𝑣 and 𝑆 𝑡 , we denote user interest by the sum of feature similarities, which is formulated as follows:EQUATIONwhere 𝐼 (𝑢) = {𝑖 |𝑟 𝑢,𝑖 = 1, 𝑟 𝑢,𝑖 ∈ R} refers to the set of items that user 𝑢 has interacted with, 𝑆 𝑣 (𝑢) refers to the sum of visual similarity of all items in 𝐼 (𝑢), and 𝑆 𝑣 (𝑢) has the shape of 1 × 𝑛. 𝑆 𝑡 (𝑢) refers to the sum of textual similarity. With 𝑆 𝑣 (𝑢) and 𝑆 𝑡 (𝑢), we are able to select top-𝑘 items to represent user interest. The process of selecting top-𝑘 items is formulated as follows: EQUATIONwhere Ŝ𝑣 𝑖,𝑗 and Ŝ𝑡 𝑖,𝑗 refer to the interest-aware graphs for visual and textual modalities, respectively. To maintain the latent relationship among multimodal features, we conduct 𝑘NN sparsification [1] In our work, we normalize Ŝ𝑣 𝑖,𝑗 , Ŝ𝑡 𝑖,𝑗 , S𝑣 𝑖,𝑗 and S𝑡 𝑖,𝑗 in the same way to guarantee robust graph convolutional learning. Taking Ŝ𝑣 𝑖,𝑗 as an example, the normalization process is expressed as Ŝ𝑣 𝑖,𝑗 = (𝐷) -1 2 Ŝ𝑣 𝑖,𝑗 (𝐷) -1 2 , where 𝐷 𝑖𝑖 = 𝑗 Ŝ𝑣 𝑖,𝑗 is the diagonal degree matrix of Ŝ𝑣 𝑖,𝑗 . Once the normalization is done, we merge the interest-aware graphs and similarity graphs for each modality into the interestaware item-item graphs as follows:EQUATIONTo learn interest-aware multimodal representations, we implement graph convolutional networks on G 𝑣 and G 𝑡 , and formulate the computation of the 𝑙-th layer as:EQUATIONwhere E 𝑙 𝑖𝑣 and E 𝑙 𝑖𝑣 refer to the embedding matrices of the 𝑙-th layer for visual and textual modalities. For the first layer, initial embedding matrices 𝐸 0 𝑖𝑣 and 𝐸 0 𝑖𝑡 are calculated by fusing multimodal features with item embeddings as follows: where 𝑀𝐴𝑃-𝑉 (•) and 𝑀𝐴𝑃-𝑇 (•) are linear mapping functions or networks for visual and textual features, M 𝑣 and M 𝑡 are multimodal features for all items in visual and textual modalities.EQUATIONBased on interest-aware multimodal representations 𝐸 𝑖𝑣 and 𝐸 𝑖𝑡 derived from Eq.( 11), we integrate them with interaction graphs to enhance user interest in multimodal features. The process of integration is formulated as follows:EQUATIONwhere G 𝑢𝑖 = R + R 𝑎 refers to the interaction graph that contains both first-order and second-order interest, ⊕ refers to the concatenate operation. Finally, we fuse multimodal representations of all modalities with an attention mechanism [17, 19, 23, 26] , which is formulated as follows:EQUATIONwhere 𝛼 = softmax(𝑄 ( Ẽ𝑣 𝑢𝑖 )⊕𝑄 ( Ẽ𝑣 𝑢𝑖 )) refers to the attention weights, and 𝑄 (•) refers to the widely used linear attention layer [17, 23] .As mentioned in Section 1, the inconsistency of user interest between user-item embeddings and multimodal embeddings seriously degrades recommendation performance. To handle this challenge, we propose to first enhance user interest in multimodal features and then deploy an interest consistency maintenance module. To enhance the user interest, we design a contrastive learning module on embeddings of users and items as follows:EQUATION)where ê𝑢 is user representation derived from embedding matrix 𝐸 = 𝐸 𝑢𝑖 + Ẽ𝑢𝑖 , 𝑒 𝑖 and ẽ𝑖 are item representations derived from embedding matrices 𝐸 𝑢𝑖 and Ẽ𝑢𝑖 , and 𝜏 is the temperature factor.To maintain the consistency of user interest between user-item embeddings and multimodal embeddings, we propose to deploy a contrastive learning module on embeddings of user-item graph and item-item graph as follows:EQUATIONwhere 𝑒 𝑖 and ẽ𝑖 are item representations derived from embedding matrices 𝐸 𝑢𝑖 and Ẽ𝑢𝑖 , 𝑒 𝑢 and ẽ 𝑢 are user representations, and 𝜏 is the temperature factor.In our work, we first construct a second-order interaction graph to learn second-order interest, and then construct interest-aware itemitem graphs for visual and textual modalities. We perform graph convolutional learning on each of the three graphs and obtain a embedding matrix 𝐸 𝑢𝑖 for the user-item graph, and an embedding matrix Ẽ𝑢𝑖 for the item-item graph. Following the widely used setting [4, 23, 28] , we optimize the prediction model by minimizing the Bayesian personalized ranking loss as follows:EQUATIONwhere R = {(𝑢, 𝑖 + , 𝑖 -)|𝑟 𝑢,𝑖 + = 1, 𝑟 𝑢,𝑖 -= 0, 𝑟 𝑢,𝑖 + ∈ R, 𝑟 𝑢,𝑖 -∈ R}. r𝑢,𝑖 + = ê𝑢 • êT 𝑖 + refers to the interaction score of a positive sample, where ê𝑢 and ê𝑖 + are user representation and item representation derived from embedding matrix 𝐸 = 𝐸 𝑢𝑖 + Ẽ𝑢𝑖 .Finally, we optimize the whole model by minimizing the following loss function:EQUATIONwhere 𝜆 𝑢𝑖 and 𝜆 𝑖𝑖 are hyper-parameters that control the contribution of two contrastive learning modules.In the model inference, the interaction score between user 𝑢 and item 𝑖 is computed as follows:EQUATIONwhere ê𝑢 and ê𝑖 are user representation and item representation derived from embedding matrix 𝐸 = 𝐸 𝑢𝑖 + Ẽ𝑢𝑖 . 
experiments: 
 In this section, we conduct extensive experiments on three public datasets to study the following research questions: RQ1: Does the SOIL improve recommendation performance? RQ2: What is the contribution of the three graphs and contrastive learning modules in improving performance? RQ3: What is the effect of different hyper-parameters on the model performance? RQ4: Does the second-order interest identify the items with which a user would like to interact? RQ5: Do contrastive learning modules improve consistency between user-item embeddings and multimodal embeddings?To evaluate our method, we conduct extensive experiments on three widely used Amazon review datasets [5, 13] . Following [4, 23, 28] , we choose the following categories: (a) Baby, (b) Sports and Outdoors, and (c) Clothing, Shoes and Jewelry. They are commonly referred to as Baby, Sports, and Clothing. All three datasets contain visual images and textual descriptions of the items. Following the widely used setting [4, 23, 26, 28] , we use pre-extracted features: 4,096-dimensional for visual data and 384-dimensional for textual data. In our work, all items and users are filtered using the 5-core setting, and the statistics of datasets are reported in Table 1 .Evaluation Metric. Following [4, 24, 29] , we adopt the all-ranking protocol with two widely used metrics to evaluate the performance of top-𝐾 recommendations: Recall (Recall@𝐾) and Normalized Discounted Cumulative Gain (NDCG@𝐾). We report the average performance of all users with 𝐾 set to 10 and 20. In our research, we randomly divided the historical interactions of each user into training, validation, and testing sets with ratios of 8:1:1. Implementation Details. In our work, we implement all compared methods and the proposed method with the MMRec framework [27] . For a fair comparison, we follow the experimental settings and pre-extracted visual and textual features in the MMRec framework. We use Recall@20 on the validation set as the indicator for early stopping. The embedding size for both users and items is set to 64 for all models. The number of GCN layers for the user-item graph is set to 2, and for item-item graphs, it is set to 1. We initialize all embedding parameters with the Xavier method [3] , and optimize all models with the Adam optimizer [8] . Our method is implemented by Pytorch 1.12 and is trained on a single NVIDIA RTX 2080TI. Hyper-parameters 𝜆 𝑢𝑖 and 𝜆 𝑖𝑖 are chosen from {0.001, 0.01, 0.1, 1.0}. The 𝑘 for the 𝑘NN sparsification in Eq.( 7) and Eq.( 9) are chosen from {5, 10, 15, 20}. The 𝑘 for top-𝑘 second-order interest in Eq.( 2) is chosen from 1 to 10. The code will be released on our website, and we also provide the model structure and the test code at the anonymous website (URL: https://anonymous.4open.science/r/soil-B1C3) for validation during the review phase.Compared Methods. In our work, we validate our approach by comparing it with the following traditional methods and multimodal recommendation methods:(1) BPR [14] is a traditional method presenting a Bayesian personalized ranking optimization criterion to optimize the ranking models directly. (2) LightGCN [7] is a traditional method that simplifies the GCN to make it more appropriate for recommendation. (3) VBPR [6] is a multimodal method that incorporates visual signals into the collaborative filtering model and optimizes the model with Bayesian personalized ranking loss. (4) LATTICE [24] is a multimodal method to construct item-item relationship graphs from multimodal features directly.(5) SLMRec [16] is a multimodal method that performs data augmentation on multimodal features and deploys contrastive learning modules to optimize the model. ( 6) BM3 [29] is a self-supervised multimodal recommendation method that reconstructs the user-item interaction graph and aligns multimodal features. ( 7) FREEDOM [28] is a multimodal denoising method that freezes the item-item graph in LATTICE [24] and denoises the useritem graph to improve recommendation performance. (8) MGCN [23] is a multimodal approach that reduces the effect of modal noise by treating user-item interactions and modal features as different views and performing graph convolutional learning in each of them separately. (9) LGMRec [4] is a multimodal approach that learns representations from local and global graphs to model local and global user interest jointly.The ability to improve recommendation performance.To evaluate the effect of our approach, we perform extensive experiments on three datasets and compare the results with several multimodal methods. We report the results of our method and other multimodal methods in Table 2 . According to Table 2 , our method is able to surpass the previous methods by a wide margin in all evaluation metrics. From the results, our method outperforms previous state-of-the-art methods by 5.59%, 7.82%, and 7.18% on Baby, Sports, and Clothing in terms of Recall@10. Similarly, our method is able to surpass previous state-of-the-art methods by 4.58%, 9.57%, and 8.64% on Baby, Sports, and Clothing in terms of NDCG@10. For the results of Recall@20, our method is able to achieve the improvement of 2.59%, 4.43%, and 5.60% compared to previous state-of-the-art methods on Baby, Sports, and Clothing. Furthermore, our method surpasses other methods by 3.18%, 6.85%, and 6.54% on Baby, Sports, and Clothing in terms of NDCG@20. Among all the compared methods, MGCN has a similar network structure to our approach. According to the comparison in Table 2 , our method is able to achieve the improvement of 9.68%, 7.82%, and Table 3 : Results of the ablation study. SOI is short for the second-order interaction graph without first-order interest. IA is short for the interest-aware graph without the similarity graph. The ✓ indicates that the corresponding component is included.Baby Sports Clothing SOI IA L 𝑢𝑖 L 𝑖𝑖 R@10 R@20 N@10 N@20 R@10 R@20 N@10 N@20 R@10 R@20 N@10 N@20 0.0556 0.0884 0.0303 0.0387 0.0650 0.0996 0.0354 0.0443 0.0509 0.0768 0.0279 0.0344 ✓RQ2: The contribution of different components. To validate the contribution of each component in our work, we perform the ablation study on all evaluation datasets and report the experimental results in Table 3 . In the ablation study, we focus on the effects of the second-order interaction graph, the interest-aware graph, the user-item contrastive learning module, and the item-item contrastive learning module in our framework. The first line in Table 3 indicates the baseline model without the above four components. From Table 3 , our method is able to surpass the baseline model by 22.3%, 20.9%, and 35.0% on Baby, Sports, and Clothing in terms of Recall@10. Furthermore, our method is able to achieve the improvement of 20.4%, 22.9%, and 35.1% on Baby, Sports, and Clothing in terms of NDCG@10. From Table 3 , the second-order interaction graph and interestaware graph are able to achieve the improvement of 9.53%, 2.6% and 17.2% compared to the baseline model on Baby, Sports, and Clothing in terms of Recall@10. The two contrastive learning modules are able to surpass the baseline model by 14.0%, 11.0%, and 30.6% on Baby, Sports, and Clothing in terms of Recall@10. From the last four lines in Table 3 , both the interest-aware graph and the second-order interest graph contribute significantly to improving the performance of the recommender systems. This observation verifies that integrating second-order interest into the item-item graph is effective in improving recommendation performance. The comparison of all the results in Table 3 suggests that the primary contributors to performance improvement are the two contrastive learning modules, while the second-order interaction graph and interest-aware graphs are secondary contributors.RQ3: The effect of hyper-parameters in our approach. Effect of hyper-parameters 𝜆 𝑢𝑖 and 𝜆 𝑖𝑖 . The hyper-parameter 𝜆 𝑢𝑖 controls the contribution of the loss function L 𝑢𝑖 that is utilized to optimize the contrastive learning module of the user-item level. The hyper-parameter 𝜆 𝑖𝑖 controls the contribution of the loss function L 𝑖𝑖 , which is utilized to optimize the contrastive learning module at the item level. We conduct experiments with different combinations of the two hyper-parameters and report the results in Figure 4 (a). From the results, the performance of the model first increases and then gradually declines when 𝜆 𝑖𝑖 increases. For the hyper-parameter 𝜆 𝑢𝑖 , the performance of the model shows a similar trend when 𝜆 𝑢𝑖 increases. As 𝜆 𝑢𝑖 decreases, the performance of the model decreases faster as 𝜆 𝑖𝑖 increases. Our method is prone to achieve better prediction performance with smaller 𝜆 𝑢𝑖 and 𝜆 𝑖𝑖 .Effect of 𝑘 in interest-aware item-item graphs. In our work, we adopt different 𝑘 values in the top-𝑘 operations of Eq.( 7) and Eq.( 9). The 𝑘 for interest in Eq.( 7) controls the number of samples when selecting interest-aware samples. The 𝑘 for similarity in Eq.( 9) controls the number of samples in selecting similar samples for each item. We conduct experiments with different combinations of the two 𝑘s and report the results in Figure 4(b) . Note that the results are reported with all other parameters fixed and without tuning. From the results, the performance on Baby slowly grows up when 𝑘 for similarity increases, while the performance on Clothing fluctuates when 𝑘 for similarity increases. According to the results of 𝑘 for interest, the performance on Baby first grows up and then slowly decreases when 𝑘 increases. In our work, 𝑘 for similarity is set to 10 for all datasets. The 𝑘 for interest is set to 10 for Baby and 5 for both Sports and Clothing.Effect of 𝑘 when selecting second-order interest. In our work, we adopt different 𝑘 values in the top-𝑘 operations of Eq.( 2). The 𝑘 in Eq.( 2) controls the number of samples when selecting second-order interest samples. We conduct experiments on Baby and Sports and report the results in Figure 5 . Note that the results are reported with all other parameters fixed and without tuning. From the results of Recall@10 in Figure 5 , the performance on Sports slowly grows as 𝑘 increases, while the performance on Baby first increases and then slowly decreases as 𝑘 increases. The results of NDCG@10 show similar trends to the results of Reacall@10. According to the ablation study in Table 3 , the second-order interaction graph is undoubtedly capable of improving the model performance. The observation in Figure 4 indicates that the 𝑘 for selecting second-order interest depends on the dataset and varies widely across datasets. In our work, to achieve better performance, the 𝑘 for second-order interest is set to 2 for Baby, 6 for Sports and 3 for Clothing. RQ4: The effectiveness of second-order interest. In our work, we propose to exploit the second-order interest of each user to retrieve the items in the candidate list. To qualitatively verify that the proposed second-order interest is practical in selecting potential interaction candidates, we compare the items on the second-order interest list with those on the interaction list of the test set and present the comparison results in Figure 6 . Note that second-order interest is generated with the training data. From the results, our method is able to select two potential interaction candidates for user 2465 and user 13979. For users 289 and 1472, our method is able to select one potential interaction candidate. This observation verifies that the proposed second-order interest is able to retrieve potential interaction candidates.RQ5: The effect of contrastive learning modules. In our work, we design contrastive learning modules at both the user-item and item-item levels to maintain user interest consistency between user-item embeddings and multimodal embeddings. To verify the ability to enhance the consistency of interest, we randomly selected user-item embeddings and multimodal embeddings and calculated the cosine similarity between them. The results are illustrated in Figure 7 . In Figure 7 , the vertical axis represents user-item embeddings, and the horizontal axis represents multimodal embeddings. The results indicate that in the method employing comparative learning modules, the multimodal embeddings and user-item embeddings of the same interaction are more closely aligned, whereas the similarity among different interactions is reduced. This observation verifies that contrastive learning modules in our work are able to enhance the feature consistency between user-item embeddings and multimodal embeddings. 
conclusion: 
 This work presents a Second-Order Interest Learning framework to exploit the second-order interest from unrecorded candidate items. In this framework, we exploit candidate items from the perspective of constructing interest-aware graphs and self-supervised optimization. To construct interest-aware graphs, we first design an augmented user-item interaction graph that contains both firstorder and second-order interest, and then we integrate the user interest with multimodal features to construct the interest-aware item-item graph for each modality. In self-supervised optimization, we design a user-item level contrastive learning module and an item-item level contrastive learning module to enhance the user interest between user-item embeddings and multimodal embeddings. Extensive experiments verify that our method is capable of significantly improving the recommendation performance.  
 
 Appendix:  


Decision: The reviewers agree that the paper is interesting and relevant to a large part of ACM Multimedia community. They further agree that the idea of utilizing what authors term as second-order interests instead of conventional historical interactions encoded in user-item matrix is interesting. However, they have identified certain drawbacks as well, such as the need for better describing the exact novelties of the proposed approach as well as the need for clarifying the equations in Section 3.3.

I believe that the paper should be accepted to the program of ACM Multimedia 2024. When preparing a revised version of the paper the authors should carefully address reviewer comments and proofread the paper for any grammar (e.g. an issue with the tenses was suggested by one of the reviewers) and typographical errors.

Reviews: review: paper_summary: paper_summary: This paper introduces a comprehensive framework designed to enhance the accuracy and relevance of recommendations in multimodal recommender systems. The core design of this work is introducing a concept called second-order interests, which are essentially potential user interests that are typically overlooked in conventional user-item interaction data because they do not result in actual interactions. This paper utilizes multimodal information to obtain this information and augment standard user-item graphs. It further employs contrastive learning to refine the user and item embeddings at both the user-item and item-item levels, thereby enhancing the consistency and relevance of the learned embeddings. The proposed methods are evaluated on three real-world datasets, showing improvements in recommendation accuracy over existing methods.
strength_weakness: strength_weakness: - Interesting idea to exploit second-order interest. Exploiting more useful information from limited feedback information is a key problem in recommender systems. Leveraging multimodal information is reasonable.

- The proposed method has been compared with many SOTA methods, and the performance is good.
limitations: limitations: - The novelty is somewhat limited, and the proposed solution seems straightforward. Both constructing an augmentation graph and contrastive learning are not new.

- The proposed solution may add extra burden in training and inference, which should be discussed and analyzed.
paper_summary: paper_summary: Conventional multimodal recommender systems learn user interests from user-item interaction graphs, but they only capture first-order interests and ignore suboptimal items. To address this problem, the authors propose a Second-Order Interest Learning (SOIL) framework that extracts second-order interests from unrecorded suboptimal items. The framework constructs user-item and interest-aware item-item graphs for visual and textual modalities using multimodal feature similarity. Graph convolutional networks are used to learn user and item representations from these graphs. Contrastive learning modules further enhance the exploitation of both first- and second-order interests. Evaluated on three real-world datasets, the SOIL framework significantly improves prediction performance.
strength_weakness: strength_weakness: The authors have made the code and dataset publicly available, which increases the credibility of the paper's results.
The experimental results seems convincing.
limitations: limitations: a)	At the end of the second paragraph of the Introduction, it is mentioned that "Experimental results in recent reports have demonstrated that learning on interaction-based graphs improves prediction accuracy significantly". Suggest adding a citation after 'reports'.
b)	The article frequently uses the simple past tense or present perfect tense. It is recommended to use the simple present tense.
c)	It is recommended that the author add the differences and connections between the method in this paper and the methods mentioned in the Related Work section.
d)	The formatting of symbols in the text is inconsistent. Symbols representing matrices should be uppercase, upright, and bold, while symbols representing vectors should be lowercase, upright, and bold.
e)	Please increase the clarity of Figure 1. It is recommended to enlarge the text in the image above the word 'Search'.
paper_summary: paper_summary: To exploit more accurate user interests, the paper proposes a second-order interest learning framework. In this work, the author denotes items on the interaction list as first-order interests and items on the candidate list as second-order interests. These interests are separately used to construct interest graphs and learn structural information within the graphs via LightGCN. Furthermore, the paper constructs interest-aware item-item graphs using a feature similarity measure, which enhances the relationship between multimodal features and user interests. Finally, pairwise contrastive learning is employed to align the information between ID embeddings and multimodal embeddings.
strength_weakness: strength_weakness: 1. The paper highlights the importance of second-order interest in completing user preferences and proposes a structured method by calculating the attractiveness score.
2. The paper's motivation is well-articulated, with the authors clearly outlining the three problems they aim to solve. This clarity strengthens the rationale behind the proposed methods.
3. The paper conducts extensive experiments and ablation studies on three datasets, verifying the effectiveness of the proposed method.
limitations: limitations: 1. There is an ambiguous description of the interest-aware graph construction process in Section 3.3. Specifically, the author first calculates the similarity sum for each modality feature via Equation 6, but then introduces the top-𝑘 function to select k items to represent user interest.
2. There is a redundant formulation of Equation 14.
3. There is a misleading explanation of Equation 16. The paper states that it "deploys a contrastive learning module on embeddings of the user-item graph and item-item graph," but the formulation does not reflect the purpose as described by the authors.