CIRP: Cross-Item Relational Pre-training for Multimodal Product Bundling

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 16:13:13

Abstract: Product bundling has been a prevailing marketing strategy that is beneficial in the online shopping scenario. Effective product bundling methods depend on high-quality item representations capturing both the individual items' semantics and cross-item relations. However, previous item representation learning methods, either feature fusion or graph learning, suffer from inadequate cross-modal alignment and struggle to capture the cross-item relations for cold-start items. Multimodal pre-train models could be the potential solutions given their promising performance on various multimodal downstream tasks. However, the cross-item relations have been under-explored in the current multimodal pre-train models. To bridge this gap, we propose a novel and simple framework Cross-Item Relational Pre-training (CIRP) for item representation learning in product bundling. Specifically, we employ a multimodal encoder to generate image and text representations. Then we leverage both the cross-item contrastive loss (CIC) and individual item's image-text contrastive loss (ITC) as the pre-train objectives. Our method seeks to integrate cross-item relation modeling capability into the multimodal encoder. Therefore, even for cold-start items that have no relations, their representations are still relation-aware. Furthermore, to eliminate the potential noise and reduce the computational cost, we harness a relation pruning module to remove the noisy and redundant relations. We apply the item representations extracted by CIRP to the product bundling model ItemKNN, and experiments on three e-commerce datasets demonstrate that CIRP outperforms various leading representation learning methods.

Main paper: 
 introduction: 
 Product bundling is a prevailing strategy in retail markets, which aims to promote sales and improve consumer satisfaction by combining a set of products into bundles. Especially in the era of online shopping, an extensive array of products from diverse retail sectors are gathered on a single platform, thereby boosting bundling opportunities. Albeit beneficial, the ever-growing number of products poses challenges to the task of product bundling, which has also attracted particular attention in the research community.In order to develop an effective product bundling model, the challenge is to learn high-quality representations of product items from two aspects. (1) The representations need to capture the rich semantic features of individual items. For instance, the textual description of an electronic device represents its brand and functions, while the image of fashion apparel highlights intricate details, such as print and texture. Such multimodal semantics are essential for bundling items that are similar or compatible in terms of functionality or aesthetics. (2) The representations should be able to model the diverse and implicit relationships among items. Such relations (e.g., , co-purchasing or sequential interaction) imply auxiliary but crucial information to bundle certain items. For example for the famous beer and diaper case 1 , they can hardly be put together if only semantic features are considered, while the co-purchase relationship implies that they could be a good bundling option. Given the pervasive success of representation learning models pre-trained on large-scale dataset [5, 12, 33] , it is promising to pre-train an item representation learning model based on large-scale e-commerce data.Considering the two aspects of semantic and relational modeling, current pre-training methods can broadly be categorized into two streams: feature fusion and graph learning, as shown in Figure 1 . Specifically, feature fusion-based methods [11, 38] learn item representations of each modality separately (e.g., , visual, textual, and relational) and subsequently fuse them into a multimodal representation. Graph learning-based methods [19, 37] use pre-extracted item semantic features as node attributes and apply graph learning algorithms to refine these features through graph propagation. In spite of their wide usage, both of these methods suffer from two main limitations. First, the features from multiple modalities are learned separately, lacking in-depth cross-modal alignment or enhancement. Second, they struggle with accurately capturing relational patterns for cold-start items, which are new items that have not yet established any relational data with other items. In the light of these limitations, the powerful multimodal foundation models [22, 33] could be the potential solution given their outstanding performance on various multimodal downstream tasks. However, they focus mostly on the semantic modeling of image and text, with limited exploration into cross-item relations.To bridge this gap, we propose a novel but simple framework, named Cross-Item Relational Pre-training (CIRP). Specifically, we first construct an item-item relational graph based on the co-purchase item pairs. From this graph, we sample a pair of items that are directly connected and adopt two unimodal encoders, i.e., , image and text encoders [22] , to generate each item's multimodal representations of image and text. Thereafter, we harness the cross-item contrastive (CIC) loss to enforce the representations of related items to be close to each other. Simultaneously, we keep the image-text contrastive (ITC) loss of individual items to retain the cross-modal alignment. This simple pre-training framework can naturally integrate cross-item relations into the multimodal encoder (by the CIC loss), while preserving the in-depth aligned multimodal semantics (by the ITC loss). More importantly, even for cold-start items, CIRP can generate relation-aware multimodal representations, which could hardly be achieved by previous item representation learning methods of feature fusion and graph learning. Furthermore, considering the potential noise and heavy computational cost brought by the extensive amount of item-item relations, we propose a novel relation pruning module to remove those noisy or redundant connections. In the downstream task of product bundling, we use CIRP to extract item representations, which are then fed to the product bundling model ItemKNN. Experiments on three large-scale ecommerce show that CIRP can significantly boost the performance of product bundling compared with various leading methods for item representation learning. More interestingly, when pruning 90% of the relations, our method only experiences a slight performance drop, while just taking 1/10 of the pre-training time. The main contributions of this work are summarized as follows:• To the best of our knowledge, we are among the first to integrate the cross-item relational information into a multimodal pre-train model for product bundling. • We develop a novel framework CIRP that can simultaneously model both individual item's semantics and cross-item relations. We also propose a relation pruning module to improve pretraining efficiency and efficacy.• Experimental results on three product bundling datasets demonstrate the competitive performance of our method in terms of both efficacy and efficiency. 
related work: 
 We briefly review the works related to this paper from two streams: 1) multimodal pre-training, and 2) product bundling.Multimodal pre-training has experienced enormous progress in recent years. CLIP [33] is the pioneering work that unleashes the power of large-scale multimodal data by using a simple cross-modal contrastive loss. Following this trend, a surge of multimodal pretraining works emerge, such as BLIP [22] and BEiT-3 [42] , which achieve impressive performance on various downstream tasks. Especially after the breakthrough of Large Language Models (LLMs) brought by ChatGPT, researchers swiftly grasp this opportunity by integrating the language understanding capability of LMMs into the multimodal models, such as BLIP-2 [21] , LLaVA [26, 27] , MiniGPT-4 [49] , etc. These multimodal pre-trained models have demonstrated impressive performance on various downstream vision-language tasks. However, these models are not specifically designed to capture the relational data.Multiple studies in the E-commerce domain consider both multimodal semantic and relational pre-training, including K3M [50] , KnowledgeCLIP [32] , KG-FLIP [17] , FashionKLIP [44] , etc. The main objective of these works is to incorporate knowledge graph into the pre-trained multimodal models. Nevertheless, the relations used in these works are the triplet-formatted facts, while the cross-item relations are not captured. Consequently, these models can only be used for multimodal understanding-based tasks, such as classification, caption, cross-modal retrieval, etc., while the tasks (e.g., product bundling) that require the modeling of cross-item relations cannot be tackled well.Product bundling aims to combine a list of individual products into a bundle. It has been widely used in various business sections, such as e-commerce [39, 40] , fashion [30] , music stream [31] , games [4] , trips [25] , and food [24] , etc. Conventional product bundles are manifested by retailers manually, which is applicable due to a relatively small number of items. With the explosion of E-commerce, the number of items on a single platform boosts drastically, necessitating automatic product bundling methods. There are multiple works that are developed for personalized product bundling [1-3, 7, 14, 45] . However, most of them only rely on the relations between items, thus they cannot deal with cold-start items. Therefore, it is necessary to incorporate the multimodal semantic features of items and build multimodal product bundling methods [31] . We deem that the key of multimodal product bundling is to learn high-quality relation-aware multimodal representation, which is reasonably to be achieved by a pre-train model. To evaluate the performance of pre-train model, we leverage a simple method, ItemKNN [36] , which is also adopted by the previous work [40] for product bundling.Existing pre-train methods that can capture both cross-item relational and semantic data are the graph learning models, including .sg/Wantdo-Womens-Winter-Trench-XX-Large/dp/B06XW5VKWC/ref=sr_1_2_sspa?crid=1QBESAYH8NBS1&keywords=winter%2Bcoat%2Bwoman&qid=1699882640&sprefix=coat%2Bwo%2Caps%2C318&sr=8-2-2V0TmFtZT1zcF9hdGY&th=1&psc=1.sg/dp/B0CL74JKWX/ref=sspa_dk_detail_1?psc=1&pd_rd_i=B0CL74JKWX&pd_rd_w=ZajKv&content-id=amzn1.sym.e8a65a17-9528-4864-b5a1-87609e247ff8&pf_rd_p=e8a65a17-9528-4864-b5a1-87609e247ff8&pf_rd_r=0D88PC0HGX9YMPFCKT7X&pd_rd_wg=HwHMp&pd_rd_r=8595b59b d353&s=fashion&sp_csd=d2lkZ2V0TmFtZT1zcF9kZXRhaWw GCN [20] , GPT-GNN [16] , HeCo [43] , MPKG [10] , etc. These methods use pre-extracted item features to initialize the node representation, and then leverage graph propagation to integrate the relational information. They focus more on the relational information, falling short in modeling the semantics, especially the cross-modal alignment. Even worse, for cold-start items that have no relations before, such graph learning methods can hardly endow any relational data into the item representation. Our work would attempt to integrate the relational information into the multimodal backbone model with the relational pre-training framework." " ! # " # ℒ " $%& ℒ ",# &$& ℒ # $%& ℒ ",# Pre-train Objective GNN Enc GNN Dec $ % & ' $ % & Relational InputTo be noted, our task targets at bundle construction rather than personalized bundle recommendation [9, 29, 30, 34, 46, 48] , which aims to recommend pre-defined bundles to users according to users' preference. Nevertheless, these two tasks are highly related to each other since bundle construction is a pre-requisite step for platforms before they can offer bundle services. 
method: 
 We first introduce the preliminary of the pre-training and the downstream task. Then, we present our proposed CIRP, as shown in Figure 2 , which includes the relation pruning module and the pretraining framework. Finally, we briefly describe how to use the pre-trained representations for the task of product bundling.Problem Formulation: We target at pre-training a multimodal item representation learning model for the downstream task of product bundling. Given a large set of product items I = {𝑖 𝑛 } 𝑁 𝑛=1 , associated with each item's semantic features (i.e., text T = {𝑡 𝑖 } and image V = {𝑣 𝑖 }, where 𝑖 ∈ I) and item-item relations (i.e., an item-item graph, will be introduced later), we aim to train a model Φ(𝑣, 𝑡; Θ) that consumes an item's image 𝑣 and text 𝑡, then outputs a multimodal representation (v, t) that endow both semantic and relational information, where Θ is the parameters of the representation learning model. Thereafter, the item representation can be incorporated into product bundling models. Item Graph Construction: There are multiple types of relations that could be considered for pre-training, such as co-purchase (two items that are frequently purchased together), sequential interaction (two items are interacted with a certain user consecutively), and knowledge graph (two items are related via the knowledge graph), etc. Following the work [40] , we construct the item-item relation graph based on the co-purchase data, which has been demonstrated to be helpful relations for product bundling. Specifically, if two items are purchased consecutively by the same user in a short time period (i.e., one day), these two items will be connected with an undirected edge. Thereby, all the items I and edges E form a homogeneous graph G = {I, E}. It is worth noting that we aim to develop a pre-training framework for general types of relations among items. Different relation types or graph construction details may impact the pre-training, which will be left for future work.The graph is prone to include noisy or redundant edges, which are useless or even harmful to cross-item relational modeling. It is imperative to remove such relations from the item-item graph. More importantly, given the extensive amount of item-item relations, it is computationally heavy for large-scale pre-training. Therefore, to enhance the quality of the item-item relation graph and accelerate pre-training, we propose a relation pruning module. It comprises of two steps: we first train a graph auto-encoder to obtain the node representations; then we use the learned node embedding to prune the original graph. works [6, 13] and use the LightGCN kernel as the graph encoder. It includes two main components: information propagation and layer aggregation. The information propagation is defined as:EQUATIONwhere N 𝑖 denotes the neighbors of item 𝑖 in the graph, e (𝑘 ) 𝑖 ∈ R 𝑑 is the 𝑘-th layer representation of item 𝑖 during information propagation, and 𝑑 is the dimensionality of the representation. The first layer of the node embedding (e (0) 𝑖 ) is randomly initialized. After 𝐾 layers of information propagation, we aggregate the representations of multiple layers and obtain the final item representation e 𝑖 , represented as:e 𝑖 = 𝐾 ∑︁ 𝑘=0 e (𝑘 )𝑖 .(EQUATIONThe decoder part is a typical link prediction task. Given a pair of items (𝑖, 𝑗), we use the inner product to calculate the score of how likely these two items could be connected to each other, formally represented as: 𝑠 𝑖,𝑗 = e ⊺ 𝑖 e 𝑗 . We resort to the Bayesian Personalized Ranking (BPR) [35] loss to optimize the model, and the loss used to train the graph auto-encoder is denoted as:EQUATIONwhere 𝜎 (•) is the sigmoid function, 𝑗 ′ is the negative sample, Θ denotes all trainable parameters for the graph auto-encoder, and 𝜆 is the coefficient for the L2 regularization.After training the graph auto-encoder, we obtain an embedding for each item, which preserves its relational information with other items. We use these embeddings to prune the graph, to achieve a more concise graph with cleaner relations.For each item 𝑖, we rank all its first-order neighbors on the graph according to the pair-wise inner-product scores. We deem that the pairs with higher scores are more likely to be reliable and of high quality, while the pairs with lower scores are prone to be noisy or redundant. For each item, we remove 𝛽% of its first-order relations that have the lowest scores, where 𝛽 is a hyper-parameter to control the pruning ratio.After the graph pruning, we obtain a smaller graph, denoted as G. We deem that our relation pruning method is a self-supervised bootstrapping approach to denoising the graph data. More importantly, the pruning is dependent on the item embeddings that are learned from high-order graph propagation, justifying that the remained item-item relations are more reliable.The architecture of the pre-train framework encompasses two parts: multimodal encoder and pre-train objective.We inherit the backbone of BLIP [22] to encode the image and text input. Specifically, we use the visual transformer [8] as the image encoder, which inputs a sequence of image patches and uses the [𝐶𝐿𝑆] token to represent the feature of the whole. In parallel, we use BERT [5] as the text encoder, which inputs the sequence of textual tokens and makes use of the [𝐶𝐿𝑆] token to represent the entire text. Given the image 𝑣 𝑖 and text 𝑡 𝑖 of an item 𝑖, the multimodal encoder outputs its visual and textual representations, denoted as v 𝑖 and t 𝑖 , respectively.In order to make the pre-train model capture both individual items' semantics and cross-item relational information, we jointly optimized two pre-train losses, i.e., imagetext contrastive loss (ITC) and cross-item contrastive loss (CIC).• ITC Loss. We inherit it from BLIP [22] and ALBEF [23] , targeting at the cross-modality alignment between image and text. The ITC loss for item 𝑖 is represented as:EQUATIONwhere v 𝑖 , t 𝑖 are the visual and textual representations output from the multimodal encoder. Contrast(•) denotes the contrastive loss function implemented by ALBEF [23] , where a momentum encoder and soft labels from the momentum encoders are utilized. The details of the equations for contrastive loss are provided in the supplementary part. • CIC Loss. We employ the CIC loss to model the cross-item relations by pulling close a pair of related items in the representation space. For every pair of items (𝑖, 𝑗) ∈ G, we obtain their image and text representations (v 𝑖 , t 𝑖 , v 𝑗 , t 𝑗 ) through the multimodal encoder. We then harness a contrastive loss to form the CIC, denoted as:EQUATIONwhere Contrast(•) is the contrastive loss same with Equation 4.The difference between CIC and ITC is that the image-text contrastive pair of CIC is from different items, while the contrastive pair of ITC is from the same item. • Optimization. We first finetune BLIP on the image-text pairs of our pre-training dataset, to fully utilize the multiple training objectives of the original BLIP (i.e., ITM and LM) and capture the domain characteristics. Thereafter, we take the finetuned multimodal encoder to initialize our CIRP encoder. The overall pre-train objective of CIRP is obtained by combining the ITC and CIC loss, denoted as:EQUATIONIt is worth noting that even though the finetuned BLIP already well captures the semantics of image and text, the ITC loss is still essential during our pre-training. Empirical study (please refer to Section 4.3) shows that only using the CIC loss could mislead the pre-training and destroy the learned semantic modeling capability.The main objective of our pre-training framework is to perform the task of multimodal product bundling. Since this work focuses on learning high-quality item representations through the pre-train model, we prefer a representative but simple model for product bundling, thus preventing any potential bias introduced by the downstream model. Hence, we follow the previous work [40] and employ ItemKNN [36] . Specifically, given a partial bundle 𝑃 = {𝑖 𝑛 } 𝑁 𝑝 𝑛=1 that consists of a set of 𝑁 𝑝 seed items, the task of product bundling aims to complete ) Next, we obtain the multimodal representation of each item by averaging both visual and textual features x 𝑖 , formally written as:EQUATIONAs following, we aggregate the representations of items in the partial bundle using the simple average pooling strategy, resulting in a partial bundle representation p, represented as:EQUATIONThen we rank all the candidate items I based on the affinity score between p and x 𝑖 , where 𝑖 ∈ I. Finally, we get the top-k items Î that are most probably be selected to complete the input partial bundle, denoted as:EQUATIONwhere cos(•, •) denotes the cosine similarity function, and 𝑘 is the hyper-parameter during testing. It should be noted that ItemKNN is a model-free method, which means it includes no auxiliary trainable parameters while only relying on the item representations extracted by the pre-train model. We acknowledge there are more model-based bundle recommendation models, such as auto-encoder models [31] . However, given the extremely small scales of the datasets [39, 40] , the bundle construction models would severely overfit to the small datasets. We leave the study of using modelbased product bundling methods for future work. 
experiments: 
 We conduct extensive experiments on e-commerce datasets to answer the following research questions:• How does our proposed CIRP model perform on the task of product bundling, compared to various baseline methods? • Whether are the key modules in CIRP effective?• What are the key and interesting properties of CIRP?Even though product bundling is mature in business, there are few public datasets that are suitable for product bundling, especially in the multimodal setting. We utilize the Amazon review dataset for pre-training and the three bundle datasets, i.e., Clothing, Electronic, and Food [39, 40] , for product bundling. The statistics of the three datasets are shown in Table 1 . We unify all three datasets into a big one for pre-training. For the downstream task of product bundling, we use the same pretrained model and evaluate it on the three datasets separately. Since ItemKNN does not need training, we use the whole bundle dataset for testing. We follow the previous work [39, 40] and employ the Recall@k and NDCG@k to evaluate the performance of product bundling, where 𝑘 ∈ {10, 20}.4.1.2 Baselines. Since we are pioneering in using pre-train models for product bundling, there are few baselines that exactly match our task on these datasets. Therefore, we consider three streams of item representation learning baselines according to the input information, i.e., relation-only (REL-only), semantics-only (SEMonly), and relation and semantics (REL-SEM). We briefly introduce these baselines, and their implementation details are presented in the supplementary notes.• REL-only methods only use the relational data, without any image or text features. They can be broadly categorized into two types: non-sequential (MFBPR, LightGCN, and SGL) and sequential methods (Caser, GRU4Rec, and SASRec). MFBPR [35] is the pre-train model used by Sun et al. [40] . We additionally adopt two more powerful non-sequential methods of LightGCN [13] and SGL [47] . For sequential methods, we implement three classical models: the CNN-based method Caser [41] , the RNN-based method GRU4Rec [15] , and the self-attention-based method SAS-Rec [18] . To be noted, given the sequential essence, during the product bundling, we treat the input partial bundle as a sequence and directly use the pre-trained model to get the partial bundle representation p, which is then used to rank all the items. This setting is designed to faithfully and maximally retain the sequential methods' capabilities. • SEM-only models only leverage the item semantic information (i.e., the image and text), without any item-item relational data. We implement the leading methods CLIP [33] and BLIP [22] , of which the parameters are fixed. We also finetune CLIP and BLIP on our image-text pairs, resulting in the CLIP-FT and BLIP-FT methods.• REL-SEM approaches utilize both relational and semantic data, including feature fusion (FF-) methods and graph learning (GL-) methods. Both types of methods require pre-extracted multimodal features, where we use the best-performing BLIP-FT as the feature extractor. For feature fusion methods, we concatenate the multimodal features with relational features. We have FF-LightGCN and FF-SGL, using the relational embedding obtained by the REL-only model LightGCN and SGL, respectively. For graph learning methods, we use the multimodal features to initialize the node embedding and employ GCN [20] and GCL (Graph Contrastive Learning [47] ) as the graph model, resulting in the GL-GCN and GL-GCL baselines.Our model is initialized from the pre-trained BLIP [22] with BERT as text encoder and ViT-Base [8] as image encoder. The BLIP is finetuned on our image-text pairs for 10 epochs and our pre-train model is trained on the relation Clothing Electronic Food R@10 N@10 R@20 N@20 R@10 N@10 R@20 N@20 R@10 N@10 R@20 N@20 pairs for 10 epochs using 4*A5000 (24G) GPUs. We use AdamW optimizer [28] with a weight decay of 0.05. The learning rate is set to 3.0 × 10 -5 and decayed linearly with a ratio of 0.9 at the end of every training epoch. The batch size is 24 for BLIP finetuning and 16 for CIRP. For relation pruning, we train the graph autoencoder and perform graph pruning on each dataset separately.For each dataset, we split the pre-train item relations into training, validation, and testing sets with the ratio of 8:1:1. We search for the best hyper-parameter of LightGCN with grid search. Wherein, learning rate is searched from 1.0 × 10 -3 to 1.0 × 10 -4 , and weight decay is searched from 1.0 × 10 -4 to 1.0 × 10 -7 . The embedding size is tuned in the range of {16, 32, 64, 128 }, and the number of propagation layers is searched from {1, 2, 3}. After determining the best hyper-parameter for each dataset, we train the graph autoencoder with full data using the best hyper-parameter, and then we use it for graph pruning. The implementation details of all the baseline methods, including REL-only, SEM-only, and REL-SEM baselines, are presented in the supplementary file.Table 2 explicates the overall performance of our method CIRP and the baseline methods. We have the following observations. First, our approach CIRP beats all the baselines, especially in the Electronic data, which achieves over 25% relative performance improvements. Second, SEM-only methods are competitive and outperform all the REL-only methods, demonstrating their strong generalization capability. Interestingly, for non-sequential REL-only or SEM-only baselines, stronger models yield better performance. This justifies that either relational or semantic data pertains essential information for product bundling, and the performance is positively correlated with the pattern modeling capability on the pre-training dataset. However, for sequential-based REL-only methods, their performances are significantly worse than their non-sequential counterparts. This may because: 1) the items within a bundle are insensitive to the sequential order; and more importantly 2) the sequence in the pretrain dataset is too short. Third, even though our method surpasses REL-only and SEM-only methods, other REL-SEM models struggle to do it. This observation further strengthens that the pre-train framework plays a pivotal role in maximizing the utility of all the data. Finally, the effects of each type of data vary across different datasets. For example, when comparing REL-SEM and SEM-only methods, after incorporating the relational data, the performance gain on Electronic dataset is much more than those on Clothing and Food datasets. This indicates that relations are more prominent in Electronic while semantics are more crucial for Clothing and Food when bundling products. We progressively remove modules of CIRP and curate three ablated models to demonstrate the effectiveness of each key module; where "-ITC&CIC" represents removing both losses of ITC and CIC, which is identical with the BLIP-FT baseline. "-ITC" means only removing the ITC loss while keeping the CIC loss. "-RP" corresponds to the model variant without any relation pruning. The results of the ablated models are presented in Table 3 . First, "-ITC&CIC" is worse than CIRP, illustrating that the loss combination of ITC and CIC is reasonable and effective. Second and interestingly, "-ITC" underperforms not only CIRP but also "-ITC&CIC". This shows that the cross-modal alignment loss (ITC) plays a crucial role in cross-item relational modeling. Only applying CIC loss cannot well capture the cross-item relations, even worse, it will corrupt the semantic modeling capability pertained within the backbone model. Jointly optimizing both ITC and CIC loss is key to the success of CIRP. Third, "-RP" is slightly worse than CIRP, showing that relation pruning is able to enhance the performance.We are interested in multiple key properties of CIRP and conduct multiple model studies to investigate the details.To evaluate our model's generalization capability on cold-start items, we remove all the items, which exist in both pre-train and downstream datasets, from the pre-train dataset. Afterwards, we re-train our model and baselines, and evaluate them on the product bundling task. To be noted, RELonly and SEM-REL models cannot deal with cold-start items since item relations are necessary for them. Given that, we only implement the strongest SEM-only baseline BLIP-FT. The results in Table 4 show that cold-start has little impact on both BLIP-FT and CIRP-RP (with no relation pruning). We can derive that multimodal pre-train models' generalization capability, which has been verified in various vision-language downstream tasks, does also apply to the cross-item relational task of product bundling. Moreover, the performance gap between the baseline model and our method is consistent no matter whether in warm or cold-start setting. This implies that the cross-item relational patterns have approximately the same level of generalization capability with semantic patterns. 3 . First, with the pruning ratio increasing, the performance of our model first goes up and then drops. This phenomenon aligns well with our hypothesis that pruning prioritizes noisy relations. When the pruning ratio keeps increasing, more benign relations start to be removed, as a result, the performance curves turn down. Second, with the pruning ratio increasing, the convergence time drops significantly. For example, when the pruning ratio is 10%, the convergence time is only half of the original cost, while the performance slightly increases. When the pruning ratio is 90%, the convergence time decreases to 10% of the original setting, with only a marginal performance drop. The results show that the relation pruning can significantly improve the pre-train efficiency with marginal or even no performance drop.In order to demystify the working mechanism of our method, we quantitatively study the representation characteristics. Specifically, we make statistics of the average cosine similarity of random item pairs and the item pairs within the same bundle, denoted as s𝑎𝑣𝑔 and s𝑖𝑛𝑡𝑟𝑎 , respectively. We compare three models of BLIP, BLIP-FT, and CIRP, the results of which are depicted in Table 5 . Generally, the intra-bundle similarity score is higher than that of random item pairs, for all sthree methods and datasets. This implies that the items within the same bundle are semantically close to each other. When comparing BLIP and BLIP-FT, the average item similarity score decreases. This shows that finetuning BLIP enforces the items scatter and occupy as much space as possible, thus endowing the model's improved representation capability to discriminate trivial differences, which could not be captured before finetuning. For the intra-bundle cross-item similarity, it decreases significantly on BLIP-FT while increases on CIRP conversely, explicitly illustrating that our proposed cross-item relational pre-train can re-gather the items within the same bundle.To further qualitatively demonstrate how the embedding space differs w.r.t. different models, we randomly select 20 bundles and project their included items' embeddings into a 2D plane, as shown in Figure 4 . Clearly, we can find out that BLIP-FT makes the gathered items expand to a larger space, however, it simultaneously pushes away the items within the same bundle. By applying cross-item relational pre-train, the item distribution becomes more compact, especially for items within the same bundle. This observation aligns well with the quantitative results presented in Table 5 .To directly demonstrate how the model performs on product bundling, we pick two pairs of items and present them in Table 6 . Each item pair is from the same bundle, where the first item is provided as the query while the second item is the ground-truth item for product bundling. The number in Table 6 corresponds to the ranking of the second item in the candidate set for different models. We can see that by finetuning BLIP and applying crossitem relational pre-training, the ranking of the second item keeps increasing. Interestingly, the two cases illustrate two representative bundling strategies. The first item pair is curated based on semantic similarity, that's why BLIP-FT already works very well. The second item pair is bundled based on implicit relation, i.e., functionally compatible, which is hard to be identified from the semantic features. That's why both BLIP and BLIP-FT fail to recognize the relations between these two items. However, our method CIRP is able to capture such implicit relations and ranks the HDMI Cable high when given a Streaming Media Player as the query. 
conclusion and future work: 
 We explored an interesting task of integrating the cross-item relations into a multimodal pre-train model for product bundling. We developed a novel framework CIRP that leverages both intra-item cross-modality constrastive loss (ITC) and cross-item contrastive loss (CIC). More importantly, to alleviate the noise in the relations as well as reduce the training cost, we designed a relation pruning method to enhance and accelerate the pre-train. We tested our method on large-scale e-commerce datasets and evaluated its performance on three product bundling datasets. Experimental results demonstrated the effectiveness and efficiency of our method.Despite the great progress, there are multiple potential directions to be explored in the future. First, current work only focuses on the first-order direct relations among items, while the higher-order relations that contain additional crucial information should also be considered. Second, our method only considers one type of itemitem relations, and more types of heterogeneous relations could be more helpful yet challenging to model. Therefore, integrating heterogeneous relations, such multiple behaviors, cross-domain relations, and relations from knowledge graph, into the pre-train model should be investigated in the future. Third, cross-item relational pre-training is a general paradigm and can be generalized to more downstream tasks, such as session-based and sequential recommendations. Finally, it is worthwhile to embrace the powerful LLMs, for example, we can directly use LLMs for product bundling or incorporate relations into the LLMs to boost the performance. 
 
 Appendix: https://en.wikipedia.org/wiki/Association_rule_learning 


Decision: This paper proposes a novel framework for item representation in product bundling, Cross-Item Relational Pre-training (CIRP). The topic is relevant and important, well suited to the conference. Even though one reviewer is not convinced, the other three align towards acceptance, due to its performance and the scalability of the approach.

Reviews: review: paper_summary: paper_summary: this paper introduces a new framework designed to improve item representation learning in the context of product bundling. The approach leverages multimodal encoders and novel pre-training objectives to capture both individual item semantics and cross-item relationships, particularly focusing on the cold-start issue where items lack explicit relational data. The key components of CIRP include cross-item contrastive loss (CIC) and individual item's image-text contrastive loss (ITC), along with a relation pruning module to filter out noisy or redundant relations. Experimental results show that CIRP outperforms several baselines, demonstrating its effectiveness in enhancing the generalization capability of models on cold-start items.
strength_weakness: strength_weakness: Strengths of the CIRP model include its ability to integrate cross-item relational modeling into multimodal encoders, its performance improvements in the cold-start setting, and the successful demonstration of the importance of both ITC and CIC losses in improving item representations. The model also highlights the significance of relation pruning in refining the learned representations.
limitations: limitations: 1 Figure 1 does not contain any motivation about why the authors chose this style for learning representations. Also I think in this important figure, the authors should give some insights about the Multimodal Product Bundling, where the proposed framework could apply. 
2 Vision Language Model is a keyword, but its connection is weak to the context of this paper.
3 The technical contribution is too weak for MM. In my opinion, the authors only propose combining two representation learning in a pretrain model for recommendation. Also, the pre-training work could be replaced by LLM models easily.
paper_summary: paper_summary: The paper presents a novel framework for enhancing product bundling in e-commerce through improved item representation learning, aiming to develop high-quality item representations capturing both individual semantics and cross-item relations for effective product bundling as Current methods (feature fusion and graph learning) struggle with cross-modal alignment and cold-start items. They Introducing Cross-Item Relational Pre-training (CIRP), which uses a multimodal encoder and contrastive losses to integrate relational information into item representations.
strength_weakness: strength_weakness: 1.Improved Cross-Modal Alignment:
Unlike previous methods that separately learn features from multiple modalities, CIRP ensures in-depth cross-modal alignment and enhancement by using multimodal encoders and contrastive losses .

2.Effective Handling of Cold-Start Items:
CIRP can generate relation-aware representations even for new items that do not have pre-existing relational data, addressing a significant limitation of earlier methods .

3.Noise Reduction and Efficiency:
The framework includes a relation pruning module that removes noisy and redundant relations, which not only reduces computational cost but also improves the efficiency of the pre-training process. This allows CIRP to maintain high performance while significantly cutting down on pre-training time .

4.Superior Performance in Product Bundling:
Experimental results on three large-scale e-commerce datasets demonstrate that CIRP outperforms leading representation learning methods, proving its effectiveness in real-world applications .

5.Scalability:
By efficiently handling a large amount of relational data and leveraging advanced multimodal encoders, CIRP is well-suited for the ever-growing number of products in e-commerce platforms, making it highly scalable .
limitations: limitations: 1. Lack of Novelty: The proposed pre-training item representation lacks innovation, as similar ideas have been explored in previous works [1,2]. The authors merely apply two existing models without modification for separate modality representation and align them using contrastive loss.

   [1] Hou, Yupeng, et al. "Towards universal sequence representation learning for recommender systems." Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.
   
   [2] Li, Youhua, et al. "Multi-Modality is All You Need for Transferable Recommender Systems." arXiv preprint arXiv:2312.09602 (2023).

2. Figure 2 lacks a clear description of the icons representing the inputs to the separate multimodal encoder. Additionally, the absence of a 'cls' token in the image encoder input, as mentioned in section 3.3.1, is not addressed. Recall@10 is used to evaluate the effect of the relation pruning rate, but Recall@20 is used in most other tables, leading to inconsistency.


3. Cold-Start Item Experiment: In section 4.4, the comparison for cold-start items is only made against the strongest SEM-only baseline BLIP-FT, which may not be the most appropriate choice for cold-start scenarios. A more comprehensive comparison with other baselines would provide better insights.
paper_summary: paper_summary: This paper proposes a novel CIRP method for item representation learning in product bundling. This method leverages both the cross-item contrastive loss (CIC) and individual item’s image-text contrastive loss (ITC) as the pre-train objectives. Extensive experiments on three publicly datasets demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The motivation is novel. The paper achieves a simple pre-training framework to naturally integrate
cross-item relations into the multimodal encoder by the CIC loss, while preserving the in-depth aligned multimodal semantics by the ITC loss. 
2. The experiment results demonstrate the effectiveness of the proposed method, especially for cold-start items.
3. The manuscript is easy to follow as it is well-organized.
limitations: limitations: 1. A negative sample in ITC loss may be a positive sample in CIC loss. So, the design of negative samples is very important. Further clarification is needed.
2. The pre-training time seems to have a linear relationship with the number of datasets, and the pre-trained model seems unsuitable for downstream new datasets.
3. Is relation pruning pre-trained separately for each dataset?
4. In table 1, how is the bundle of each dataset constructed?
5. How can different datasets be mutually enhanced through pre-training?
paper_summary: paper_summary: This paper introduces a novel framework CRIP aimed at improving item representation learning for product bundling in e-commerce.
strength_weakness: strength_weakness: 1. The paper addresses multimodal product bundling, an intriguing and relevant topic.

2. This innovative approach fills a significant gap in the current methodologies which focus primarily on semantic modeling without adequately addressing cross-item relations.

3. CIRP employs both cross-item contrastive loss (CIC) and individual item's image-text contrastive loss (ITC) as pre-training objectives. This dual approach ensures that the model captures both the semantic details of individual items and the relational aspects between items, making it highly effective for product bundling tasks.

3. It demonstrates significant improvements over baseline approaches and provides a detailed analysis.

4. The proposed relation graph pruning approach significantly speeds up pre-training time.

5. The paper is well-written and well-organized

6. The authors have committed to releasing the code.
limitations: limitations: Although the authors employ graph pruning to reduce training costs significantly, fully fine-tuning the multimodal encoders remains costly. The authors might consider exploring more efficient methods for this process in future research, such as [1].

[1] "IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFT." arXiv preprint arXiv:2404.02059 (2024).