ABSTRACT

CCS CONCEPTS

KEYWORDS

INTRODUCTION
1
With the progress of data mining and machine learning, session- based recommendation, which employs customer session data to forecast their next click, is gaining momentum. Sequential recom- mender [3, 7] has garnered popularity owing to its capability of capturing users' transient and enduring preferences, rendering it widely applicable across diverse recommendation contexts. Numerous methodologies have been proposed to model sequen- tial recommendation scenarios, spanning from early matrix factor- ization approaches (e.g., FPMC [9]) to more recent sequence neural networks (such as GRU4Rec [3], STAMP [7], and Transformer [10]). While they may differ, their underlying principles are akin: captur- ing behavioral patterns that mirror user preferences based on the sequential access structure and establishing a mapping between user preferences and specific items grounded in their interaction relationships. However, the dependence on item interaction re- lationships curtails their efficacy when faced with the cold-start predicament of insufficient user interactions with certain items. Recent research has explored the use of natural language text, such as item titles and descriptions, to gain domain knowledge and address the challenges mentioned earlier [4, 6]. This approach involves utilizing pre-trained language models like BERT to obtain text representations and learning how to transform these represen- tations into item representations. While previous attempts have shown promise, there are still unresolved challenges that need to be addressed: (1)Insufficient integration of multi-modal information: Existing methods [4] combine the text representation of an item with its corresponding item representation, but this approach can introduce interference between modalities. For example, if an item has dense interaction data and the item embedding's representation capability surpasses that of the text representation, incorporating the text representation may negatively impact the expressive power of the item representation. (2) Partial Semantic Synchronization: Cur- rent approaches often treat candidate items as targets, resulting in parameter updates that only align the sequential representation with item representations that match sequence preferences. As a result, the alignment between item representations and sequential representations is considered, while the alignment in the reverse direction is overlooked. In order to address the aforementioned challenges, we propose an adaptive hybrid multimodal framework for session-based rec- ommendation. Figure 1 provides an illustrative overview of the framework. This innovative framework employs multiple distinct models to encode autonomous representations of item IDs and their corresponding text sequences derived from a user's historical items. These representations are then leveraged to generate separate pre- dictions for the subsequent click. To achieve dynamic fusion of the predictions, we employ the CatBoost algorithm, leveraging its capa- bility to compute the conditional probability distribution of a class based on specific feature conditions. Additionally, we introduce Decoupled Contrastive Learning to enhance the representation ca- pability of item text. This method separates the negative sample space into two independent subspaces: sequential representation and item representation. By decoupling these two spaces and utiliz- ing a dual-momentum queue [1], a substantial number of negative samples are updated and stored. Two types of contrastive learning losses are then computed. This process ultimately enables bidirec- tional deep alignment between sequential representations and item representations. Furthermore, our team actively participated in the KDD CUP 2023, where we conducted extensive evaluations of our proposed solution using the Amazon-M2 dataset [5]. Through our rigorous experiments, we achieved competitive results, securing a 5th place ranking in task 1.
METHODOLOGY
2
2.1 ItemCF. Despite the remarkable success of deep-learning- 2.1.1 based approaches in recommender systems, traditional methods like collaborative filtering (CF) continue to find extensive applications on online platforms, owing to their irreplaceable strengths. These strengths include interpretability, minimal reliance on hardware, and high efficacy in both training and deployment. In this competition, we incorporated several crucial factors into the construction of the item similarity matrix, including position distance information, directional cues in interactions, and normal- izations based on item popularity. Formulas 1-4 encapsulate the fundamental concepts we employed to achieve these outcomes.
Our research presents a unique approach by incorporating po- sitional weights. It stems from the observation that the last item clicked by users, after multiple interactions, often reflects their meticulously chosen ultimate preference.
2.1.2  ID-based Embedding : GRU. Recurrent neural networks are the most popular choice for modeling sequential data. We designed a customized RNN which achieves a 0.379 public leaderboard score. The model architecture is shown in Figure 2. During the phase of shared embedding lookup on the user's last N clicked items, we utilize mean pooling on the embeddings of the items as well as their associated side information (such as price and brand) to derive fused embeddings. Subsequently, these embeddings are fed into a GRU layer for the extraction of sequential features. To address the challenge of vanishing gradients, we introduce a sequence-level residual connection block. This block conducts mean pooling on the initial fused embeddings prior to the GRU layer. It is then added to the output of the GRU layer to generate the final sequential representation. Differing from the approach adopted in previous studies [10], which involves mean-pooling the GRU layer outputs at each time step, our method preserves the original fused embeddings, ensuring facilitating smoother gradient propagation. 2.1.3 Language Embedding. Contrastive learning has found wide applications in information retrieval problems [2, 8]. Recently, its potential has also been explored in the field of session-based rec- ommendation [6]. Motivated by these studies, this paper proposes a vector-based approach for item representation. The model ar- chitecture is shown in Figure 3.We segment the attribute texts of items using [SEP] and encode them using a pre-trained BERT model. This encoding process includes both previously browsed items and the target item. Finally, we perform average pooling on the out- puts of the first layer and apply L2 normalization to obtain the representation vector for each item. Each BERT encoder has a momentum encoder, whose param- eters θ are updated by exponential moving average of the base encoder as follows: Θ t ← βΘ t −1 + (1 − β)Θ base . In each iteration step t, we maintain two memory queues: one for sequence represen- tations and the other for target item representations. These queues store K vectors, which are encoded by the momentum encoder from the most recent batches. With each optimization step, the oldest vectors in the memory queues are replaced by the vectors from the current batch. The momentum coefficient β ∈ [0, 1], usu- ally set close to 1 (e.g., 0.999), ensures consistency across batches. Additionally, setting K to a large value (>10000) ensures an abun- dant supply of negative samples, thereby promoting the learning of robust representations. To train the encoder, we employ the ArcCon Loss [11], a loss function that introduces a hyperparameter m to add a decision boundary, promoting compactness among sentence representations with the same semantics and increasing the dissimilarity between representations of sentences with different semantics:
In Equation 5, the variable s denotes the temperature hyperparame- ter. In essence, Equation 5 can be perceived as a (K+1)-way softmax classification, where y = y i represents the positive sample, and the items stored in the queue { y i } i ž j represent the negative samples. It is important to note that gradients do not propagate through the momentum encoder or the storage queue during backpropagation. An important contribution in our study is the introduction of the training method called Decoupled Contrastive Learning. We divide the set of negative samples into two separate subsets, each containing all sequential representations and target item represen- tations, respectively. By doing so, we can construct a symmetrical contrastive learning loss with the following form: min λ 1 L(xs,xt) + λ 2 L(xt,xs) + λ 3 L(xs,xs) + λ 4 L(xt,xt)    (6) The hyperparameter λ serves to harmonize the losses of each task. Upon completion of the training process, we discard the momentum encoders and memory queues, retaining solely the base encoders for the computation of sentence representations.
Multimodal Fusion Rerank
2.2
In this section, we retrieve items for each session using three re- trieval methods, resulting in three sets of retrieved items. To com- bine the scores from the retrieval methods, we multiply them to- gether, represented as yi = Π t =1 yi, where yi is the score for each retrieval method. From the fused scores, we select the top 120 items as the preliminary retrieval results. Next, we integrate the scoring of retrieval results from different modalities and perform fine-grained feature engineering. We employ the CatBoost algorithm, which is based on decision trees and can effectively merge the retrieval results from different modalities by adaptively combining features. 2.2.1  Item Hot Features. The fundamental characteristics of item view frequencies in the dataset are considered. We have also incor- porated the sort orders feature to enhance the model's ability to learn the ranking task with greater precision. 2.2.2 Session Features. In harmony with item features, we conduct composite statistics based on the item characteristics that users en- gage with within each session. This includes computing aggregate metrics such as the mean popularity and average price of all items within a session. Graph Features.  This study constructs a co-occurrence graph 2.2.3 based on itemcf's co-occurrence relationship matrix. Each item is a node in the graph, and the weights between items serve as edge weights. By examining the degree relationships between items and their neighboring nodes, these features describe the importance and connectivity of candidate items within the co-occurrence graph. Our analysis includes node importance measures like PageRank, centrality features including betweenness centrality, Katz centrality, and degree centrality, as well as neighbor node relationship char- acteristics such as mean, count, maximum, and standard deviation
EXPERIMENTS
3
Datasets
3.1
CONCLUSION AND FUTURE WORK
ACKNOWLEDGMENTS
Overall Performance
3.2

of edge weights. Integrating these features enhances the model's understanding of candidate item characteristics. We applied data augmentation techniques referenced in the litera- ture [3,8] to enhance the Amazon-M2 dataset across the UK, DE, and JP regions. Through a five-fold cross-validation method, the data was partitioned into training and validation sets. Notably, the validation set exclusively comprised sessions without data augmen- tation, with the final item serving as the label. The validation set not only served for evaluating the model's performance but also facilitated training set generation during the reranking stage. To ensure efficiency, the validation was conducted exclusively on the UK region's data. This paper introduces a hybrid multimodal recommendation system that combines itemID and language modalities for item retrieval. We incorporate the catboost algorithm and employ fine-grained feature engineering to merge the retrieval results from both modalities. Our future work includes investigating the applicability of our algorithm in real-world e-commerce scenarios at a large scale. This work is jointly supported by the National Natural Science Foundation of China (U19B2028), the National Science and Tech- nology Major Project of the Ministry of Science and Technology of China (2022YFB4300603), the Sichuan Science and Technology Program (2023YFG0151) and the Development of a Big Data-based Platform for Analyzing the Coupling Relationship of Strip Produc- tion Processes Project. Our approach achieved considerably performance gain over the baseline solution, and ranked top-5 in task-1. The main results are shown in Table 1. [1] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 (2020). Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In Empirical Methods in Natural Language Processing (EMNLP). [3] Balazs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2015. Session-based recommendations with recurrent neural networks. arXiv preprint arXiv:1511.06939 (2015). Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong [4] Wen. 2022. Towards universal sequence representation learning for recommender systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Dis- covery and Data Mining. 585–593. [5] Wei Jin, Haitao Mao, Zheng Li, Haoming Jiang, Chen Luo, Hongzhi Wen, Haoyu Han, Hanqing Lu, Zhengyang Wang, Ruirui Li, Zhen Li, Monica Xiao Cheng, Rahul Goutam, Haiyang Zhang, Karthik Subbian, Suhang Wang, Yizhou Sun, Jiliang Tang, Bing Yin, and Xianfeng Tang. 2023. Amazon-M2: A Multilingual Multi-locale Shopping Session Dataset for Recommendation and Text Generation. (2023). [6] Jiacheng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang, and Julian McAuley. 2023. Text Is All You Need: Learning Language Representations for Sequential Recommendation. arXiv preprint arXiv:2305.13731 (2023). [7] Qiao Liu, Yifu Zeng, Refuoe Mokhosi, and Haibin Zhang. 2018. STAMP: short- term attention/memory priority model for session-based recommendation. In Proceedings of the 24th ACM SIGKDD international conference on knowledge dis- covery & data mining. 1831–1839. [8] Yīngwei Ma, Yue Yu, Shanshan Li, Zhouyang Jia, Jun Ma, Rulin Xu, Wei Dong, and Xiangke Liao. 2023.  MulCS: Towards a Unified Deep Representation for Multilingual Code Search. In 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 120–131. [9] Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010. Factor- izing personalized markov chains for next-basket recommendation. In Proceedings of the 19th international conference on World wide web. 811–820. [10] Benedikt Schifferer, Jiwei Liu, Sara Rabhi, Gilberto Titericz, Chris Deotte, Gabriel De Souza P. Moreira, Ronay Ak, and Kazuki Onodera. 2022. A Diverse Models Ensemble for Fashion Session-Based Recommendation.  In Proceedings of the Recommender Systems Challenge 2022. 10–17. [11] Yuhao Zhang, Hongji Zhu, Yongliang Wang, Nan Xu, Xiaobo Li, and Binqiang Zhao. 2022. A contrastive framework for learning sentence representations from pairwise and triple-wise perspective in angular space. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 4892–4903. We have delved into exploring the impact of applying Decoupled Contrastive Learning on model performance. Table 2 presents the performance of the model in terms of the MRR@100 metric after three epochs of training.It is important to note that the default hy- perparameters utilized in this study were set as λ 1 :λ 2 :λ 3 :λ 4 =0.35:0.35: 0.15.0.15. Table 2: The influence of Decoupled Contrastive Learning. We observed that by decoupling within the negative sample space and employing symmetric contrastive loss, the model's per- formance can be enhanced by 0.011. Additionally, adjusting the weight ratios of each loss function also influences the model's ef- fectiveness. As illustrated in Table 3, we evaluated the performance of the Multimodal Fusion Rerank methodology. Initially, we presented the performance metrics of various standalone models for comparative purposes against Multimodal Fusion. The experimental findings demonstrate that incorporating the catboost algorithm and conduct- ing meticulous feature engineering both contribute significantly to elevating the performance of Multimodal Fusion. 4