Mitigating Sample Selection Bias with Robust Domain Adaption in Multimedia Recommendation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-29 07:20:43

Abstract: Industrial multimedia recommendation systems extensively utilize cascade architectures to deliver personalized content for users, generally consisting of multiple stages like retrieval and ranking. However, retrieval models have long suffered from Sample Selection Bias (SSB) due to the distribution discrepancy between the exposed items used for model training and the candidates (almost unexposed) during inference, affecting recommendation performance. Traditional methods utilize retrieval candidates as augmented training data, indiscriminately treating unexposed data as negative samples, which leads to inaccuracies and noise. Some efforts rely on unbiased datasets, while they are costly to collect and insufficient for industrial models. In this paper, we propose a debiasing framework named DAMCAR, which introduces Domain Adaptation to mitigate SSB in Multimedia CAscade Recommendation systems. Firstly, we sample hard-to-distinguish samples from unexposed data to serve as the target domain, optimizing data quality and resource utilization. 
Secondly, adversarial domain adaptation is employed to generate pseudo-labels for each sample. To enhance robustness, we utilize Exponential Moving Average (EMA) to create a teacher model that supervises the generation of pseudo-labels via self-distillation. 
Finally, we obtain a retrieval model that maintains stable performance during inference through a hybrid training mechanism. We conduct offline experiments on two real-world datasets and deploy our approach in the retrieval model of a multimedia video recommendation system for online A/B testing. Comprehensive experimental results demonstrate the effectiveness of DAMCAR in practical applications.

Main paper: 
 introduction: 
 Multimedia recommendation is essential for many mobile Internet platforms, aiming to accurately present multi-modal items aligned with users' interests [36, 40, 53, 57, 58, 62, 64] . To support the increasing demand for online deployment, industrial scenarios have widely adopted cascade architectures [38, 48, 54, 56] . Such architectures utilize a funnel-shaped filtering strategy-from broad retrieval to precise ranking-to efficiently deliver personalized content for users. Figure 1 illustrates a typical multimedia cascade recommendation system comprising two stages: retrieval and ranking 1 . Simple models are employed in the retrieval stage to swiftly filter out irrelevant items from a large candidate pool [10, 28, 49] . The ranking stage then employs sophisticated models for precise ranking, selecting the top few as winning candidates [3, 44, 45, 65, 66] . Finally, those exposed items with user behaviors (e.g., finish playing or not) are recorded for continuous model training [7, 19] . However, in cascade architectures, retrieval models face the challenge of Sample Selection Bias (SSB) [24, 33, 36, 61] . As illustrated in Figure 1 , this bias originates from a notable phenomenon: There exists a significant discrepancy between the distribution of exposed items used for model training and the distribution of retrieval candidates (almost unexposed) during inference. Such bias impedes the ability of retrieval models to comprehensively learn the distribution characteristics across the entire data space, affecting models' generalization performance and failing to reflect users' real interests [7, 24] . In essence, SSB violates the foundational traininginference consistency assumption [19, 48] , like requiring a student to take an exam beyond the prescribed syllabus.To mitigate SSB, traditional methods utilize retrieval candidates for data augmentation, indiscriminately treating unexposed data as negative samples [7, 24, 27] . Although reducing the distribution discrepancy, they overlook the potential positive samples within unexposed data. Such mislabeling introduces training noise, diminishing the quality of decision-making during inference.Another line of research attempts to collect unbiased datasets through specific uniform strategies and then guide the models trained on exposed data by knowledge transfer [33-35, 37, 63] . However, these solutions encounter prohibitive data collection and processing costs, making them impractical for real-world applications [36] . Furthermore, the obtained unbiased datasets may be insufficient for industrial models with billions of parameters [19] .Given that the primary cause of SSB is the distribution inconsistency between training and inference data, we draw on unsupervised domain adaptation techniques to tackle this issue [41, 60] . Unsupervised domain adaptation has demonstrated notable success in fields like image recognition [4, 26] and natural language processing [2, 5] , by aligning distributions of the labeled source domain and the unlabeled target domain to ensure training-inference consistency. However, when applying it to resolve SSB, researchers face several key challenges: (i) Unsupervised domain adaptation requires unbiased datasets as target domains for model training, which is often difficult to achieve in practical applications [36, 56] . (ii) There is lacking effective mechanisms to guarantee the quality of the pseudo-labels generated for target domains, while they are critical to model performance [18, 51] .In this paper, we propose a novel debiasing framework named DAMCAR, which introduces Domain Adaptation to mitigate SSB in Multimedia CAscade Recommendation systems. DAMCAR consists of three modules: (i) Target domain generation. We construct a directed weighted bipartite graph from user historical behaviors and employ a random walk algorithm to select hard-to-distinguish samples from unexposed data to serve as the target domain. This method not only obtains informative samples for model training but is also resource-friendly by focusing only on a subset of unexposed data rather than all of them. (ii) Robust pseudo-label generation. We design a label generator that introduces an adversarial network between the source domain (exposed data) and the target domain (sampled unexposed data). This encourages the model to learn feature representations that are indistinguishable across data distributions, thereby generating unbiased pseudo-labels for each sample. Additionally, to improve the model's robustness and the reliability of pseudo-labels, we use Exponential Moving Average (EMA) to create a teacher model that guides the generation of pseudo-labels via self-distillation. (iii) Hybrid training. We utilize a hybrid training mechanism that combines both pseudo-labels and the scores from the ranking model to obtain a retrieval model that maintains stable performance during inference. Compared with traditional solutions, DAMCAR provides customized samples and corresponding labels to the retrieval model at the end of each training phase without affecting its efficiency. To summarize, we make the following contributions in this paper:• We propose DAMCAR, aimed at mitigating SSB by leveraging domain adaptation to bridge the gap between training and inference data distributions for retrieval models, which focuses on the generation of the target domain and pseudo-labels. • To enhance robustness and reliability, we use EMA to create a teacher model that steers the generation of pseudo-labels for target domain samples via a self-distillation mechanism. • We conduct offline experiments on two datasets and implement our approach in a multimedia video recommendation system for online A/B testing. Experimental results demonstrate the effectiveness of DAMCAR in real-world applications. 
related works 2.1 multimedia cascade recommendation: 
 In industrial scenarios, the online deployment of multimedia recommendation systems necessitates balancing effectiveness and efficiency. While sophisticated models offer superior recommendation performance, they tend to introduce high latency [36, 43, 57] . Conversely, simple models with limited capacity can efficiently handle massive requests [28, 67] . Thus, a prevalent approach is to adopt funnel-shaped cascade architectures [17, 25, 38, 48, 54, 56] . Simple models are employed in the early retrieval stage to swiftly filter out irrelevant items from a large candidate pool [10, 28, 49] , while sophisticated models are then utilized in the ranking stage for precise ranking [3, 44, 45, 65, 66] . Some studies explore cost-aware cascading systems that incorporate computational cost to determine model assignments for different stages [8, 54, 59] . Other solutions optimize cascade systems through gradient transfer, stage combination, feature sharing, and item aggregation [15] [16] [17] 25] . Despite these advancements, most existing multimedia cascade recommendation systems suffer from SSB due to the distribution discrepancy between training and inference data [48, 56] , potentially impacting model performance. In this paper, we focus on mitigating SSB by providing customized samples and corresponding labels to retrieval models without compromising the training efficiency.Many recent efforts are devoted to mitigating SSB for improved recommendation performance [19, 27, 33, 48, 63] . (i) Some approaches focus on the augmentation of training data. They either employ all retrieval candidates, treating unexposed data as negative samples [7, 24] or utilize random negative sampling and hard negative sampling to select negative samples [11, 27, 29, 48] . These methods reduce the distribution discrepancy, but overlook the potential positive samples within unexposed data, introducing training noise.(ii) Another line of research is to generate unbiased datasets by employing specific uniform strategies for content delivery and collecting user feedback [33, 37] . The unbiased datasets are used to train imputation models for unexposed data [1, 6, 13, 63] or guide the models trained on exposed data by knowledge transfer [33] [34] [35] 37] . However, these approaches incur high costs in unbiased dataset collection and processing that may hurt the user experience, rendering them impractical for real-world applications [19, 36] . (iii) Unsupervised domain adaptation is a potential research direction for mitigating SSB [4, 26] , which trains a model that performs well on the unlabeled target domain by using labeled samples on the source domain. For example, in [56] , a scoring model is trained on exposed data (source domain) and generates pseudo-labels for an unbiased dataset (target domain). The retrieval model is trained on the target domain, ensuring the training-inference consistency. However, it requires an unbiased dataset as the target domain for model training, which is often difficult to achieve in practical applications [19, 36] . Moreover, there is currently lacking effective mechanisms to ensure the quality of the pseudo-labels generated for target domains [18, 51] . Building upon existing research, we leverage unexposed data to generate a target domain with informative samples and develop a robust pseudo-label generation mechanism, aiming to mitigate SSB from two aspects of data and labels. Pseudo Labels Ground-Truth Labels𝑦 = 1 𝑆 !"# EMA Stop GradientFigure 2 : The architecture of DAMCAR, which consists of three modules: (i) Target domain generation (left). We generate a highquality unexposed sample set S 𝑠𝑎𝑚 from S \ S 𝑒𝑥𝑝 to serve as the target domain, by constructing a directed weighted bipartite graph and employing a random walk algorithm. (ii) Robust pseudo-label generation (middle). We generate pseudo-labels for each sample through adversarial domain adaptation and then use EMA to create a teacher model that guides the generation via self-distillation for enhanced robustness. (iii) Hybrid training (right). We train the retrieval model Ret(•) through a hybrid mechanism that combines modified pseudo-labels with the scores obtained from the ranking model Rank(•). 
preliminaries: 
 In this section, we provide a formal definition of the problem along with the necessary notations. For a multimedia recommendation system, it includes a user set U = {𝑢 1 , . . . , 𝑢 𝑁 } and an item set I = {𝑖 1 , . . . , 𝑖 𝑀 }, where 𝑁 represents the number of users and 𝑀 represents the number of items. We define S = U × I as the entire sample set, encompassing all user-item pairs. In the cascade architecture, the retrieval model Ret(•) initially scores the entire sample set S and selects a subset with high scores. The subset is then forwarded to the ranking model Rank(•) for further processing, ultimately delivering recommendation results for users. After collecting user interactions, an exposed sample setS 𝑒𝑥𝑝 = {(𝑢, 𝑖, 𝑦, 𝑒 = 1) | 𝑢 ∈ U, 𝑖 ∈ I, 𝑦 ∈ {0, 1}} is produced:• 𝑢: the features of a user, which include personal attributes and historical behaviors. • 𝑖: the features of an item, which include item attributes like ID, texts, images, and other multi-modal information. • 𝑦: the ground-truth label of an observed user behavior, e.g., finish playing (𝑦 = 1) or not (𝑦 = 0). • 𝑒: the exposure label, where 𝑒 = 1 denotes item 𝑖 has been exposed to user 𝑢 with corresponding behavior 𝑦.The ranking model Rank(•) is trained on the exposed sample set S 𝑒𝑥𝑝 [19, 48, 56] , which typically employs a sophisticated structure to deeply interact with user interests and item features for precise ranking, e.g., Deep Factorization-Machine (DeepFM) [22] and Deep & Cross Network (DCN) [55] . On the other hand, the retrieval model Ret(•) adopts a simple dual-tower structure to match user features with item features, focusing on efficiently selecting potentially user-interested items from a large candidate pool, e.g., Deep Structured Semantic Model (DSSM) [28] . However, in the cascade architecture, the retrieval model Ret(•) suffers from SSB, as highlighted in previous studies [24, 33, 36, 61] . This issue arises from a notable phenomenon: Training Ret(•) on a limited labeled subset S 𝑒𝑥𝑝 comprising only exposed items while requiring it to score the entire sample set S with mostly unexposed items during inference. SSB causes the training-inference inconsistency, which may impact the model's recommendation performance. 
method: 
 In this paper, we propose a debiasing framework named DAMCAR to mitigate SSB in multimedia cascade recommendation systems, which consists of three modules: (i) Target domain generation ( §4.1), (ii) Robust pseudo-label generation ( §4.2), and (iii) Hybrid training ( §4.3). Figure 2 illustrates the architecture of DAMCAR.To optimize data quality and resource utilization, we collect an unexposed sample set S 𝑠𝑎𝑚 = {(𝑢, 𝑖, 𝑒 = 0) | 𝑢 ∈ U, 𝑖 ∈ I} from S \ S 𝑒𝑥𝑝 to serve as the target domain, rather than utilizing all of them. In contrast to existing sampling methods [11, 27, 29, 48] , our focus is on selecting hard-to-distinguish samples that are difficult for the model to judge, containing more information in enhancing training. These unlabeled samples are equipped with pseudo-labels in §4.2, instead of being treated indiscriminately as negative. Specifically, we leverage graph connectivity to identify the degree to which unexposed samples are close to the model's decision boundary. This graph-based perspective compensates for the insufficient exposure at the individual user-item pair level by exploiting the inherent multi-hop relationships in the network structure. It enables the computation of connectivity between users and items, even without direct links, which is particularly crucial for samples that are inherently hard-to-distinguish.We start by constructing a directed weighted bipartite graph with the exposed sample set S 𝑒𝑥𝑝 , where nodes represent users or items and edges represent interactions. The weight 𝑣 𝑢,𝑖 denotes the degree of engagement between user 𝑢 and item 𝑖, such as watch time. We define P (𝑢 → 𝑖) as the transition probability from user 𝑢 to item 𝑖, which is calculated as the ratio of the weight 𝑣 𝑢,𝑖 to the sum of weights for all items that user 𝑢 has interacted with:EQUATIONwhere I (𝑢) is the set of item nodes directly connected with user 𝑢.Similarly, the transition probability from item 𝑖 to user 𝑢, denoted as P (𝑖 → 𝑢), is determined by comparing the weights 𝑣 𝑢,𝑖 with the sum of weights from all users who have engaged with item 𝑖:EQUATIONwhere U (𝑖) is the set of user nodes directly connected with item 𝑖. Now, we apply a random walk algorithm with restart mechanisms to obtain hard-to-distinguish samples for each user from S \ S 𝑒𝑥𝑝 . We center on user 𝑢 with a radius of 𝑅 and randomly visit nodes along the path of 𝑢 → 𝑖 → 𝑢, according to the transition probability P. When the next hop exceeds the radius 𝑅, the process returns to the starting node 𝑢 for resampling, which continues until 𝐻 visits are completed. Let T (𝑢, 𝑖) represent the number of times each item 𝑖 is visited. We denote the set of visited exposed items as I 𝑢 = {𝑖 | (𝑢, 𝑖, 𝑦, 𝑒 = 1), T (𝑢, 𝑖) ≥ 1} and the set of visited unexposed items as I * 𝑢 = {𝑖 * | (𝑢, 𝑖 * , 𝑒 = 0), T (𝑢, 𝑖 * ) ≥ 1}. Then, we define hard-to-distinguish samples as follows:EQUATIONThat is, when the number of visits T (𝑢, 𝑖 * ) for an unexposed item 𝑖 * ∈ I * 𝑢 is greater than or equal to the average number of visits for exposed items {𝑖 | 𝑖 ∈ I 𝑢 }, the sample (𝑢, 𝑖 * , 𝑒 = 0) will be added to the unexposed sample set S 𝑠𝑎𝑚 . |I 𝑢 | represents the number of visited exposed items for user 𝑢.Here, we define exposed samples as the source domain S 𝑒𝑥𝑝 = {(𝑢 𝑠 , 𝑖 𝑠 , 𝑦 𝑠 , 𝑒 = 1)} and sampled unexposed samples as the target domain S 𝑠𝑎𝑚 = {(𝑢 𝑡 , 𝑖 𝑡 , 𝑒 = 0)} for greater clarity. We introduce adversarial domain adaptation to learn exposure-independent pseudolabels for retrieval model training.Initially, we employ a unified encoder E (•) to extract features from samples in both the source and target domains, yielding their deep representations z 𝑠 and z 𝑡 , respectively:EQUATIONThen, we feed the deep representations z 𝑠 and z 𝑡 into the label generator Φ(•) to generate the pseudo-labels ŷ𝑠 for source domain samples and the pseudo-labels ŷ𝑡 for target domain samples:EQUATIONwhere 𝑔(•) is the linear transformation. In this stage, we optimize the model's generation accuracy by minimizing the binary crossentropy loss [12] on source domain samples:EQUATIONwhere |S 𝑒𝑥𝑝 | is the number of source domain samples.To mitigate SSB, we aim to align the distributions of the source and target domains, ensuring that the learned deep representations of samples are exposure-independent. For this purpose, we incorporate a domain classifier Ψ(•) that predicts whether a sample belongs to the source or target domain and is trained in an adversarial manner. Specifically, we desire the domain classifier Ψ(•) to make accurate predictions (minimize the training loss), and then through Gradient Reversal Layer (GRL) [18] , we compel the encoder E (•) to best fool Ψ(•), i.e., making z 𝑠 and z 𝑡 indistinguishable across data distributions as much as possible (maximize the training loss). The process can be formulated as follows:d𝑠EQUATIONEQUATIONwhere |S 𝑠𝑎𝑚 | is the number of target domain samples and L 𝑑𝑜𝑚 is the training loss of domain classifier Ψ(•). In this way, we tightly align the representations of source domain samples and target domain samples, generating unbiased pseudo-labels for each sample.However, this unsupervised domain adaptation approach cannot guarantee the quality of the pseudo-labels generated for the target domain. To address this issue, we incorporate a self-distillation mechanism to supervise the learning of pseudo-labels, thereby enhancing model robustness and label quality. Inspired by ensemble learning techniques [9, 30] , which posit that a combination of historical models yields greater performance than a single model, we treat each batch's update as the student model, and an ensemble of student models across multiple batches forms the teacher model. For updates of the teacher model's parameters, we employ Exponential Moving Average (EMA) to control the influence of the newly updated student model on it as follows:EQUATIONwhere parameters of the teacher model in batch 𝑡 are updated from the corresponding parameters of the student model, with a smoothing parameter 𝛼 ∈ (0, 1). We can elevate the student model to a teacher model without altering the original model structure. Subsequently, we use the outputs 𝑦 𝑡 𝑡𝑒𝑎 generated by the teacher model to supervise the learning of pseudo-labels for target domain samples, with the training loss defined as follows:EQUATIONOverall, to generate robust unbiased pseudo-labels, we need to minimize the two losses for label classification (L 𝑐𝑒 and L 𝑠𝑢𝑝 ) for each batch in S 𝑇 do 3:Calculate pseudo-labels ŷ using Eq. ( 4) and (5) 4:Calculate domain labels d using Eq. ( 7)Update Θ 𝑠𝑡𝑢 of E 𝑠𝑡𝑢 (•) and Φ 𝑠𝑡𝑢 (•) using Eq. (11) 6:Update Θ 𝑡𝑒𝑎 of E 𝑡𝑒𝑎 (•) and Φ 𝑡𝑒𝑎 (•) using Eq. ( 9) 7:end for 8: end for 9: Modify pseudo-labels ŷ into 𝑦 𝑚𝑝𝑙 using Eq. ( 12) 10: Calculate ranking scores 𝑦 𝑟𝑎𝑛𝑘 using Eq. ( 15) 11: for each epoch do for each batch in S 𝑇 do 13:Calculate predictions ŷ𝑟𝑒𝑡 using Eq. ( 13) 14:Update parameters of Ret(•) using Eq. ( 17) 15:end for 16: end for while maximizing the loss for domain classification (L 𝑑𝑜𝑚 ). The total loss is defined as follows:EQUATIONwhere 𝜆 1 and 𝜆 2 are the weights of L 𝑠𝑢𝑝 and L 𝑑𝑜𝑚 , respectively.In this module, we use a hybrid training mechanism to obtain the retrieval model Ret(•), as depicted in Algorithm 1. Following the previous steps, samples in the target domain S 𝑠𝑎𝑚 are assigned pseudo-labels ŷ𝑡 , while samples in the source domain have both pseudo-labels ŷ𝑠 and ground-truth labels 𝑦 𝑠 . To minimize the noise in the retrieval model training and emphasize the importance of positive samples, we continue to use 𝑦 𝑠 = 1 as the training labels for positive exposed samples. For other samples, we utilize pseudolabels as the targets. Thus, for the training sample set S 𝑇 = S 𝑒𝑥𝑝 ∪ S 𝑠𝑎𝑚 , the modified pseudo-labels 𝑦 𝑚𝑝𝑙 is defined as follows:EQUATION)Then, we input each training sample (𝑢, 𝑖, 𝑦 𝑚𝑝𝑙 ) ∈ S 𝑇 into the retrieval model Ret(•) to obtain the predictions:EQUATION)For the retrieval model training, we measure the difference between the predictions ŷ𝑟𝑒𝑡 and the modified pseudo-labels 𝑦 𝑚𝑝𝑙 as follows: (14) where |S 𝑇 | is the number of training samples. Additionally, we input each training sample into the pre-trained model Rank(•) to obtain the ranking scores: We employ a knowledge distillation strategy to narrow the gap between the output distributions of the retrieval model and the ranking model [52] , ensuring consistency in the cascade system:L 𝑚𝑝𝑙 = - 1 |S 𝑇 | ∑︁ S 𝑇 𝑦 𝑚𝑝𝑙 log ŷ𝑟𝑒𝑡 + (1 -𝑦 𝑚𝑝𝑙 ) log(1 -ŷ𝑟𝑒𝑡 ),EQUATION)EQUATION)Finally, our minimization objective is defined as follows:EQUATIONwhere 𝜆 3 is the weight of L 𝑐𝑜𝑛 . 
experiments: 
 To deeply evaluate DAMCAR, we conduct extensive experiments with the aim of answering the following research questions:• RQ1: How does DAMCAR perform compared with baselines?• RQ2: What is the role of each designed module in DAMCAR?• RQ3: What is the impact of hyperparameters on performance?• RQ4: Why does DAMCAR actually work in mitigating SSB?• RQ5: How effective is DAMCAR in real-world applications?5.1.1 Datasets. We conduct a series of experiments on two video recommendation datasets with the required multi-modal features, i.e., WeChat2 and TikTok3 . Table 1 summarizes the statistics.• WeChat: This dataset is collected from the short videos platform of WeChat4 , recording the user behaviors in two weeks. It contains pre-trained multi-modal feature embeddings with visual, acoustic, and textual information. • TikTok: This dataset contains user interactions with short videos, collected from the TikTok5 platform. The multi-modal features are the visual, acoustic, and textual content of videos. Following the settings of previous works [46, 47, 56] , we divide the datasets into training, validation, and test sets according to the timestamps of user interactions in a ratio of 6:2:2. Moreover, we set the ground-truth label to finish playing (𝑦 = 1) or not (𝑦 = 0).Testbed. We build a two-stage cascade recommendation system as the testbed for performance comparison [48, 50] , consisting of retrieval and ranking stages. In the retrieval stage, we utilize Deep Structured Semantic Model (DSSM) [28] as the backbone, a dual-tower structural model extensively applied in industrial scenarios. The dimensions of the user feature embedding and the item feature embedding are both set to 128. In the ranking stage, we employ two common CTR models, i.e., Deep Factorization-Machine Table 2 : Performance comparison of different methods in a two-stage cascade recommendation system, where DSSM is used for retrieval and DeepFM is used for ranking. The best and second-best results are marked in bold and underlined, respectively.Ret.-R@100 P@100 F@100 R@200 P@200 F@200 R@100 P@100 F@100 R@200 P@200 F@200 BC 0.0740 0. -N@10 M@10 H@10 N@20 M@20 H@20 N@10 M@10 H@10 N@20 M@20 H@20 BC 0.0477 0. Ret.-R@100 P@100 F@100 R@200 P@200 F@200 R@100 P@100 F@100 R@200 P@200 F@200 BC 0.0740 0.0113 0.0196 0.1375 0.0106 0.0197 0.0627 0.0063 0.0114 0.1222 0.0062 0.0118 KD 0.0825 0.0127 0.0220 0.1467 0.0114 0.0212 0.0655 0.0065 0.0118 0.1284 0.0065 0.0124 TL 0.0793 0.0124 0.0214 0.1458 0.0114 0.0211 0.0670 0.0066 0.0120 0.1415 0.0070 0.0133 AR 0.0821 0.0124 0.0215 0.1526 0.0117 0.0217 0.0709 0.0069 0.0126 0.1445 0.0071 0.0135 MUDA 0.0834 0.0132 0.0228 0.1458 0.0118 0.0218 0.0704 0.0072 0.0131 0.1566 0.0075 0.0143 DAMCAR 0.0917 0.0135 0.0235 0.1620 0.0124 0.0230 0.0768 0.0076 0.0138 0.1579 0.0078 0.0149 Rank.-N@10 M@10 H@10 N@20 M@20 H@20 N@10 M@10 H@10 N@20 M@20 H@20 BC 0.0494 0.0294 0.0129 0.0732 0.0335 0.0137 0.0374 0.0303 0.0064 0.0474 0.0319 0.0057 KD 0.0606 0.0389 0.0150 0.0761 0.0386 0.0137 0.0388 0.0315 0.0066 0.0510 0.0329 0.0066 TL 0.0577 0.0358 0.0145 0.0733 0.0370 0.0133 0.0375 0.0289 0.0069 0.0509 0.0310 0.0068 AR 0.0596 0.0382 0.0149 0.0790 0.0399 0.0146 0.0429 0.0333 0.0079 0.0544 0.0343 0.0071 MUDA 0.0639 0.0411 0.0162 0.0729 0.0346 0.0139 0.0406 0.0307 0.0078 0.0511 0.0318 0.0068 DAMCAR 0.0706 0.0455 0.0177 0.0824 0.0408 0.0154 0.0465 0.0374 0.0092 0.0574 0.0371 0.0080 (DeepFM) [22] and Deep & Cross Network (DCN) [55] , respectively, achieving precise ranking by deeply capturing feature interactions.Referring to [48, 56] , we compare DAMCAR with five baselines: (i) Binary Classification (BC) is considered as the base method for comparison, training the retrieval model on exposed data, which does not account for SSB. (ii) Knowledge Distillation (KD) uses the ranking model's predictions as the ground-truth labels for retrieval model training [48] . (iii) Transfer Learning (TL) utilizes unexposed data with the pseudo-labels generated by the ranking model to fine-tune the item's embedding tower and keep the user tower unchanged [42] . (iv) Adversarial Regularization (AR) trains a domain classifier from the intermediate output of the retrieval model and uses loss's negative as regularization [21] . (v) Modified Unsupervised Domain Adaptation (MUDA) uses unexposed data with the pseudo-labels derived by transforming the ranking model's predictions into binary classes for retrieval model training [56] .In the retrieval stage, we adopt three widely used metrics, i.e., Recall (R@K), Precision (P@K), and F1@K (F@K), which measure the quality of relevant candidates returned by the retrieval model. In the ranking stage, Normalized Discounted Cumulative Gain (N@K), Mean Average Precision (M@K), and Hit Ratio (H@K) are utilized to assess the ranking performance. Detailed calculations for these metrics can be found in [39] . To simulate different recommendation scenarios, we set K to 100/200 for retrieval and 10/20 for ranking, respectively. 5.1.5 Implement Details. For all methods, we use AdaGrad optimizer [14] with an initial learning rate of 1𝑒 -2 . The batch size is 512 for training and 128 for testing. In DAMCAR, we set 𝑅 = 25 and 𝐻 = 50 on WeChat, while 𝑅 = 30 and 𝐻 = 60 on TikTok, for the target domain generation. The encoder E (•) consists of a three-layer MLP with hidden dimensions of 300, 200, and 100, while the linear transformation 𝑔(•) utilizes a two-layer MLP with hidden dimensions of 128 and 100, both using ReLU [20] as the activation function. For Table 4 : Loss weights tuned in the experiments.Value Range Description𝜆 1 {5𝑒 -1 , 6𝑒 -1 , 7𝑒 -1 , 8𝑒 -1 , 9𝑒 -1 } Weight of L 𝑠𝑢𝑝 𝜆 2 {1𝑒 -1 , 2𝑒 -1 , 3𝑒 -1 , 4𝑒 -1 , 5𝑒 -1 } Weight of L 𝑑𝑜𝑚 𝜆 3 {1𝑒 -1 , 2𝑒 -1 , 3𝑒 -1 , 4𝑒 -1 , 5𝑒 -1 } Weight of L 𝑐𝑜𝑛the label generator Φ(•) and the domain classifier Ψ(•), we use a single-layer MLP with the Sigmoid [23] activation function. We use GRL [18] to reverse gradients for matching the update direction.The smoothing parameter 𝛼 is set to 0.99. Furthermore, we use grid search to find the best weights of the three different losses. The value ranges are shown in Table 4 .Table 2 and Table 3 present the results of DAMCAR and baselines in two different cascade recommendation systems, respectively. In Table 2 , DSSM is employed in the retrieval stage, while DeepFM is used in the ranking stage. To evaluate the generalization capability of our approach, Table 3 maintains the same retrieval model but substitutes the ranking model with DCN.From the experimental results, we have the following observations: (i) Compared with the base method BC, DAMCAR gains significant improvements in both retrieval and ranking metrics. That is, it correctly retrieves a sufficiently large number of candidates and precisely ranks them to be delivered to users, implying that mitigating SSB can effectively enhance recommendation performance. (ii) DAMCAR outperforms all debiasing baselines on two datasets and under different settings of K. We attribute the performance gain to three factors. Firstly, DAMCAR collects a highquality subset from unexposed data as the target domain, for data augmentation. Compared with existing sampling methods, our target domain samples contain richer information in enhancing the retrieval model training. Secondly, DAMCAR employs an effective mechanism to generate robust pseudo-labels, in contrast to relying solely on the ranking model's predictions, e.g., TL and MUDA. The performance improvement of AR over BC further supports the necessity of adversarial learning. Thirdly, the results of KD show that ensuring consistency in the cascade system benefits the final recommendation performance, and DAMCAR takes this as one of the training objectives through a hybrid mechanism. (iii) Despite employing different ranking models, DAMCAR enhances the overall recommendation performance under both configurations. This improvement indicates that enhanced retrieval performance positively impacts the ranking model's ability.In this part, we conduct the ablation study to explore the contribution of each designed module in DAMCAR. We compare DAMCAR with three variants: (i) w/o TDG, where target domain generation is replaced by randomly sampling the same number of samples from unexposed data; (ii) w/o EMA, i.e., without using EMA to create a teacher model that guides the generation of pseudo-labels for target domain samples; and (iii) w/o MPL, which only uses the generated pseudo-labels without modifying them by ground-truth labels. Figure 3 shows their final ranking performance on two datasets, using DSSM and DeepFM as the backbone models. These results illustrate two facts: (i) All designed modules are important to DAMCAR. The ranking performance will deteriorate if any module is removed. For example, in Figure 3 (a) and Figure 3(b) , each variant has worse performance in terms of N@10, M@10, and H@10 on WeChat and TikTok, respectively. (ii) Among the three modules, target domain generation has the greatest impact on performance, because generating a high-quality target domain is an important basis for generating pseudo-labels and performing hybrid training. Compared with random sampling, which often fails to select samples close to the decision boundary, our method selects samples that significantly enhance the model's discriminative ability. In addition, incorporating an EMA-based supervision mechanism and enhancing the model's perception of positive exposed samples through MPL can further improve the performance.In Figure 4 , we investigate the impact of several key hyperparameters on performance for DAMCAR.Effects of 𝑅 and 𝐻 . We first evaluate the ranking quality of DAMCAR under different values of the radius 𝑅 and the number of visits 𝐻 , i.e., two crucial settings for target domain generation. From the results, we can find that the metrics first rise and then fall, with the optimal settings of 𝑅 = 25 and 𝐻 = 50 on WeChat, while 𝑅 = 30 and 𝐻 = 60 on TikTok. We attribute the variation of settings to differences in dataset sparsity. For the hyperparameters 𝑅 and 𝐻 , they determine the search range and number of the random walk algorithm. Excessively large values may gather samples far from the model's decision boundary, while overly small values fail to collect adequate informative samples, both of which hinder the generation of a high-quality target domain, leaving the hard-to-distinguish unexposed samples without correct attention. To further verify why DAMCAR works in mitigating SSB, we show the prediction distribution of the retrieval model. In Figure 5 , we train the retrieval model on WeChat and TikTok using both DAM-CAR and BC, then showcase the predictions for target domain samples. We can find that BC tends to recognize unexposed data as negative samples, leading to inconsistency between training and inference. In contrast, DAMCAR effectively uncovers potential positive samples (i.e., unexposed positive samples) and delivers them to To validate the effectiveness of our approach in real-world applications, we deploy it on the cascade recommendation system for an online multimedia video platform and conduct A/B testing. Since each module of DAMCAR is implemented independently, it does not affect the training efficiency of the retrieval model. Specifically, we generate the target domain from historical data and train a separate DNN model to produce corresponding pseudo-labels. These offline-generated data and corresponding labels are then utilized to enhance the retrieval model for mitigating SSB. We provide metrics in three dimensions to comprehensively evaluate the performance of DAMCAR: (i) Engagement: Measure user engagement with the platform. We use Watch Time as a metric to quantify the total time spent by users. (ii) Interaction: Measure user satisfaction with the recommended results. We employ three common metrics, i.e., Like (clicking the like button), Follow (following the video creator), and Share (sharing the video with others). (iii) Diversity: Measure the variety of content consumed by users. We use Vertical Categories to signify the diversity of presented content, representing the number of unique video categories shown to all users. The results of online A/B testing are shown in Table 5 , demonstrating the effectiveness of our approach in industrial scenarios for an improved user experience. 
conclusion: 
 In this paper, we introduce a debiasing framework named DAM-CAR, as a comprehensive solution to mitigate Sample Selection Bias (SSB) in multimedia cascade recommendation systems. We start by sampling a target domain from unexposed data and use adversarial domain adaptation to generate unbiased pseudo-labels for target domain samples, narrowing the gap between training and inference data distributions. To further enhance the robustness of models and the reliability of pseudo-labels, we employ Exponential Moving Average (EMA) to create a teacher model, supervising the learning of pseudo-labels through a self-distillation mechanism. Experiments conducted on two real-world datasets and the online deployment of an industrial multimedia cascade recommendation system prove the practical benefits of DAMCAR, significantly improving recommendation effectiveness without sacrificing efficiency. 
 
 Appendix: In fact, multimedia cascade recommendation systems also include a pre-ranking stage to bridge retrieval and ranking[31,32]. However, our study primarily focuses on retrieval models, excluding the pre-ranking stage for simplicity.https://algo.weixin.qq.com/2021/problem-descriptionhttps://www.biendata.com/competition/icmechallenge2019https://www.wechat.com/enhttps://www.tiktok.com 


Decision: Addressing the Sample Selection Bias in a multimedia cascade recommendation system is an important task. In this paper, the authors leverage the domain adaptation. The experimental results indicate that the proposed method consistently outperforms the baseline approaches. Overall, the paper is well-organized and well-written.

Reviews: review: paper_summary: paper_summary: This paper introduces DAMCAR, a debiasing framework designed to mitigate Sample Selection Bias (SSB) in multimedia cascade recommendation systems by leveraging domain adaptation. The framework comprises three primary modules: target domain generation, robust pseudo-label generation, and hybrid training. By sampling a target domain from unexposed data and employing adversarial domain adaptation, DAMCAR generates unbiased pseudo-labels for target domain samples, effectively narrowing the gap between training and inference data distributions. The framework also uses Exponential Moving Average (EMA) to create a teacher model that enhances the robustness of models and the reliability of pseudo-labels. Extensive experiments on real-world datasets and an online deployment demonstrate that DAMCAR significantly improves recommendation effectiveness without sacrificing efficiency, offering a practical solution for industrial applications facing issues with SSB.
strength_weakness: strength_weakness: 1. The research direction presented in the paper is both meaningful and holds significant potential for generating positive impacts.
2. The experimental results indicate that the proposed method consistently outperforms the baseline approaches, highlighting its overall superiority.
3. The paper is well-organized and clearly written.
limitations: limitations: 1. Not provide a thorough exploration or validation of the random walk algorithm used in target domain generation, particularly concerning its effectiveness in accurately identifying hard-to-distinguish samples near the decision boundary. 
2. The paper tests its proposed framework, DAMCAR, exclusively on DSSM, DeepFM, and DCN models, which are not the most current state-of-the-art models in the field of recommendation systems. Including experiments with more advanced and recent recommendation models could strengthen the evidence of the framework's generalizability and effectiveness across different architectures.
paper_summary: paper_summary: The paper proposes a debiasing framework called DAMCAR to mitigate sample selection bias in multimedia recommendation systems that utilize cascade architectures with retrieval and ranking stages. Sample selection bias arises due to the distribution discrepancy between the exposed training data used for the retrieval model and the mostly unexposed candidates it needs to score during inference. DAMCAR bridges this gap through domain adaptation techniques. It has three modules: 1) Target domain generation, where it constructs a weighted user-item bipartite graph and uses random walks to sample hard-to-distinguish unexposed data as an informative target domain. 2) Robust pseudo-label generation, employing adversarial domain adaptation to align feature representations of the source (exposed) and target (sampled unexposed) domains, and using exponential moving average to create a teacher model that enhances pseudo-label quality via self-distillation. 3) Hybrid training that combines the generated pseudo-labels with scores from the ranking model to train the retrieval model robustly for stable inference performance. Comprehensive experiments on datasets and an online recommendation system demonstrate DAMCAR's effectiveness.
strength_weakness: strength_weakness: 1. The idea of applying unsupervised domain adaptation techniques to mitigate sample selection bias in multimedia recommendation systems is a novel contribution. Previous debiasing methods either relied on negative sampling/treating unexposed as negative which can be noisy, or required expensive collection of unbiased datasets.
2. Leveraging graph connectivity to identify hard-to-distinguish unexposed samples for the target domain is an interesting perspective that exploits multi-hop relationships.
3. The technical details like constructing the bipartite user-item graph, transition probabilities, random walk sampling algorithm etc. seem pragmatic.
4. The paper conducts offline experiments on two real-world datasets and implements DAMCAR in a practical multimedia video recommendation system and reports positive results from online A/B testing.
limitations: limitations: 1. There are some design choices made in DAMCAR's modules that could benefit from deeper analysis, such as, the specific random walk sampling algorithm used for target domain generation, the purpose of designing g() in Equation 5.
2. All datasets used for validation are not publicly available (the datasets are not available at the URLs in the paper), so a more detailed description of the datasets is needed to demonstrate the general applicability of the method.
paper_summary: paper_summary: This paper presents a new method to solve the problem of sample selection bias in multimedia cascade recommendation. First, the author performs a random walk algorithm in the user-item bipartite graph to determine the hard-to-distinguish samples other than the exposed samples. Secondly, the author uses adversarial generation method to generate pseudo-labels for the above samples, and iterates the model parameters according to batches to improve the quality of pseudo-labels. Finally, the authors trained all the samples using a hybrid training strategy. Comparison experiments, ablation experiments and parametric experiments were performed, and online A/B tests were used to demonstrate the validity of the model.
strength_weakness: strength_weakness: 1. The method of using adversarial generation to reduce sample selection bias is feasible and reflects its innovation. Specifically, the paper uses a domain classifier to try to classify exposed and unexposed samples, and pits the encoder against the domain classifier. This is a more natural way of distributing false labels, in line with the intuition that exposed and unexposed samples converge to the same distribution.
2. This article carries out extensive experiments. In addition to comparison experiment, parameter experiment and ablation experiment, the author carried out visual analysis of the model effect. At the same time, the author conducts A/B test of online deployment, which fully proves that the model has good performance in practical application.
3. The article is well organized and the language is clear. The author is able to clearly demonstrate the challenges, innovations and specific steps of the algorithm through formal language.
limitations: limitations: 1. For the random walk algorithm of the bipartite graph, it is suggested to give further explanation. This algorithm is used in the section 'Target Domain Generation' to extract hard-to-distinguish samples. Maybe there are other algorithms that perform the same function. The paper can illustrate the advantages of this algorithm, even if the ablation experiment proves it to a certain extent.
2. An adversarial training method is used to generate false labels, and the parameters are adjusted through multi-stage training, which may have a relatively large impact on the calculation cost. I wonder the machine parameters of the experiment and the difference in the algorithm's running time compared to other benchmark methods.
3. Beyond the discussion of this paper, there are several papers that are relevant and important. Authors should cite these articles and discuss them in an 'introduction' or 'Related work' section. Such as the following:
- Mitigating Confounding Bias in Practical Recommender Systems With Partially Inaccessible Exposure Status. 2023.
- Be causal: De-biasing social network confounding in recommendation. 2023.
- Implicit feedbacks are not always favorable: Iterative relabeled one-class collaborative filtering against noisy interactions. 2021.
4. Besides, authors should not use a bracketed year when citing literature. This problem arises in references [5], [7], [12], [14], [21], [22], [24], [25], [30], [35], and[41].