Adversarial Experts Model for Black-box Domain Adaptation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 14:18:56

Abstract: Black-box domain adaptation treats the source domain model as a black box. During the transfer process, the only available information about the target domain is the noisy labels output by the black-box model. This poses significant challenges for domain adaptation. Conventional approaches typically tackle the black-box noisy label problem from two aspects: self-knowledge distillation and pseudo-label denoising, both achieving limited performance due to limited knowledge information. To mitigate this issue, we explore the potential of off-the-shelf vision-language (ViL) multimodal models with rich semantic information for black-box domain adaptation by introducing an Adversarial Experts Model (AEM). Specifically, our target domain model is designed as one feature extractor and two classifiers, trained over two stages: 
In the knowledge transferring stage, with a shared feature extractor, the black-box source model and the ViL model act as two distinct experts for joint knowledge contribution, guiding the learning of one classifier each. While contributing their respective knowledge, the experts are also updated due to their own limitation and bias. In the adversarial alignment stage, to further distill expert knowledge to the target domain model, adversarial learning is conducted between the feature extractor and the two classifiers. A new consistency-max loss function is proposed to measure two classifier consistency and further improve classifier prediction certainty. Extensive experiments on multiple datasets demonstrate the effectiveness of our approach. Our source code will be released.

Main paper: 
 introduction: 
 Unsupervised domain adaptation involves transferring knowledge from a labeled source domain to an unlabeled target domain. The goal is to achieve excellent performance for the target model [ Adversarial learning is employed to distill knowledge to the target domain from two experts.. For the sake of data security and personal privacy protection, source-free domain adaptation is proposed, which only employs the model from the source domain and data from the target domain for training [20, 39] . In fact, the data distribution of source domain can be estimated by the information carried on the source model, which also implies the risk of data leakage [21, 28] . Moreover, providing AI services through cloud API has become a future development trend. Therefore, black-box domain adaptation [21] has emerged, treating the source domain as a black box. Then, using the labels output by the black-box model, source information is transferred to the target domain model. The current research on black-box domain adaptation is limited. The closely related unsupervised domain adaptation methods consist of three technical routes: traditional distribution alignment [14, 25, 27] , adversarial learning [2, 9, 23] , and self-supervised learning [8, 12, 37] . Inspired by the works on unsupervised domain adaptation problem, there are also three main approaches on source-free domain adaptation [17] , i.e, generative learning [5, 18, 40] , distribution alignment [7, 24] and self-supervised learning [4, 15, 19, 20] . The above mentioned approaches cannot be applied to the problem of black-box domain adaption directly, because both of source data and model are not accessible.Typically, due to the distribution difference between the source and target domains, the labels output by the black-box model are noisy. This can lead to negative transfer to the target domain. Consequently, there are two types of black-box domain adaptation methods to solve this problem. The first category is self-distillation learning through knowledge distillation of target data [21, 28] . The second category is pseudo-label denoising learning, utilizing the target data to get denoised labels [43, 44] . All of the above methods only exploit the information inherent in the target domain data, resulting in the target domain model lacking high-level semantic information. The performance hits ceiling. Therefore, introducing additional supervision information is an intuitive idea. Vision-language (ViL) multimodal models (e.g., CLIP [30] ) align text and images in a high-level semantic space using contrastive loss, which is a good auxiliary model to help adaptation. Unfortunately, directly using ViL model is not a good idea because it has error labels for some specific domains. Moreover, simply aligning the target domain model to the black-box and the ViL model labels is also not wise. Because the knowledge learned by these two models is heterogeneous, there are inconsistent labeled target samples.Based on the aforementioned observation, we propose a novel adversarial experts model (AEM) for black-box domain adaptation. In our method, the target domain model is designed as a shared feature extractor and two classifiers. The two classifiers are dominated respectively by the black-box expert and the ViL model expert. The overall framework of AEM is divided into two stages, namely knowledge transferring and adversarial alignment. In the knowledge transferring stage, the pseudo-labels of overall target domain samples from both the black box and the ViL model are used for supervised learning. As the expert also has limitations, the expert knowledge is updated based on the characteristics of the target domain in each epoch. In the adversarial alignment stage, the difference between the predictions from two classifiers is maximized in the hope of identifying hard samples. And then, the classification difference is minimized to update the feature extractor, expecting to extract consistency features for two classifiers. Through adversarial learning between the feature extractor and the two classifiers, the target domain model gradually consolidates and aligns the expert knowledge. The learned features are more discriminate and domain invariant. Additionally, we propose a new classifier Consistency-Max (CM) loss that, while ensuring prediction consistency and certainty, forces the output predictions to be more confident. Extensive experiments demonstrate the effectiveness of AEM, which even rivals the state-of-the-art unsupervised domain adaptation methods using ViL model.Our contributions can be summarized as follows: (1) We introduce a novel framework for adversarial learning in black-box domain adaptation, leveraging the heterogeneous knowledge from two experts. This approach enriches the target domain model with multifaceted knowledge, ensuring accurate information transfer.(2) Updates are made to the black-box labels and the prompt of the ViL model. Unlike prior methods that solely rely on self-distilled black-box labels, we refine the two experts' predictions. The knowledge from two experts will gradually evolve to better suit the target domain. (3) We propose a novel CM loss to address certainty limitations in previous approaches. Compared with previous methods using metrics like 𝐿 1 norm or Kullback-Leibler divergence to measure prediction distance, simple yet effective CM loss not only measures prediction consistency but also improves prediction certainty. 
related works: 
 Domain adaptation. Unsupervised domain adaptation (UDA) can be roughly divided into three categories: distribution alignment, self-supervised learning and adversarial learning. Despite achieving good results, UDA methods face privacy leakage issues. Additionally, due to end-to-end supervised training from the source domain, the resulting target domain model exhibits source bias. This implies that the final model inherently carries negative transfer problems. Source-free domain adaptation (SFDA) relies solely on pre-trained source models without access to the source data. There are three main types of existing source-free domain adaptation methods [17] . The first category involves generative learning [5, 18, 40] , where the goal is to create source-related data to align the target domain with the source domain. The second approach relies on aligning distributions [7, 24] . Batch statistics stored within a pre-trained source model is leveraged to approximate the distribution of inaccessible source data. Subsequently, cross-domain adaptation is achieved by directly minimizing the distribution gap between the source and target domains. The third category of methods is based on self-supervised learning, including self-supervised knowledge distillation [19, 20] , pseudo-labeling denoising [4, 15] and so on. Source-free domain adaptation approaches also can not be applied directly to the black-box domain adaptation because the source model is unknown.Black-box domain adaptation. Existing black-box domain adaptation methods can be summarized into two approaches. The first approach is self-distillation learning, aiming to gradually adapt models or labels using a teacher-student architecture [21, 28] . Symbolically, DINE [21] takes the pesudo-labels from the black-box model as the teacher, gradually distilling knowledge to the target domain model. The second approach is based on pseudo-label denoising, which leverages information from the target domain data to alleviate label noise in black-box model outputs [43, 44] . For example, BiMem [43] employs a bi-directional memory module to store useful information of long-term and short-term contexts, thereby alleviating the issue of noisy labels in black-box scenarios. Although these methods have achieved good results, they only utilize noisy label information from the black box and information from the target domain itself, failing to capture high-level semantic information. This imposes performance limitation on the target domain model.Vision-language multimodal model. Vision-language (ViL) multimodal models have achieved remarkable performance in various fields. Through training on massive datasets, ViL models effectively capture the connections between different modalities of data and derive output results from a higher semantic level. Current research on ViL models can be broadly categorized into two main streams. The first stream focuses on model optimization, exploring ways to train ViL models with higher accuracy or reduced resources [16, 45] . For example, BLIP [16] harnesses noisy web data through caption bootstrapping, which advances the performance of many vision-language tasks. The second stream delves into downstream adaptation researches [3, 41] . For example, USL-VI-ReID [3] establishes a trainable cluster-aware prompt, which acquires textual descriptions that facilitate subsequent unsupervised training. We also apply ViL model to black-box domain adaptation. Instead of The method is divided into two stages: knowledge transferring and adversarial alignment. In the knowledge transferring stage, we utilize two experts for supervised learning in the target domain. Additionally, the experts' knowledge is adaptively adjusted based on target domain information. In the adversarial alignment stage, adversarial learning is employed between the feature extractor and two classifiers to come to a consensus on the hard samples and align the knowledge from two experts. naively using ViL model, the black-box model and the ViL model are considered as two experts. Adversarial learning is applied to combine and fine-tune expert knowledge. 
proposed method: 
 In the setting of black-box domain adaptation, the labeled source domain data and the source domain model are inaccessible, which is treated as a black-box model. The only information available from the source domain is the noisy labels of the target samples obtained from the black-box model, where the target domain data, consisting of 𝑁 samples, is denoted as 𝑋 = {(𝑥 𝑖 )} 𝑁 𝑖=1 , and the noisy labels obtained from the black-box model are denoted as𝑌 1 = {(𝑦 1 𝑖 )} 𝑁 𝑖=1 .The target domain shares the same class space C = {1, 2, ..., 𝐾 } as the source domain. The goal is to train a high-performance target domain model using the information mentioned above.Overview. As depicted in Fig. 2 , the proposed AEM framework comprises two experts: the black-box expert bringing source domain knowledge and the ViL model expert containing high-level semantic knowledge. Each expert guides the learning of a classifier, denoted as ℎ 1 , ℎ 2 respectively while sharing a common feature extractor 𝑓 . The overall process is divided into two stages, namely knowledge transferring stage and adversarial alignment stage. In the knowledge transferring stage, the two classifiers are trained under the guidance of their respective experts, while the feature extractor is trained under the joint guidance of both experts. Furthermore, AEM adapts the knowledge of two experts to suit the target domain at each epoch. In the adversarial alignment stage, we customize a Consistency-Max (CM) loss to ensure the consistency and certainty of the predictions from two classifiers. Utilizing the proposed loss, we fix the feature extractor and maximize the prediction discrepancy to update the two classifiers. Then, the classifiers are fixed and the prediction discrepancy is minimized to update the feature extractor. In this way, two expert knowledge are aligned to the target domain.The black-box model possesses rich source domain knowledge. However, as the categories in the source and target domains remain the same while their distributions differ, the pseudo-labels obtained from the black-box model for target domain often carry more noise, resulting in noisy pseudo-labels. These noises introduce uncertainty and erroneous signals into the training data, potentially leading the model to learn incorrect representation and causing negative transfer. To address these challenges, we propose introducing a ViL model expert, such as CLIP [30] which incorporates high-level semantic information. ViL models learn rich semantic representations from both visual and textual contexts through contrastive learning, thereby enhancing the robustness of feature representations. However, it's worth noting that ViL models may exhibit sub-optimal performance in specific domains or tasks. Therefore, we suggest that the target domain model learns from both blackbox and ViL experts. Successful knowledge transferring from these experts is guaranteed by the knowledge dissemination strategy and knowledge feedback strategy, which will be detailed later. Knowledge dissemination strategy. To start with, the outputs from both experts are used for preliminary pseudo-label supervised learning. Let 𝑌 1 = {(𝑦 1 𝑖 )} 𝑁 𝑖=1 denotes the set of soft pseudolabels output by the black-box model, and 𝑌 2 = {(𝑦 2 𝑖 )} 𝑁 𝑖=1 denotes the set of soft pseudo-labels output by the ViL model. And then the corresponding one-hot label sets can be obtained, denoted as Ŷ 1 = {( ŷ1 𝑖 )} 𝑁 𝑖=1 from black-box and Ŷ 2 = {( ŷ2 𝑖 )} 𝑁 𝑖=1 from ViL experts, respectively. Next, we compute cross-entropy loss separately between the predictions of the two classifiers ℎ 1 , ℎ 2 and the onehot labels obtained from black-box and ViL experts respectively, written as follows:EQUATIONwhere𝑝 𝑗 𝑖 = ℎ 𝑗 (𝑓 (𝑥 𝑖 )), 𝑗 ∈ {1, 2}, is the softmax predictions by target model.As the target domain shares the same class space as the source domain, it implies that the information contained in the target domain is more relative to the black-box expert. Therefore, we further constrain the output of classifier ℎ 1 to be consistent with the black-box expert by maximizing mutual information loss [20] , which is defined as follows:EQUATIONwhere 𝑚(𝑝) = -𝐾 𝑘=1 𝑝 𝑘 log 𝑝 𝑘 is the entropy function. This term forces the black-box guided classifier to produce unambiguous predictions and encourages the label distribution to be uniform.Based on the above analysis, we update the feature extractor 𝑓 and the classifier ℎ 1 guided by the black-box expert. To enhance generalization, the widely used MixUp [28, 42] strategy is integrated into the objective function:EQUATIONSince the ViL model is trained on massive data, it is well at generalization. Therefore, for the classifier ℎ 2 guided by the ViL expert, we simply use cross-entropy for straightforward training:EQUATIONThrough the training based on Eqs.( 3) and ( 4), we obtain a shared feature extractor and two classifiers guided by two different experts, respectively. By this process, the target domain model acquires the ability to classify easy samples under the guidance of two experts. Knowledge feedback strategy. Although the experts impart knowledge to the target domain model, their expertise lies outside the target domain. For instance, the knowledge from the black-box model is biased towards the source domain, while the knowledge from the ViL expert is highly generalized and not fine-tuned. In order to achieve better performance in the target domain, the knowledge from both experts should be promptly updated based on the target domain. Therefore, we adapt the knowledge of both experts to be suitable for target domain at each epoch.Since the black-box model only outputs noisy labels for the target domain and cannot be adjusted, we use the EMA (Exponential Moving Average) strategy to update its pseudo-labels:EQUATIONwhere 𝛼 is a hyperparameter to adjust the update rate. Differing from the sealed and non-fine-tunable black-box model, we hope to continuously learn the high-level semantic information of ViL expert. Thus we are committed to adapting the ViL expert to the target domain. Inspired by CoOp [45] , with the ViL model frozen, we utilize black-box labels to fine-tune the prompt of the ViL expert, enabling its knowledge to be made use of in the target domain. Specifically, a consistency-max (CM) loss, introduced and analyzed in Section 3.2, is used to update the prompt:EQUATION)where 𝑤 is the prompt of ViL model to be fine-tuned.Using the EMA strategy, the black-box labels are gradually replaced by the outputs of ℎ 1 , thereby incorporating more information from the target domain. The noise issue of the black-box output labels will also be alleviated. From the perspective of knowledge distillation, the black-box labels serve as the teacher's role in the teacher-student model, carrying more authoritative information. Therefore, we use the updated black-box labels to fine-tune the prompt of ViL. Through this process, the output of ViL will also be more suitable for the target domain model.Through knowledge transferring, the target domain model has learned heterogeneous knowledge from two experts. It means the knowledge learned by the two classifiers is unrelated to some extent. That is, for the same sample, the two classifiers may produce inconsistent results. Additionally, since only one feature extractor is used for learning knowledge from two heterogeneous spaces, the extracted features are likely to lack discrimination. So, it is difficult for the classifiers to make decisions and resulting in ambiguous decision boundaries. Hence, we consider to further reinforce and align expert knowledge to the target domain through adversarial training.Consistency-max loss. In traditional MCD-based methods [6, 13, 32] , 𝐿 1 norm divergence is commonly used to measure prediction consistency between two classifiers. However, solely enforcing consistency to align different distributions is insufficient. In addition to enforcing consistency, decision boundaries should be more accurate. But it is hindered due to the lack of ground truth labels in the black-box domain adaptation setting. Fortunately, there exists a basic fact, i.e., for a given sample, the probability that both experts predict it to belong to a same incorrect category is very low. Therefore, a simple but effective divergence measurement loss is proposed. To be specific, letEQUATION𝑖 )} 𝑁 𝑖=1 denote the softmax predictions of the black-box guided classifier and ViL guided classifier, respectively. The Consistency-Max loss (CM loss) is defined as:EQUATION)Next, we demonstrate the effectiveness of the proposed CM loss.For 𝑖-th sample, as per the Cauchy-Schwarz inequality:EQUATIONwhere 𝐾 represents the number of classes. Since the predicted outputs are softmax probabilities, each item of the prediction has 𝑝 𝑘 > 0. Thus, we can derive:EQUATIONHence, it can be inferred as follows:EQUATION)From Eq.( 10), it is evident that the maximum value 1 is achieved if and only if both classifiers assign a value of 1 to a same class for the 𝑖th sample. Due to the low probability of two experts predicting the 𝑖th sample as the same incorrect class, the predictions are more consistent and certain using CM loss constraint, which tends to be more correct.Adversarial alignment of experts knowledge. We employ adversarial thought to align knowledge from different experts by leveraging the proposed CM loss. First, the discrepancy is maximized between the outputs of the two classifiers with the fixed feature extractor. Additionally, we employ experts supervision to ensure the correct classification of the classifiers:EQUATIONNext, the classifiers are fixed and only the feature extractor is updated. The discrepancy between classifiers' output is minimized as follows: minEQUATIONIn summary, maximizing the CM loss under experts' constraints enables the identification of hard samples. Subsequently, minimizing the CM loss yields the feature extractor to learn the features that are indistinguishable by two classifiers. Namely, the extracted features are more discriminate and domain invariant. Gradually, through adversarial learning and CM loss, difficult samples are selected and further classified by the target model. From a macro perspective, the two expert knowledge are distilled to the target domain.During the training process, we repeat the aforementioned steps until the model converges. For inference, we utilize the ViL guided classifier. The overall algorithm is outlined in Algorithm 1. 
experiments 4.1 experimental setup: 
 Dataset description. We use four open datasets including Office-31, Office-Home and VisDA-2017 for comparative experiments. Office-31 [31] is a commonly used dataset that contains three domains including Amazon (A), DSLR (D), and Webcam (W) with 31 categories. Office-Home [36] dataset consists of around 15,500 images spread across 65 categories, covering four diverse domains: Art (A), Clip Art (C), Product (P), and RealWorld (R). VisDA-2017 [29] dataset comprises synthetic-to-real images. The source domain consists of 152397 synthetic images and the target domain includes 55388 realistic images.Model architecture. For a fair comparison, our network employs the same structure as the state-of-the-art black-box domain adaptation methods [21, 28] . For both the source and target domains, for 𝑏 = 1:𝐵 do Step 1: Update black-box guided 𝑓 , ℎ 1 by Eq.( 3) and ViL guided 𝑓 , ℎ 2 by Eq.( 4); 5:Step 2: Update ℎ 1 , ℎ 2 by maximizing the discrepancy and minimizing the cross entropy loss Eq.( 11) ; 6:Step 3: Update 𝑓 by minimizing the discrepancy Eq.( 12) ; 7:end for 8:Update black-box labels 𝑌 1 by Eq.( 5) and prompt of ViL model by Eq.( 6); 9: end for 10: return Adapted target model.the pre-trained model ResNet-50 [11] is employed as a feature extractor for the Office-31 and Office-Home datasets, and ResNet-101 is utilized for the VisDa-2017 dataset. The same as DINE and RAIN [21, 28] , between the feature extractor and the fully connected layer classifiers, a bottleneck structure is used which is a series of layers in a pipeline, consisting of a fully connected layer, batch normalization, fully connected layer, weight normalization. This structure is combined with the feature extractor as 𝑓 in the adversarial alignment learning. For the frozen ViL model, we utilize the widely used CLIP ViT-B/32 to obtain the output results.Implementation details. Following previous works, all the black-box models are trained with the most commonly used crossentropy loss as Eq.( 1). In terms of AEM parameter, as indicated in the method part, we have only one hyperparameter, which is the momentum setting as 𝛼 = 0.9 for updating the black-box pseudolabels. For the ViL model's prompt, we initialize it as 'a photo of a [CLS].'. The black-box pseudo-labels and the learnable prompts are updated at the beginning of each epoch. For model parameter settings, we adhere to the recommended training configurations outlined in [21, 26] , encompassing weight decay (1e-3), bottleneck size (256), and batch size (64). Our optimizer employs mini-batch SGD. The same learning rate scheduler 𝜂 𝑝 = 𝜂 0 (1+𝛾𝑝) -𝑏 is adopted for all datasets. For Office-31 and Office-home datasets, we set 𝛾 = 10 and 𝑏 = 0.75, while for VisDa-2017, we set 𝛾 = 10 and 𝑏 = 2.25. All experiments are implemented on the PyTorch platform with a single NVIDIA RTX GPU.Competitors. With the source (black-box) model as a baseline, we compare AEM with the state-of-the-art methods, which can be categorized into three types: (1) The first type is unsupervised domain adaptation methods, where both the source domain data and ground truth labels are accessible, such as MCD [32] , AFS [46] , NCL [22] , DAPL [10] , AD-CLIP [33] , PDA [1] . (2) The second type is source-free domain adaptation, where only the source domain model is accessible. It contains SHOT [20] , D-MCD [4] , C-SFDA [15] , TPDS [34] . (3) The third type is black-box domain adaptation. The only available information is the noisy labels output by the source model, DINE [21] , RAIN [28] , BETA [38] , RFC [44] to name a few.In addition, with a checkmark ' ✓ ' in column M in all tables, it indicates the utilization of the ViL model.The comparative results of AEM on three datasets are shown in Tables 1 to 3 . Compared to other black-box domain adaptation methods, our approach emerges as a frontrunner. On Office-31 dataset, our method attains a remarkable accuracy of 91.9%, surpassing the second-highest method, RFC, by 2.7%. On Office-Home dataset, our average accuracy outperforms other leading methods by 7.6%, with optimal results across all 12 tasks. Similarly, on the VisDa-2017 dataset, we achieved the highest accuracy of 89.3%. Compared with the source-free domain adaptation methods, AEM maintains its leading position even under stringent conditions, namely without the source model. On the Office-31, Office-Home, and VisDa-2017 datasets, AEM outperforms the highest method each by 1.4%, 7.1%, and 0.6%, respectively. Moreover, the results of AEM can even match the state-of-the-art UDA methods that have access to source domain data and labels. Specifically, on the three datasets, our method outperforms the highest-performing UDA methods by 0.7%, 4.7% and 1.6%. When stacked up against methods utilizing vision-language information, such as AD-CLIP, DAPL, and PDA, which belong to UDA, our method achieves comparable results. Worth noting is that on the office dataset, AEM achieves the highest accuracy across all 12 specific tasks. We also compare our method with CLIP model and the adapted prompt CLIP model on Office-31 and Office-Home datasets, which shows AEM is better in terms of performance, model parameters and FLOPs (see Appendix).Through the analysis of experimental data, we have the following observations. First, most of the methods only use source domain information and target data, thus failing to capture the high-level semantic information within the data. In contrast, we make use of both the source domain information from black-box expert and the high-level semantic information from ViL expert. Then, generally speaking, from the results in column 'M', it's clear that the methods, such as AD-CLIP, DAPL and PDA, leveraging ViL expert knowledge outperform those that do not. However, while they address UDA problems, we tackle a more challenging scenario of black-box domain adaptation with stringent source domain access constraints. Our method even achieves a higher average accuracy than theirs across the three datasets. This is attributed to AEM's utilization of adversarial principles. In the adversarial alignment stage, hard samples are identified through the classifiers, and the feature extractor is improved to extract more discriminative and domain invariant features. Also, during the training, the knowledge of both experts is gradually aligned in the target domain. Furthermore, our method continuously updates the knowledge of both experts to make their guidance in the target domain more adaptable and relevant. Moreover, the proposed CM loss better measures the discrepancy between different prediction distributions, ensuring the consistency and certainty of predictions simultaneously.In order to better demonstrate the effectiveness of our method, we conduct extensive ablation analysis and visualization experiments on Office-31 dataset. The analysis on Office-Home dataset is shown in Appendix.Ablation study on knowledge feedback strategy. To demonstrate the effectiveness of the proposed knowledge feedback strategy, three additional variants are compared, i.e., training without knowledge feedback strategy (w/o_bv), training without updating the black-box labels (w/o_b), and training without updating the 4 , updating the knowledge of both experts yields the best results, surpassing the second highest on average by 5.7% and 4.7% for each task.Ablation study on different classifier combination. Fig. 3 illustrates the accuracy curve of three different predictions, i.e., 𝑝 1 , 𝑝 2 and 𝑝 𝑚𝑖𝑥 . The accuracy of these three predictions tends to be stable after 10 epochs and the differences are subtle, all around 2.0%. Overall, it appears that the classifier guided by ViL tends to perform slightly better. This is because the classifier guided by ViL contains more high-level semantic information. Therefore, in Section 4.2 and the following ablation studies, we choose the classifier guided by the ViL expert for comparison. Furthermore, as shown in Fig. 3 , the accuracy on each task reaches the maximum around the 10th epoch and then is stable, which means the training of the target domain has converged.Ablation study on adversarial learning. To validate the effectiveness of adversarial learning, we use a consistency-based method (w/o_adv) to compare. Instead of maximizing classifier discrepancy, the method w/o_adv solely utilizes CM loss for constraint, updating both the feature extractor and the classifiers accordingly. As shown in Fig. 4 (a), compared to the method w/o_adv, AEM achieves an average accuracy improvement of 7.6%. This is because using consistency loss alone not only fails to address the issue of difficult samples, but also introduces noise into the target domain model due to noisy labels from black-box models. Although these noises are partially corrected by the ViL model, they also lead incorrect optimization directions to the ViL model. In AEM, we select difficult samples by maximizing classifiers' discrepancy. Then we update the feature extractor by minimizing prediction discrepancy, enabling the extracted features to be more discriminative.Ablation study on classifier consistency loss. Fig. 4 (b) displays the results obtained using different classifier discrepancy metric loss during the adversarial alignment stage. We compare the proposed CM loss with 𝐿 1 loss and KL divergence loss. The results show that using CM loss outperforms 𝐿 1 loss by 2.9% and KL divergence loss by 5.8% on average. The reason why KL performs worst is that it is a biased estimate. However, in the framework of AEM, the two classifiers should have equal competitive relationships. Thus the bias towards either side will result in a deterioration of model performance. 𝐿 1 loss performs slightly worse because it merely constrains the two predictions of the same sample to be more consistent. The proposed CM loss not only requires consistency between predictions, but also constrains the predictions to be more certain, thus demonstrating better performance. Visualization on t-SNE. The t-SNE [35] visualization results on tasks A→D and D→A are shown in Fig. 5 .  
conclusion: 
 We propose a new framework for black-box domain adaptation based on adversarial learning, named AEM. In contrast to previous methods that exploit self-information, we introduce an additional vision-language model into black-box domain adaptation to capture high-level semantic information. Moreover, AEM is characterized by jointly leveraging the existing knowledge of black-box and ViL experts and continuously updating the experts themselves. And then, adversarial learning is employed to distill two heterogeneous knowledge to the target domain model. Moreover, the proposed simple yet effective consistency-max (CM) loss ensures the consistency and certainty of predictions. The effectiveness of our approach has been validated through experiments on multiple datasets. 
 
 Appendix:  


Decision: This paper deals with a relatively less studied and interesting problem: black-box domain adaptation. In this problem, the source domain model is wrapped as an API instead of a trainable model. And the source data is also inaccessible.   
To tackle this challenging scenario, this paper proposes to use pre-trained Vision-language models as the ViL expert and enforce interactions and constraints between this new ViL expert and existing Black-box source expert. Specifically, two training stages are designed: knowledge transferring and adversarial alignment (better visualized in Rebuttal Figure 2 than the main text).   
In those two stages, knowledge feedbacks and consistency-max loss are novel technical modules for the black-box domain adaptation problem, for correcting noisy labels and aligning different experts.   
Good experiments including SOTA comparison, ablation, and visualization are demonstrated.  

From intial comments, reviewer bdAz raised the concern of backbone (ResNet50 and ViT-B/16), i.e., baselines such as PDA mostly used ResNet while the proposed method utilized ViT-B/16 (unfair comparision). In the rebuttal, the authors made a comprehensive rebuttal and explanation (including previous lack of clarification issue: "PDA using ViT-B/16" in Table 1 of main text). Moreover, ViT-B/16 comparison is conducted on Table 1 of Rebuttal, indicating slightly improvement of proposed AEM over PDA (89.9vs89.7). Considering the difficulty of black-box DA over UDA, this is meaningful to me. Reviewer bdAz kept the score as **Borderline Reject** after rebuttal, but after carefully reading the paper and rebuttal, I think the concern is addressed. 
Other two reviewers raised concerns about technical details, Figure 2 presentations, failure cases etc. In the rebuttal, the authors tried to address these concerns. These two reviewers are happy with the rebuttal, and lean towards **Borderline Accept** the paper. 

Finally, the decision points are: (1) the problem and proposed solution is less investigated and meaningful. The audience will benefit from this paper. (2) The overall writing and organization (after revised Figure 2 in the rebuttal) are well and clear, which makes the paper easy to read and follow. (3) The authors clarified technical details and addressed most of the concerns. 

Overall, I would like to recommend **Accept (Poster)** for this paper. Meanwhile, I also encourage the authors to include the clarification and revision of the rebuttal in the revised paper.

Reviews: review: paper_summary: paper_summary: This paper proposes an Adversarial Experts Model (AEM) for black-box domain adaptation, which leverages vision-language multimodal models to incorporate rich semantic information. The target domain model consists of a feature extractor and two classifiers, trained over two stages: knowledge transferring and adversarial alignment. The black-box source model and the ViL model act as experts, guiding the learning of the classifiers while being updated to address their limitations. Adversarial learning and a novel consistency-max loss function are employed to distill expert knowledge and improve prediction certainty. Experiments on multiple datasets demonstrate the effectiveness of the proposed approach.
strength_weakness: strength_weakness: a. Novel adversarial learning framework leveraging heterogeneous knowledge from two experts (black-box source model and ViL model) for accurate information transfer.
b. Refinement of expert predictions by updating black-box labels and ViL model prompts, allowing knowledge to evolve and better suit the target domain.
c. Improved prediction consistency and certainty through a novel consistency-max (CM) loss, addressing limitations of previous approaches.
d. Comprehensive experiments on multiple datasets demonstrating the effectiveness of the proposed approach in various black-box domain adaptation scenarios.
limitations: limitations: Upon reviewing the comparative experiments, it has been noted that the PDA method utilized both ResNet50 and ViT-B/16 as backbone architectures. However, when presenting the performance comparison, it appears that only the results of PDA with the ResNet50 backbone were included. To provide a more comprehensive evaluation and convincingly demonstrate the effectiveness of the proposed adversarial approach, it is strongly recommended to conduct experiments using the same ViT-B/16 backbone for PDA and report the complete results. This will ensure a fair comparison and allow readers to assess the true impact of the adversarial learning framework on the black-box domain adaptation task.
paper_summary: paper_summary: This paper proposes a novel Adversarial Experts Model (AEM) for black-box domain adaptation. The method leverages a black-box source domain model and a pre-trained Vision-Language (ViL) model as two experts to transfer knowledge and conduct adversarial training in the target domain, improving the performance of the target domain model. The main contributions include: (1) Introducing a new adversarial learning framework that jointly exploits and refines the heterogeneous knowledge from the source domain and ViL experts. (2) Proposing a knowledge feedback strategy to dynamically update the knowledge of both experts based on target domain characteristics. (3) Proposing a simple yet effective consistency-max loss to constrain the consistency and certainty of classifier predictions. Experiments on multiple datasets demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The knowledge feedback strategy to adaptively update the pseudo-labels from the black-box model and the prompt of the ViL model is a neat way to tailor the experts' knowledge to the target domain over the course of training.
2. The adversarial learning framework, consisting of maximizing classifier discrepancy to identify hard samples and minimizing it to align expert knowledge, is a principled approach.
3. The consistency-max loss is simple yet effective in enforcing both prediction consistency and certainty.
4. Extensive experiments on three datasets demonstrate the superior performance of AEM over state-of-the-art methods in both black-box DA and the more relaxed source-free DA settings. Ablation studies validate the usefulness of different components.
limitations: limitations: Weaknesses & Questions
1. The paper does not provide a clear and quantitative definition of "hard samples" that are identified in the adversarial alignment stage. Without a rigorous formulation, it is difficult to assess whether the proposed method truly focuses on the hard samples and how effectively it handles them. 
2. The limitations and failure cases of AEM are not thoroughly discussed. Every method has its inherent assumptions and limitations, so it is important to analyze and present the scenarios or cases where AEM may fail or underperform.
3. In the knowledge feedback stage, could you share what the new prompt of the ViL expert looks like after fine-tuning? How does it differ from and improve upon the original prompt?
4. In the experimental results of Tables 1-3, could you analyze the reasons why AEM's advantages are more significant on Office-Home than  on Office-31 and VisDA-2017? Is it related to the characteristics of the datasets?
paper_summary: paper_summary: This paper proposes a domain adaptation method to achieve domain transfer through adversarial training between expert models and pre-trained models. Through the designed two-stage training method, the purpose of improving the image classification prediction performance of the final model is achieved.
strength_weakness: strength_weakness: The paper is well written, the description of the method is relatively clear, and a large number of experiments have been conducted. The comparison with existing methods demonstrates the effectiveness of the method proposed in this article.
limitations: limitations: However, the paper has the following shortcomings:
1. Expert models and adversarial training will inevitably increase training time. The cost of each method in model training should be reflected in the experimental section.
2. Figure 2 is confusing and needs to be redrawn to show the relationship between the methods proposed in the paper.
3. This paper uses 3 data sets instead of 4.