Style Adaptation and Uncertainty Estimation for Multi-Source Blended-Target Domain Adaptation

Authors: yuwu  lu
haoyu  huang
xue  hu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 07:06:49

Abstract: Blended-target domain adaptation (BTDA), which implicitly mixes multiple sub-target domains into a fine domain, has attracted more attention in recent years. Most previously developed BTDA approaches focus on utilizing a single source domain, which makes it difficult to obtain sufficient feature information for learning domain-invariant representations. Furthermore, different feature distributions derived from different domains may increase the uncertainty of models. To overcome these issues, we propose a style adaptation and uncertainty estimation (SAUE) approach for multi-source blended-target domain adaptation (MBDA). Specifically, we exploit the extra knowledge acquired from the blended-target domain, where a similarity factor is adopted to select more useful target style information for augmenting the source features. \!Then, to mitigate the negative impact of the domain-specific attributes, we devise a function to estimate and mitigate uncertainty in category prediction. Finally, we construct a simple and lightweight adversarial learning strategy for MBDA, effectively aligning multi-source and blended-target domains without the requirements of domain labels of the target domains. Extensive experiments conducted on several challenging DA benchmarks, including the ImageCLEF-DA, Office-Home, VisDA 2017, and DomainNet datasets, demonstrate the superiority of our method over the state-of-the-art (SOTA) approaches.

Main paper: 
 introduction: 
 Domain adaptation (DA), whose objective is to transfer knowledge from one or more well-labeled source domains to a non-labeled target domain, has been studied in recent years [1, 2, 3, 4, 5, 6] , including object classification [1, 2] , semantic segmentation [3, 4] , and object detection [5] . However, most DA approaches are based on a setting that has single source domain and single target domain [1, 2] . In the real world, unlabeled target domains are usually drawn from different distributions. Therefore, most of the existing single target-based DA approaches may not be the best answer to address domain shifts in reality.Fortunately, an increasing number of researchers have focused on the above-mentioned issues, and multi-target domain adaptation (MTDA) [7, 8, 9] has been studied. MTDA aims to learn a model that can simultaneously utilize information from single source domain and multiple target domains and then perform well in multiple target domains. For instance, HGAN [10] adopts a heterogeneous graph attention network to explore the relations among multiple target domain features. In [11] , multiple expert models employed the corresponding source-target domain pair-groups and were then aligned by a student model. Although the existing MTDA approaches have made certain progress, the standard MTDA is still facing challenges because massive amounts of unlabeled data drawn from different distributions are commonly used in real-world settings. It is time-consuming and expensive to divide massive data into multiple corresponding distributions (target domains). For example, for encrypted data stored in a cloud server, due to privacy protection, models cannot directly know the origins of these data (domain labels), which are drawn from different distributions and are blended into a large target domain. Based on the above-mentioned case, blended-target DA (BTDA), which is a more beneficial scenario in real-world settings, has been proposed [12] .Current BTDA approaches [12, 13] are mainly based on three points: 1) the adaptation process contains single source domain and multiple target domains. 2) During the adaptation process, the model cannot access both the domain labels and category labels of the target domains. 3) In blendedtarget domain, the category labels of each sub-target domain may not follow the same distribution. Therefore, simply utilizing MTDA or SSDA (single source domain adaptation) methods to handle the BTDA task may lead to negative transfer because the domain labels of target domains are unseen, and the category feature space is a hybrid space. As the first deep learning work focused on the BTDA scenario, AMEAN [12] employs two adversarial learning steps and utilizes meta-learning to minimize the domain gap between the source domain and the blended-target domain. However, insufficient information obtained from single source domain makes models difficult to align the distributions of multiple target domains. Moreover, the presence of different distributions in the blended-target domain may aggravate negative transfer. Recently, multi-source domain adaptation (MSDA) [14, 15, 16] has produced impressive results. MSDA approaches can utilize more feature information from extra source domains to learn domain-invariant representations, effectively solving negative transfer. However, as far as we know, no related works have been proposed to utilize more feature information from multiple source domains for BTDA.In this paper, to further exploit feature information from multiple domains, we pay attention to the BTDA in the case of multiple source domains, i.e., Multi-Source Blended-Target Domain Adaptation (MBDA). The comparisons of different DA settings are illustrated in Table 1 . At the same time, a style adaptation and uncertainty estimation (SAUE) method is proposed for MBDA. Different from previous works, we utilize the style information of the blended-target domain to enhance source domain features, thus building a better representation space. Specifically, we first simultaneously extract the source and target style information and then calculate the similarity factors between the source and target style information. The similarity factors are served as the weighted matrix during the feature augmentation process. Based on style adaptation, we further analyze the uncertainty in the classification model and adopt a loss function to eliminate the uncertainty introduced by the multi-source domains. In addition, as the domain labels of the blended-target domain are unavailable in MBDA, we construct an adversarial learning scheme for MBDA without the requirement of domain labels of the target domains.The main contributions of this work are summarized as follows:• An approach SAUE is proposed to explore information from multiple source domains for BTDA. As far as we know, SAUE is the first work that was proposed for MBDA, which can utilize more feature information from extra source domains to learn domain-invariant representations.• To further exploit the style information in the blended-target domain, we propose a similaritybased style adaptation strategy for MBDA, which selects target styles to enhance the source representation space.• We propose an uncertainty estimation technique to enhance the robustness of our method, which exploits valuable knowledge from multiple source domains. In addition, we construct a specific adversarial learning strategy for MBDA, which aligns domains without the requirement of domain labels. 
related works: 
 Single-source and Single-target DA (SSDA). The objective of SSDA is to learn domain-invariant representations through the relations between the source and target domains. Based on this objective, researchers have carried out widespread researches and achieved certain progress [17, 18, 19, 20, 21, 22, 23] . The current SSDA methods are mainly divided into two types: distance metric-based approaches [17, 18, 19] and adversarial learning-based approaches [20, 21, 22, 23] . Distance metricbased methods learn domain-invariant representations through feature discrepancy matching by using a distance metric function. DAN [18] utilizes multi-kernel maximum mean discrepancy (MMD) to measure the discrepancy between the source and target domains and then minimizes the discrepancy to learn domain-invariant representations. CAF [19] utilizes sliced Wasserstein distance (SWD) [24] to measure domain discrepancy. Motivated by the GANs [25, 26] , adversarial learning-based SSDA methods have also been widely researched [20, 21, 22, 23] . Adversarial learning methods perform min-max two-player games between the category classifier and domain discriminator to learn domain-invariant representations. DANN [20] , which was the earliest work in adversarial learning-based SSDA, successfully achieves domain-level adaptation via a gradient reverse layer. Different from DANN, SCDA [22] and DALN [23] remove the discriminator from their networks and model the adversarial relation between the feature extractor and category classifier. Although the above-mentioned approaches have achieved great success, due to the limitations of single source domain features and single target domain features, current SSDA methods still face some challenges in real applications.Multiple Domains DA. The motivation of multiple domains DA is to explore more useful knowledge from multiple domains for the tasks. Many researchers have focused on multiple-domain DA and proposed many excellent methods [13, 14, 15, 8] , including MSDA [14, 15, 27, 28] , MTDA [8] , BTDA [13, 12] , and MMDA (multi-source and multi-target DA) [29] . M3SDA [14] utilizes moment matching to align domain distribution. DCA [15] further extracts the multiview features from multiple source domains and then utilizes multiple classifiers and pseudo-label learning strategy to align distributions. Meanwhile, in MTDA, CGCT [8] utilizes feature aggregation and curriculum learning to learn the pseudo-labels of multiple target domains. AMDA [29] constructs multiple domain discriminators and utilizes attention mechanism to address the MMDA issue.Recently, a more realistic DA scenario, BTDA, has been studied [12, 13] . For example, MCDA [13] utilizes the mutual condition to learn domain-invariant representations, which has achieved great progress in BTDA. However, single source domain in BTDA cannot provide sufficient feature information for aligning the source and blended-target domains. Furthermore, the unseen domain labels of the target domains also aggravate the challenges. Therefore, we consider multiple source domains in BTDA and utilize the style information of the target domains to optimize the representations of source features and minimize the model uncertainty, thereby obtaining a better transfer.3 Method 
preliminary: 
 In MBDA, we have M labeled source domains S = {S m } M m=1 that are drawn from distributions{P Sm } M m=1 . S m = {x Sm i , y Sm i } |Sm| i=1, where x Sm i ∈ R d denotes the i-th source sample from the m-th source domain and y Sm i is the corresponding category label, and d denotes the number of dimensions. Meanwhile, we have an unlabeled blended-target domain T that consists of N sub-target domains {T n } N n=1 , andT = {x T j } |T |j=1 . The distributions of sub-target domains are {P Tn } N n=1 . Therefore, the distribution of blended-target domain P T is the mixture of sub-target domains, i.e., P T = N n=1 π n P Tn , where π ∈ [0, 1] and N n=1 π n = 1. Each of the source and target domains shares the same category space. The objective of MBDA is to train a model that utilizes multiple source domain features and performs well on the blended-target domain. Different from MTDA, the target domain labels are unseen in MBDA. In addition, the analysis in [12] demonstrated that directly utilizing DA methods to address BTDA transfer tasks may cause increased uncertainty and negative transfer. Therefore, we utilize the style information of the blended-target domain to augment source features and minimize the uncertainty of the model. Furthermore, the adversarial learning strategy in our method without the requirement of domain labels of the sub-target domains is suitable for MBDA setting. Figure 1 illustrates the overall architecture of SAUE.Figure 1 : Overview of the proposed SAUE approach. First, the style information of the blended-target domain is utilized to augment the source features through the similarity factors. Second, we calculate the uncertainty of the model and optimize prediction uncertainty via the Dirichlet distribution. Finally, the adversarial learning strategy without discriminator effectively guides the SAUE process to adapt the blended-target domain without the requirement of domain labels of sub-target domains.Since the principal parts of features from different domains remain domain-invariant, the domainspecific parts, which mainly contain style information, are the main discrepancies between different domains. In addition, the target feature distributions in MBDA are confused, which may cause model degradation. Therefore, we try to utilize the style information of blended-target domain to augment source features, which brings source features closer to target features. Previous work [13] has demonstrated that low-level features of deep neural networks (DNNs) mainly represent style information. Some works [13, 30] have utilized the channel-wise mean and variance of the low-level features to represent the style information of input samples. Thus, for sample x Sm i which from the m-th source domain, let its low-level feature be z Sm i ∈ R d×H Sm ×W Sm , where d denotes the channel and H Sm , W Sm denote the height and width of sample x Sm i . The channel-wise mean and variance of the low-level feature z Sm i can be defined as follows:EQUATIONLow-level features mainly represent style information, but different samples contain specific pieces of style information. Therefore, we adopt feature normalization technique to standardize the feature z Sm i , and the normalized feature zSm i is defined as:EQUATION)where ϵ is a small value used to avoid division by zero.Then, the target features will be utilized to augment the normalized source features. Previous work [13] randomly augmented source features through target style information, which yielded limited performance. In our work, we select the target style information according to a weight factor. Specifically, we leverage the Wasserstein Distance [31] to measure the style distribution discrepancy w Sm i between the source low-level feature z Sm i and the target low-level feature z T j as follows:EQUATIONThen, we utilize w Sm i to calculate the weight factor as follows:EQUATION)To ensure that the sum in Eq. ( 4) equals to 1, we utilize softmax operation to normalize each λ Sm i . Then, we can obtain the weighted target style as follows:EQUATIONFinally, the low-level source feature augmented by the blended-target style information can be calculated as follows:EQUATIONDifferent from previously developed image augmentation method [13] , our method directly utilizes target information with weight factor to augment source features instead of generating specific images. The low-level feature z SmT i augmented by diverse target styles further mitigates the impact of the domain-specific attributes.Although multiple source domains provide additional supervised information for adaptation compared to a single source domain, they also introduce more domain-specific attributes. This can cause model degradation, especially when some source domains are significantly dissimilar to the blended-target domain due to the abundance of domain-specific attributes. Evidential model learning (EDL) [32] is an interpretable approach that fuses knowledge from multiple domains using the Dempster-Shafer Rule [33, 34] , which is more beneficial to MBDA scenario. Thus, we utilize the Dirichlet-based evidential model [32] to estimate the uncertainty of our model during the training process. Specifically, for a sample x i , we have the predictionsp i = C(G(x i )) = [p i1 , p i2 , . . . , p iK ],where C and G denote the classifier and feature generator, respectively. The probability density function of p i is defined as follows:EQUATIONwhereU k = {p i | K k=1 p ik = 1 and 0 ≤ p i1 , ..., p iK ≤ 1} is the K-dimensional unit simplex and α i is the parameter of the Dirichlet distribution. B(α i ) = Γ( K k=1 α ik ) K k=1 Γ(α ik )is the K-dimensional multinomial beta function, and Γ(•) denotes the gamma function.Previous work [32] has demonstrated that DNNs can generate opinions for classification tasks as Dirichlet distributions. Therefore, given sample x i , for prediction of class c that generated by DNNs, the Dirichlet distributions connected with DNNs can be defined as follows:EQUATIONThe derivation of Eq. ( 8) is provided in Appendix B.In this work, for source sample x Sm i , we utilize the prediction of the category classifier as the evidence vector, and the parameters of the corresponding Dirichlet distribution can be defined asα Sm i = C(G(x Sm i )) + 1.To eliminate the uncertainty, we force the total evidence to zero when the model generates an incorrect prediction for the source sample, and the corresponding uniform Dirichlet distribution is D(p Sm i | ⟨1, . . . , 1⟩). For implementation, we utilize the Kullback-Leibler (KL) divergence to reduce the impact of incorrectly classified source samples in our loss function, which is defined as follows:EQUATIONwhereαSm i = y Sm i + (1 -y Sm i ) ⊙ α Sm idenotes the Dirichlet parameter used to remove the true evidence from prediction α Sm i . Specifically, the KL divergence is:EQUATIONwhere Γ(•) and Ψ (•) denotes the gamma function and digamma function, respectively. Randomly sample a mini-batch of B source samples and target samples. Augment the source features by utilizing style adaptation, i.e., Eq. ( 6).Minimize the parameters of the feature generator and category classifier by L cls .Optimize the uncertainty of model through L unc .Perform the min-max game between feature generator and classifier with L d :min G max C L d (x Sm , x T ).Existing works [20, 21, 22, 8] have demonstrated that adversarial learning strategy is beneficial in DA. However, most of the adversarial learning strategies in DA [20, 21] usually request the domain labels of the source and target domains to train their discriminators, which cannot satisfy MBDA. Inspired by the intra-and inter-class correlation [35] , we design an adversarial learning strategy for MBDA without discriminator and domain label requirements. Specifically, the category classifier is reused to discriminate which domain a feature originates from, with the guidance of the Nuclear norm ∥•∥ * . We first measure the distribution difference between the source and blended-target domains through the nuclear-norm 1-Wasserstein discrepancy (NWD) [23] and then utilize a gradient reverse layer (GRL) [20] to maximize the discriminative loss of the classifier. Simultaneously, we minimize the feature generator to play the min-max game with the classifier through the NWD. First, the NWD loss can be defined as:EQUATIONThen, the adversarial learning strategy between feature generator and classifier is defined as follows:EQUATIONOverall Objective. The overall loss function that optimizes SAUE for MBDA is defined as:EQUATIONwhere λ e = min(1, e/λ ′ e ) ∈ [0, 1] is the annealing coefficient, which prevents L unc from overpenalizing the neural network to a uniform distribution in the early training epochs, and e is the current number of epochs and λ ′ e = 40. λ d is a hyper-parameter which is initially set to λ d = 1 as in [23] . L cls is the classification loss, which ensures that the category classifier can correctly classify samples. With the the cross-entropy loss L ce , the classification loss L cls is defined as follows:EQUATIONAfter adversarial training, our model can effectively adapt the blended-target domain without the requirement of domain labels of sub-target domains. The concrete steps of SAUE are shown in Algorithm 1.Theoretical Analysis. Here, we utilize PAC-Bayesian theory [36] to optimize our classification model with uncertainty estimation and elimination. The full-bound theorem motivated by previous work [37] is illustrated in Theorem 1 and Lemma 1, and the proofs are provided in Appendix C.Theorem 1 [37] . Suppose we have given the m-th source data distribution P Sm , a hypothesis set H, and a prior distribution π over the hypothesis space Θ. For any τ ∈ (0, 1] and λ > 0, with a probability at least 1 -τ over the source samples S m ∼ P Sm , for all posteriors ρ, we have:EQUATIONwhereΨ Sm,π (λ, n) = log E π(H) E Sm∼P Sm e λ(L(H)-L(H)) .Lemma 1 [38] . The PAC-Bayes bound, involving constants τ and n, as introduced in Theorem 1, is minimized by the Bayesian posterior p(H), which represents the distribution over Θ.During uncertainty estimation and elimination, just as in Eq. ( 9), we utilize D(p Sm i |α Sm i ) as the posterior distribution and D(p Sm i | ⟨1, . . . , 1⟩) as the prior distribution. Therefore, the upper bound of the classification model can be expressed as:EQUATION)Generalization Bound. In this part, we prove why SAUE performs well on the blended-target domain via Lemma 2 and Theorem 2. The proofs and derivations are provided in Appendix D.Lemma 2 [39] . Suppose we have given the probability measures ν Sm , ν T ∈ P(F) of the m-th source feature f Sm and the blended-target domain feature f T , a hypothesis space Θ, and a subspace H ∈ Θ.Let F denote a fixed representation space and c(f Sm , f T ) denote the adaptation cost. For the ideal classifier h ′ ∈ H and any classifier h ∈ H with f Sm ∼ ν Sm and f T ∼ ν T , we have:EQUATIONwhere ϵ Sm and ϵ T denote the error on the m-th source domain and the error on the blended-target domain respectively, and ϵ T = 1 N N j=1 ϵ Tj . d H∆H denotes the H∆H-distance. Theorem 2. Based on Lemma 2, with the error of the ideal joint hypothesis η ′ = ϵ Sm (h ′ ) + ϵ T (h ′ ) which is a sufficiently small constant, for any δ ∈ (0, 1), with probility at least 1 -δ, for every h ∈ H, ϵ T (h) is bounded by the following terms:EQUATIONwhere η ′ = ϵ Sm (h ′ ) + ϵ T (h ′ ) is the ideal error for the classifier, which is a sufficiently small constant. b ′ is the size of unlabeled samples.Therefore, the final objective of the MBDA classification task is to reduce the joint domain discrepancy term . Following the protocol used in [29] , we select 126 categories and 4 domains (C, P, R, and S) in our experiments. The VisDA 2017 [42] dataset is a challenging dataset consists of 2 domains (Syn. and Rel.) and 7 categories.M m=1 |ϵ Sm (h, h ′ ) -ϵ T (h, h ′ )|.Implementation Details. We utilize PyTorch framework [43] to perform our experiments; the PyTorch version is 1.13.1 and CUDA version is 11.7. We use an ImageNet pre-trained ResNet [44] , replacing the last FC layer with task-specific FC layers. All experiments are run on a single GeForce RTX-4090 GPU, and the batch size of both the source and blended-target domains are set to 32. The optimizer is Stochastic Gradient Descent (SGD) with a momentum parameter of 0.9 and a weight decay of 1e-3. The learning rate is 1e-3 and updated by the LambdaLR [43] during the training process. 
comparisons to state-of-the-art: 
 To evaluate the effectiveness of our proposed method, we conduct extensive experiments and compare our approach with the state-of-the-art (SOTA) methods in terms of DA classification. The comparison methods include SSDA approaches, i.e., MCD [45] , DAN [18] , TSA [46] , DALN [23] , BIWAA [47] ,and SCDA [22] ; MSDA methods, i.e., MDAN [48] , DCTN [49] , and DIDA [50] . MTDA/BTDA methods: MTDA-ITA [7] and MCDA [13] ; and Multi-source Multi-target DA (MMDA), i.e., AMDA [29] and HTA [51] . The comparison results are presented in Tables 2 3 4 , in which we select two domains as source domains and combine other two domains to form the blended-target domain. Note that these approaches do not totally match the MBDA setting. Therefore, we utilize the following rule for our comparison. For SSDA setting, one column denotes one SSDA task, such as R→C in Table 2 . For MSDA methods that contain more than two source domains, we implement those methods according to their released codes, reset the source domain into two domains, such as R+S→C, and mark them with "*". Similarly, under the MTDA and BTDA settings, we reset the target into two domains and select the highest one in MTDA/BTDA task group that contains the same target domains, such as R→ C+P and S→ C+P in Table 2 . For MMDA setting, two domains are sources, and the other domains are targets, such as R+S→C+P in Table 2 . For better comparison, all results in Tables 2-4 are the averages of two target domains. Results on the DomainNet are displayed in Table 2 . Our SAUE method achieves SOTA performance in most of the experimental groups and attains the best performance in terms of average accuracy. Compared to the BTDA method MCDA in multi-source setting, our method achieves better performance because the style information of the target domain selected by the weight factor can enhance the source feature representations. Compared to MMDA method AMDA, although AMDA can access the domain labels of the target domains, our method still overpasses AMDA in terms of average classification accuracy (overpass 7.5%) and without the requirement of the domain labels of the target domains. Furthermore, both AMDA and our method are adversarial learning methods, and the comparison results further demonstrate the effectiveness of our adversarial learning strategy. These obtained improvements are mainly due to the uncertainty optimization process and the style information derived from target features. three domains. Therefore, the adaptation task faces larger domain shifts and extremely unbalanced classes. The proposed method still achieves 2.8% improvements over AMDA and achieves dramatic improvements in the Rw+Pr→Ar+Cl and Rw+Ar→Pr+Cl tasks. These results occur because the proposed method decreases the impact of unbalanced classes by enhancing the feature representations and optimizing the prediction uncertainty.Results on the ImageCLEF-DA are provided in Table 3b . Compared with the SOTA methods, our proposed method achieves an average accuracy of 84.3%, outperforming the existing approaches. Note that all four domains in ImageCLEF-DA contain 600 images. Therefore, the experimental results further demonstrate that our proposed method is effective when all the domains contain the same samples and classes.Results on the Default Version of DomainNet and VisDA 2017. To evaluate the effectiveness of SAUE in different numbers of the source and target domains. We perform comparisons on the default version of the DomainNet dataset and the VisDA 2017 dataset, respectively. The default version of the DomainNet dataset consists of 5 domains, leading to the division of transfer tasks for MBDA into two categories: C+P+Q→R+S and R+S→C+P+Q. As shown in Table 4 , SAUE outperforms the comparison methods across all transfer tasks, achieving the highest average classification accuracy. These results from large-scale datasets further demonstrate the superiority and flexibility of SAUE. Sensitivity Analysis. We evaluate the model's performance under different hyperparameter choices. Note that the hyperparameters in our method are the adversarial learning balance parameter λ d and annealing parameter λ e . As shown in Figure 2 , we test different parameter groups to analyze the parameter sensitivity of our method, where λ d = {0.1, 0.5, 1.0, 1.5, 2.0} and λ e = {10, 20, 40, 60, 80}.The corresponding numerical results of Figure 2 are illustrated in Table 5 . In Figure 2 , our method is not sensitive to λ d and the best parameter choice is λ d = 1.0, but very sensitive to λ e (with λ e = 40 working best); if λ e too small, the model will suffer from tremendous degradation due to the model is over-penalized by uncertainty loss. Ablation Study. As listed in Table 6 , we conduct ablation experiments to demonstrate the effectiveness of the style adaptation module and the loss function of the uncertainty optimization process. We test three experimental groups in the DomainNet dataset, including that: 1) remove the style adaptation (SA) module, 2) remove the uncertainty loss L unc , and 3) remove both of the above-mentioned items.The results illustrate that both style adaptation and prediction uncertainty optimization are useful for MBDA. Furthermore, we explore different style adaptation techniques. We directly change the Wasserstein Distance (WD) to randomly selected style information and report the obtained results in the fourth row of middle part of Table 6 . Compared with the random augmentation methods, our proposed method can better select the style information through weight factors, which enhances the feature representations of the source domains and reduces the impact of domain shifts. More experiment analysis is provided in Appendix E. Effectiveness of the SA Module with Different Backbones. We have analyzed the performance of our method using the ResNet-50 and ResNet-101 backbones in the default version of the DomainNet dataset in Table 7 . The experimental results show that our method achieves significant performance gains when utilizing powerful backbones (ResNet-101). We compared the performance gains of the style adaptation module with standard backbone (ResNet-50) and powerful backbone (ResNet-101).The results indicate that the performance gains (+2.8% and +2.3%) of the style adaptation module when integrated into the ResNet-50 backbone surpass those (+1.6% and +1.3%) when integrated into the ResNet-101 backbone. Effectiveness of the SAUE with ResNet and ViT backbones. We further evaluate the model's performance using different backbones, as shown in Table 8 . When using ViT as the backbone, SAUE outperforms its performance with the ResNet-50 backbone. This performance gain is primarily due to the larger number of tunable parameters in ViT-B/16, demonstrating that our method effectively leverages these additional parameters to exploit transferable knowledge from multiple domains. 
conclusion: 
 In this paper, we propose a SAUE approach for MBDA, which utilizes information from multiple source domains to adapt a blended-target domain. In particular, the style adaptation process utilizes similarity factors to select target style information to enhance the representations of the source features. The uncertainty estimation procedure utilizes the Dirichlet distribution to estimate the uncertainty of the model and then adopts the KL divergence measure to optimize the prediction uncertainty. The discriminator-free adversarial learning strategy is beneficial for MBDA. Extensive experimental results demonstrate the superior performance of SAUE to that of the competing methods. 
 
 Appendix: This supplementary material provides more details that are not presented in the main paper due to space limitations. The organization is as follows:A Broader Impacts & Limitations B provides the derivations of P (y= c|x i ) = E[D(p ic |α i )].C provides the proof and derivations of Theorem 1.D provides the proof and derivations of the generalization bound.E provides additional experiment analysis on the ImageCLEF-DA dataset.Our work focuses on the problem of multi-source blended-target domain adaptation (MBDA), which aims to train a model that generalizes well on an unlabeled and distribution-confused target domain by leveraging multiple labeled source domains. The effectiveness of our method on several real-world datasets suggests that it can benefit relevant applications and communities dealing with domain shifts, such as encrypted data analysis, medical imaging, and autonomous driving. Nevertheless, we should also be cautious about potential failures of our method when encountering more significant distribution shifts, such as an increased number of source or target domains. In the future, we plan to incorporate more sub-domains into our experiments for further verifying the performance of our method.i ) = E[D(p ic |α i )]Given sample x i , for prediction of class c that generated by DNNs can be calculated as:EQUATIONwherep i = C(G(x i )) = [p i1 , p i2 , . . . , p iK ] and p ic is the c-th element of p i . Then, given ρ(p i |x i ) ∼ D(p i |α i ), we have ρ(p ic |x i ) ∼ Beta(p ic |α ic , α i0 -α ic ), where α i0 = K k=1 α ik . Thus, we further have:EQUATIONwhere B(•, •) is the K-dimensional multinomial beta function, and B(α ic , α i0 -α ic ) = Γ(αic)Γ(αi0-αic) Γ(αic+αi0-αic) , Γ(•) denotes the gamma function. Based on this, we can further obtain the following derivation:EQUATIONIn our work, the outputs of classifier are adopting the exponential function. Thus, following [53] , the outputs of DNNs in SAUE can be viewed as the expectation of Dirichlet distribution.Theorem 1 [37] . Suppose we have given the m-th source data distribution P Sm , a hypothesis set H, and a prior distribution π over the hypothesis space Θ. For any τ ∈ (0, 1] and λ > 0, with a probability at least 1 -τ over the source samples S m ∼ P n Sm , for all posteriors ρ, we have:EQUATIONwhere Ψ Sm,π (λ, n) = log E π(H) E Sm∼P n Sm e λ(L(H)-L(H)) .Lemma 1 [38] . The PAC-Bayes bound, involving constants τ and n, as introduced in Theorem 1, is minimized by the Bayesian posterior p(H), which represents the distribution over Θ.Proof. The Donsker-Varadhan's change of measure states that for any measurable function ϕ : Θ -→ R, we have:EQUATIONThus, with ϕ(H) := λ(L(H -L(θ, S m ) and ∀ρ over hypothesis space Θ, we have:EQUATIONFor the non-negative random variable ζ π (S m ) := E π(H) [e λ(L(H)-L(H,Sm)) ], we apply Markov's inequality on it, and have:EQUATIONThis impiles that with probability at least 1 -τ over the choice of S m ∼ P n Sm , we have ∀ρ over hypothesis space Θ:EQUATIONwhereΨ Sm,π (λ, n) = log E π(H) E Sm∼P n Sm e λ(L(H)-L(H)), and we prove the statement of the Theorem 1.Lemma 2 [39] . Suppose we have given the probability measures ν Sm , ν T ∈ P(F) of the m-th source feature f Sm and the blended-target domain feature f T , a hypothesis space Θ, and a subspace H ∈ Θ. Let F denote a fixed representation space and c(f Sm , f T ) denote the adaptation cost. For the ideal classifier h ′ ∈ H and any classifier h ∈ H with f Sm ∼ ν Sm and f T ∼ ν T , we have:EQUATIONwhere ϵ Sm and ϵ T denote the error on the m-th source domain and the error on the blended-target domain respectively, and ϵ T = 1 N N j=1 ϵ Tj . d H∆H denotes the H∆H-distance.Proof. By the definition of H∆H-distance, we have:EQUATION)Theorem 2. Based on Lemma 2, with the error of the ideal joint hypothesis η ′ = ϵ Sm (h ′ ) + ϵ T (h ′ ) which is a sufficiently small constant, for any δ ∈ (0, 1), with probility at least 1 -δ, for every h ∈ H, ϵ T (h) is bounded by the following terms:EQUATIONwhere η ′ = ϵ Sm (h ′ ) + ϵ T (h ′ ) is the ideal error for the classifier, which is a sufficiently small constant. b ′ is the size of unlabeled samples.Proof. From Lemma 2, we can obtain the following terms: ϵ T (h) ≤ ϵ T (h ′ ) + ϵ T (h, h ′ ) ≤ ϵ T (h ′ ) +Finally, the expected error on the blended-target domain can be bounded by utilizing the expected measures of NWD on the joint distribution of multiple source and blended-target domains.In this part, we present more visualization results with extra comparison methods including DANN [20] and AMDA [29] . All experiments are performed on task b+c-→i+p of the ImageCLEF-DA [40] .Distribution Analysis. The t-SNE [54] feature visualization results of extra comparison methods are illustrated in they still misclassified some classes, e.g., class "bike" is misclassified into class "motobike". In contrast, benefiting from the style adaptation and uncertainty estimation, SAUE generates more correct predictions which located on the main diagonal elements of confusion matrix.Convergence. We further analyze the evolution of different loss functions with increasing iterations. The results are shown in Figure 3 . The graphical representation illustrates that all the loss functions in our approach effectively converge as the training iterations increased. This convergence showcases the adaptability and reliability of our approach in MBDA tasks. 


Decision: This paper presents a Style Adaptation and Uncertainty Estimation approach to tackle the Multi-Source Blended Target Domain Adaptation problem, where target domain distributions are blended. During the rebuttal phase, the authors provided additional experiments and discussions on challenging backbones, datasets, and difficult UDA settings. Following the response and multiple rounds of discussion, the paper received four positive scores, including one accept, one weak accept, and two borderline accept. Considering the reviewers'  positive comments and the paper's contributions, the meta-reviewer gives the final recommendation of acceptance.

Reviews: review: paper_summary: paper_summary: The paper addresses the setting of Blended target domain adaptation. The paper proposes style adaptation and uncertainty estimation for multi-source blended target domain adaptation. They propose to utilize the extra knowledge acquired from the blended target domain. Where a similarity factor is utilized to obtain target style information to augment the source features. During training, they utilize the information from the target domain with a scaling factor and Wasserstein distance between the source and target features. Moreover, they do not utilize the domain labels without which they claim to align the multi-source and blended target domains. They utilize common domain adaptation datasets to demonstrate the effectiveness of their method.
strength_weakness: strength_weakness: The paper lacks extensive validation, clarity, and novelty. Elaborated below.
questions: questions: – Can the authors demonstrate with the default version of DomainNet instead of utilizing a subset of it? Moreover, the inclusion of the VisDA dataset would also be useful. 

– Could the authors also provide detailed numerical results of Figures C and D for better clarity? 

– The idea of using the target and source features during the training process with a scaling factor and Wasserstein distance doesn’t seem novel. Moreover the paper doesn't cite [1][2] which makes the related work quite limiting. 

– The idea of selecting style information is not new. Moreover, it is not clear which layer has been utilized for the style information and what the motivation is to utilize the style content at a particular layer. 

– The distribution analysis can be moved to the appendix, which doesn’t offer significant insights of the method. 

– Akin to MCDA [12], could the authors utilize ResNet 101 instead of ResNet50 and compare the results for the default version of the DomainNet dataset? With powerful backbones, it would be interesting how the style features are still preserved. 

– Extensive implementation details have not been provided; the batch size of the blended target domain, learning rates, and which layers from the ResNet model have been utilized for this if their code will be released. Does the method necessitate a high batch size for the target domain? In this case, it might be a limitation.

References:
[1] Shen, Jian, et al. "Wasserstein distance guided representation learning for domain adaptation." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.
[2] She, Qingshan, et al. "Improved domain adaptation network based on Wasserstein distance for motor imagery EEG classification." IEEE Transactions on Neural Systems and Rehabilitation Engineering 31 (2023): 1137-1148.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper proposes a "Style Adaptation and Uncertainty Estimation (SAUE)" method for Multi-source Blended-Target Domain Adaptation (MBDA). The core objective of the SAUE method is to enhance the source domain features by leveraging style information from a blended-target domain, which is a mixture of multiple sub-target domains. The method employs a similarity factor to select beneficial target style information and integrates an uncertainty estimation technique to fortify the model's robustness. The proposed SAUE is evaluated on several domain adaptation benchmarks, including ImageCLEF-DA, Office-Home, and DomainNet, demonstrating superior performance over existing state-of-the-art techniques.
strength_weakness: strength_weakness: 1. Although the blended-target domain adaptation has attracted much attention, it is similar to the open-compound domain adaptation that has no annotation of the target domain.  Authors should compare and discuss the difference between the open-compound domain adaptation works.

2. The proposed style adaptation and uncertainty estimation are motivated and effective ways to handle domain shift problems. However, the idea has been well studied in conventional domain adaptation [a], domain generation [b], and open-compound domain adaptation fields. Therefore, the proposed method seems to lack innovation. 

    [a] Informative Data Mining for One-shot Cross-Domain Semantic Segmentation

    [b] Uncertainty modeling for out-of-distribution generalization

    [c] Open Compound Domain Adaptation with Object Style Compensation for Semantic Segmentation

3. In my opinion, the proposed style adaptation technique is not tailored for multi-source BTDA as it also works for single-source BTDA. Therefore, the statement "the first work for MBDA" seems to be weak.
questions: questions: Detailed questions refer to the weaknesses.
limitations: limitations: The limitations of the work have been discussed.
paper_summary: paper_summary: This paper works on Multi-source Blended-target Domain Adaptation setting, which learns a model from multiple source domain and evaluates the model in a mixed multi-target domain without access to the domain labels of target data. This paper utilize the style information of the blended-target domain with weight factor to enhance source domain features for feature augmentation. To avoid the negative impact of domain-specific information in multi-source data, this paper also uses KL loss to reduce the impact of incorrectly classified source samples. This paper also proposes a domain adversarial loss with the help of nuclear-norm 1-Wasserstein discrepancy to reduce the domain gap between source domain and target domain by reusing the classifier of task network. Extensive experiments on various benchmark and comprehensive analysis prove the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The style adaptation method is not compared with prior works proposed for the same purpose like MixStyle.
2. Whether Domain Adversarial Alignment without Domain Labels outperform domain adversarial learning by treating multi-source data as one single source domain and multi-target data as one target domain is not discussed.
questions: questions: Please see weakness part.
limitations: limitations: No limitation is discussed in the paper.
paper_summary: paper_summary: This paper introduces a novel challenge in domain adaptation: the problem of Multi-Source Blended Target Domain Adaptation (MBDA). In MBDA, target domain distributions are blended, necessitating a model that can leverage features from multiple source domains to perform effectively on the blended-target domain. To address this problem, a Style Adaptation and Uncertainty Estimation (SAUE) approach is proposed. SAUE utilizes a similarity factor to select style information from the blended-target domain, creating a better representation space. Additionally, it enhances model robustness by employing a Dirichlet-based uncertainty estimation model to handle the diverse distributions introduced by multiple source domains. The effectiveness of the proposed method is validated through extensive image classification experiments. A theoretical analysis of SAUE underscores its potential in solving the MBDA problem.
strength_weakness: strength_weakness: 1. The paper mentions various terms related to domain adaptation in Sections 1 and 2, such as UDA, SSDA, MSDA, MMDA, MTDA, and BTDA. What are the main differences and relationships among these terms? The authors are encouraged to provide a table summarizing the main characteristics to distinguish these terms.
2. In the ablation study, the comparison primarily focuses on highlighting the improvements of each proposed module. It is suggested that the authors compare different feature augmentation methods to further emphasize the superiority of the proposed Style Adaptation module.
3. A minor point of interest is how the proposed method performs on other backbones, such as the Vision Transformer.
questions: questions: Please refer to the Weaknesses part.
limitations: limitations: The authors have adequately claimed their limitations in Appendix B.

Comments: title: Thanks for your support
Thanks for your support
Thanks for the authors' responses
comment: Dear Reviewer NQ8Y,

We are truly thankful for your valuable time and constructive feedback!

Since the discussion deadline is approaching (11:59pm AoE on August 13), we would like to kindly inquire if our rebuttal has addressed your concerns. We are willing to address any further issues you might have : )

Sincerely,

Authors
Dear Reviewer cWFB,

We are truly thankful for your valuable time and constructive feedback!

Since the discussion deadline is approaching (11:59pm AoE on August 13), we would like to kindly inquire if our rebuttal has addressed your concerns. We are willing to address any further issues you might have.

Sincerely,

Authors
Thank you for your exceptionally prompt feedback and unwavering support of our paper. We are glad to observe that our responses have effectively resolved your concerns. Your invaluable input has undeniably enhanced the quality of our paper, and we sincerely thank you for your dedication and time.
Thank you for your active involvement and prompt feedback during the discussion phase. We are pleased to know that our rebuttal has effectively addressed your questions. We will carefully incorporate all of your valuable suggestions into the final version of our paper, including results on DomainNet (345 classes), discussions on feature learning and style information, as well as additional implementation details. Your constructive review has undoubtedly enhanced the quality of our paper, and we sincerely appreciate your dedication and time.
Thank you for your efforts. My comments have been addressed. I would encourage the authors to include DomainNet (345 classes) and additional comparisons to it in the main paper, as well as discussions about feature learning and style information and additional implementation details in the manuscript. In response, I have increased my score.
After thoroughly reviewing the questions raised by other reviewers and carefully considering the authors' responses, I am convinced that my concerns have been effectively addressed. I believe the proposed method is inspiring for real-world complex environments. As a result, I would like to raise my score.