MultiDAN: Unsupervised, Multistage, Multisource and Multitarget Domain Adaptation for Semantic Segmentation of Remote Sensing Images

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 11:01:21

Abstract: Unsupervised domain adaptation (UDA) has been a crucial way for cross-domain semantic segmentation of remote sensing images and reached apparent advents.
However, most existing efforts focus on single source single target domain adaptation, which don't explicitly consider the serious domain shift between multiple source and target domains in real applications, especially inter-domain shift between various target domains and intra-domain shift within each target domain.
In this paper, to address simultaneous inter-domain shift and intra-domain shift for multiple target domains, we propose a novel unsupervised, multistage, multisource and multitarget domain adaptation network (MultiDAN), which involves multisource and multitarget domain adaptation (MSMTDA), entropy-based clustering (EC) and multistage domain adaptation (MDA).
Specifically, MSMTDA learns feature-level multiple adversarial strategies to alleviate complex domain shift between multiple target and source domains.
Then, EC clusters the various target domains into multiple subdomains based on entropy of target predictions of MSMTDA.
Besides, we propose a new pseudo label update strategy (PLUS) to dynamically produce more accurate pseudo labels for MDA.
Finally, MDA aligns the clean subdomains, including pseudo labels generated by PLUS, with other noisy subdomains in the output space via the proposed multistage adaptation algorithm (MAA).
The extensive experiments on the benchmark remote sensing datasets highlight the superiority of our MultiDAN against recent state-of-the-art UDA methods.

Main paper: 
 related work 2.1 single source single target domain adaptation: 
 The aim of single source single target domain adaptation (SDA) is to adapt a transferable classifier to align the domain shift between one unlabeled target domain and one annotated source domain.Recently, SDA has been widely utilized for cross-domain semantic segmentation of natural images [1, 20, 27, 35, 36, 45] and remote sensing images [3, 4, 14, 28, 41] . For instance, Benjdira et al.[2] firstly trained a CycleGAN [52] to align the target and source domains in the image space. Then they applied the transformed target-like images to train the target segmentation model. On this basis, Li et al. [22, 33] further adopted high-confident predicted results of target classifier as pseudo labels, so as to optimize the target classifier according to self-supervised learning. Besides, some stage-wise UDA methods [3, 26, 43] proposed to further deal with the intra-domain shift within target domain, after mitigating the inter-domain shift between target domain and source domain. Although such recent SDA approaches have significantly bridged the domain gap between single source domain and single target domain, these methods don't explicitly consider multiple target and source domains, which would be more complex and difficult to solve in real applications.Multisource domain adaptation (MSDA) focuses on alleviating the distributional discrepancy between multiple source domains and single target domain [8, 12, 19, 29, 40, 46, 47] , while Multitarget domain adaptation (MTDA) aims to alleviate the domain shift between single source domain and multiple target domains [10, 13, 42, 51] . For MSDA problems, Lu et al. [11, 23] utilized multiple incomplete source domains to form the categories of target domain. Zhao et al. [46, 47] transformed multisource images to target-like domains and eliminate the pixel-level distribution gap between multisource images. For MTDA issues, Gholami et al. [10] proposed an information theoretic model and explored the common intrinsic space for the single source and multitarget data. Isobe et al. [13] extended [25] to semantic segmentation task and added a pixel-wise regularization to enhance the cross-domain segmentation performance. However, the MSDA methods [16, 21, 30, 48, 49] don't explicitly take into account complex distribution discrepancy across various target domains while the MTDA methods [25, 31, 32] can't fully exploiting the advantageous knowledge from multiple source domains. The significant misalignment across simultaneous multitarget and multisource will apparently weaken the model performance of MSDA and MTDA methods. Thus, the MSDA and MTDA methods still lead to sub-optimal solutions when multisource and multitarget domains are available.Multisource and multitarget domain adaptation (MSMTDA) is proposed to eliminate the domain shift between multiple target domains and multiple source domains. However, only very few studies address the MSMTDA problem [34, 37, 38] , as it is much more difficult and complex than SDA, MSDA and MTDA problems. Tasar et al. [34] adopted image translation models to conduct image-level alignment between multiple target domains and multiple source domains of satellite images. Then they trained the segmentation model on the diversified target-like images, which were randomly transferred from source images to one of the target domains, to make the segmentation model more robust to various target domains. Wang et al. [37] proposed to utilize numerous adversarial strategies to align each pair of source and target domains as well as each pair of target domains, so as to learn the common features of multiple target and source domains. Wu et al. [38] conducted hierarchical structure-level, domain-level and class-level alignments between multiple source and target domains. Although the beforementioned MSMTDA models can better handle the complex domain shift issues between multisource and multitarget data, they don't explicitly consider the intra-domain shift within every target domain alone with the inter-domain shift between various target domains, which has been validated to deteriorate the model performances. As a result, there are still some terrible predicted results for multitarget images after MSMTDA. 
method: 
 We formalize the problem of MSMTDA for semantic segmentation, where M labeled source domains {D 1 S , D 2 S , ..., D M S } and Nunlabeled target domains {D 1 T , D 2 T , ..., D N T } are available. Specif- ically, each source domain D i S (i ∈ {1, 2, ..., M }) includes source images x i s ∈ R H ×W ×3 with C-category pixel-level annotations y i s ∈ (1, C) H ×W ×C , while every target domain D p T (p ∈ {1, 2, ..., N }) involves target images x p t ∈ R H ×W ×3 with no annotations.As shown in Figure 2 , the proposed MultiDAN consists of three parts. The first part is MSMTDA module adopting feature-level multiple adversarial strategies to reduce domain shift across multiple source and target domains. Secondly, EC computes the mean entropy of target predictions generated by MSMTDA module, and then uses the entropy to cluster the diversified target images obtained from various target domains into multiple subdomains for MDA module. Thirdly, MDA module applies output-level multiple adversarial strategies to align the clean subdomains, included pseudo labels generated by our PLUS, with other noisy subdomains via the proposed MAA.The MSMTDA module follows the main spirit of multiple adversarial frameworks [27, 37] , which cope with the multisource and multitarget domain shift problem in an effective and simple way.Besides, an adaptive weighting strategy (AWS) [17] is applied to enhance the performance of segmentation model without extra manual efforts of hyperparameter tuning.The MSMTDA module trains the source-target segmentation model F ST to learn the common features across multiple source and multiple target domains. Because there is no annotations for the multiple target domains, the source-target segmentation model F ST is optimized on the multiple source domains in a fully-supervised manner. To be specific, for every source domain D i S (i ∈ {1, 2, ..., M }) included images x i s along with labels y i s , we train the source-target classifier F ST through minimizing the popular cross entropy loss with class-balanced factors [7] :L s i seд (F ST ) = - c α c (y i s ) c log(F ST (x i s ) c ) (1)where α c is a class-balanced factor for every class c ∈ C. α c denotes the inverse class frequency of effective number of class c [7] . And α c can be computed as:α c = 1-β 1-β nc ,where n c represents the pixel number of class c. β is constant and set to 0.999. Moreover, α c is normalized asα c = α c C c =1 α c to make C c=1 α c = 1, which enforces all the α c (c ∈ C) within a close scope.In order to alleviate the source-target domain shift between multiple target and source domains, we adopt feature-level multiple adversarial strategies [27, 37] to learn the shared intrinsic feature space of multiple source and target domains. Specifically, to fully consider the context dependencies across multiple target domains and multiple source domains, we firstly pair every source domain with every target domain and obtain M × N pairs ( {D 1 S , D 1 T },..., {D M S , D N T }). For each pair of source-target domains, we align source domain with target domain in the feature space through adversarial learning [45] . We apply the source-target discriminator D s i t p to differentiate the features U i s of x i s from features U EQUATIONwhere U i s and U p t are the features of x i s and x p t . Second, we pair every target domain with another target domain in a similar way and acquireN ×(N -1) 2 pairs ({D 1 T , D 2 T },..., {D N -1 T , D N T }).For each pair of target-target domains, we align the two different target domains in the feature space through adversarial learning [45] . The target-target discriminator D t p t q is adopted to distinguish the features U p t of x p t from features U q t of x q t , while the classifier F ST is trained to fool D s i t p . Such target-target adversarial learning loss L t p t q adv for optimizing F ST and D t p t q on target-target pair {D p T , D q T } (p, q ∈ {1, 2, ..., N } and p q) can be expressed asL t p t q adv (F ST , D t p t q ) = -(log(1-D t p t q (U p t )) +log(D t p t q (U q t ))) (3)where U p t and U q t are the features of x target-target discriminators, but we utilize two multitask discriminators {D s i t p ,D t p t q } (see Figure 3 ) instead, where discriminator D s i t p is used for different pairs of source domains and target domains, D t p t q is applied for different pairs of target domains. In detail, the multitask discriminator includes four shared convolutional layers and one specific convolutional layers for different source-target pairs (or target-target pairs) as depicted in Figure 3 . In this way, when the number of source and target domains increases, we only need to add specific convolutional layers, instead of adding discriminators.To reduce time consumption and manual efforts for tuning hyperparameters as well as obtaining the optimal weights between every loss function, we adopt an adaptive weighting method [17] in our final optimization objective. It could adaptively learn the weights between the segmentation loss L s i seд , source-target adversarial learning loss L s i t p adv and targettarget adversarial learning loss L t p t д adv . The final adaptively weighted optimization objective of MSMTDA module can be expressed asEQUATIONWhere i ∈ {1, 2, ..., M } and p, q ∈ {1, 2, ..., N } (p q). Parameters θ = {θ s i , θ m ip , θ t pq } are learnable and adaptive during the training process. To solve the inter-domain shift between various target domains and intra-domain shift within each target domain, we utilize entropybased clustering (EC) to cluster the diversified multitarget domains into multiple subdomains in accordance with the confidence level of predictions generated by MSMTDA module. In this way, we can tackle the multiple inter-domain shift and intra-domain shift problems of multitarget domains as multiple inter-domain shift problem between different subdomains. Because of no target annotations, we utilize mean entropy of softmax probability P t (P t = F ST (x t ))to assess the confidence level of target predictions [36] . The entropy I t can be computed asEQUATIONwhere h ∈ H and w ∈ W . H × W denote the image size of target predictions.Then, we rank and split all target images x t via calculating the mean entropy grade R(x t ) of corresponding target predictions P t . The mean entropy grade R(x t ) can be defined asEQUATIONOn this basis, we propose to separate and rank N target domains into multiple (K × N ) subdomains { D 1 , D 2 , ..., D N K } and proposed a novel MAA to alleviate the complex multiple domain shift problem. Note that K is a hyperparameter that controls the subdomain number for each target domain and stage number for MAA.To further reduce the domain shift between multiple subdomains, we propose a novel multistage domain adaptation (MDA) module. In general, we align the output space of clean subdomains, including pseudo labels produced by our PLUS, with that of noisy subdomains via the proposed MAA. And MDA module is also trained with AWS.Because of lack of target annotations, it isn't practicable to directly adapt the target classifier on the multiple target subdomains. Some UDA methods [3, 26, 43] proposed to utilize the target predictions of the classifier as pseudo labels for self-supervised learning. However, there may still be some incorrect pixels in the clean predictions during the self-training procedure, even if these clean predictions are relatively low-entropy and clean on the whole. To obtain the optimal pseudo labels during the iterative training process, we propose a novel pseudo label update strategy (PLUS) to produce more accurate pseudo labels as much as possible. Concretely, for each self-training stage, we apply the newly optimized target classifier F T to output predictions P t and compute entropy I t for x t . Besides, we employ two-dimensional (H × W ) matrix P and I, which have the same size as the predictions P t and entropy I t , to store the maximum probability value and minimum entropy value of every pixel within target image x t . Then for each pixel x (h,w ) t of x t , we update pseudo labels by assigning the class label, which has a larger probability value P (h,w,c ) t than the corresponding value µ (h,w ) ∈ P or has a smaller entropy value I (h,w ) t than the corresponding value υ (h,w ) ∈ I. At the same times, we update matrix P and I with the larger probability value or the smaller entropy value for each pixel. We formulate the proposed PLUS as follows:ŷ(h,w ) t =              c, if P (h,w,c ) t > µ (h,w ) or I (h,w ) t < υ (h,w )and argmax c P (h,w, c ) t = c 0, otherwise (7) where µ (h,w ) ∈ P and υ (h,w ) ∈ I are the probability threshold and entropy threshold for pixel x (h,w ) t of x t respectively. The update strategy for matrix P and I can be expressed in the following equation:EQUATION)where µ (h,w ) ∈ P and υ (h,w ) ∈ I are the existing values in matrix P and I. μ (h,w ) and υ(h,w ) are the new values to be updated in matrix P and I. The initial value of µ (h,w ) and υ (h,w ) are set to µ 0 and υ 0 . MAX and MIN are functions which pick up the maximum value and minimum value between different values respectively.To address the severe domain shift between multiple subdomains, we propose a novel multistage adaptation algorithm (MAA), which utilizes multiple clean subdomains along with pseudo labels generated by PLUS to adapt other multiple noisy subdomains in a multistage manner as described in Algorithm 1. Specifically, in every self-training stage, we applied the proposed PLUS to generate and update pseudo labels for subdomains {D 1 , ..., D N k }. Second, we cluster and update subdomains {D 1 , ..., D N k } into N clean subdomains {D 1 cl ean , ..., D N cl ean } according to mean entropy grade R(x t ). Then, we treat N clean subdomains along with pseudo labels as the labeled multisource domains, and deal with N noisy subdomains {D N k+1 , ..., D N (k+1) } as unlabeled multitarget domains. Similar to MSMTDA module, we utilize a target segmentation module F T and two multitask discriminators {D c j n p ,D n p n q } to narrow the clean-noisy domain shift between multiple clean (multisource) subdomains and multiple noisy (multitarget) subdomains.Firstly, we train the target segmentation model F T to extract the shared features across multiple clean subdomains and multiple noisy subdomains. Because of lack of annotations for the multiple noisy subdomains, the target segmentation model F T will be trained The trained networkEQUATIONEQUATIONn p n q ← D n p n q 2 for k = 1 to K -1 do 3 input images {x 1 ∈ D 1 , ..., x N k ∈ D N k } into F (k -1) Tand update their pseudo labels { ŷ1 , ..., ŷN k } with Equations ( 7), ( 8) and ( 9) 4 update {(x 1 , ŷ1 ) ∈ D 1 , ..., (x N k , ŷN k ) ∈ D N k } into D clN k+1 ∈ D N k +1 , ..., x N (k +1) ∈ D N (k +1) } to train F (k ) T ← F (k-1) T , D (k ) c j n p ← D (k -1) c j n p , D (k ) n p n q ← D (k -1)n p n q with Equations ( 10), ( 11), ( 12), ( 13)7 end 8 return F (K -1) T , D (K -1) c j n p , D (K -1) n p n qon the multiple clean subdomains with pseudo labels in a selfsupervised manner. For every clean subdomain D j (j ∈ {1, ..., N }) included images x j cl ean along with pseudo labels ŷj cl ean , we adapt the target classifier F T via minimizing the class-balanced cross entropy loss [7] :EQUATIONwhere α c is a class-balanced factor as described earlier.Secondly, to eliminate the inter-domain shift between multiple clean subdomains (source) and multiple noisy subdomains (target), we utilize output-level multiple adversarial strategies [27, 37] to learn the common intrinsic space of multiple subdomains. In detail, we firstly pair every clean subdomain with every noisy subdomain and acquire N × N pairs. For each pair of clean-noisy subdomains, we align clean subdomain with noisy subdomain in the output space by adversarial learning [36] . The clean-noisy adversarial learning loss L c j n p adv for optimizing classifier F T and clean-noisy discriminator D c j n p can be expressed as (11) where I j c and I p n are the entropy of clean subdomain j ∈ {1, ..., N } and noisy subdomain p ∈ {Nk + 1, ..., N (k + 1)}. And entropy map I can be calculated bu Equation (5) .L c j n p adv (F T , D c j n p ) = -(log(1 -D c j n p (I j c )) + log(D c j n p (I p n )))Thirdly, we pair every noisy subdomain (target) with another noisy subdomain (target) in a similar way and obtainEQUATIONpairs. For each pair of noisy-noisy subdomains, we align the different subdomains in the output space via adversarial learning [36] .The noisy-noisy adversarial learning loss L n p n q adv for optimizing classifier F T and noisy-noisy discriminator D n p n q (p q) can be written asEQUATIONwhere I p n and I q n are the entropy of noisy subdomain p and q respectively (p, q ∈ {Nk + 1, ..., N (k + 1)}).Similar to the adaptively weighted MSMTDA module, we use adaptive weighting method [17] in our final optimization objective of MDA module, which can adaptively learn the weights between the target segmentation loss L EQUATIONWhere j ∈ {1, ..., N } and p, q ∈ {Nk + 1, ..., N (k + 1)} (p q). Parameters σ = {σ c j , σ m jp , σ n pq } are learnable and adaptive during the training process. 
experiments 4.1 experimental setup: 
 4.1.1 Dataset. We validate the proposed MultiDAN for MSMTDA problem on the public aerial image segmentation (AIS) datasets [15] collected form Tokyo, Berlin, Paris, Chicago, Potsdam and Zurich cities. Potsdam, Chicago, Paris and Zurich datasets include 24, 457, 625 and 364 annotated aerial images with near 3000 × 3000 resolution, respectively. Berlin and Tokyo datasets involve 200 and 1 annotated aerial images with about 2500 × 2500 resolution, respectively. For the semantic annotations, blue, white and red colors correspond to road, background and building respectively. 4.1.2 Implementation Details. We apply Deeplabv3+ [6] as semantic segmentation model F . The multitask discriminators D utilize five convolutional layers as illustrated in Figure 3 . During the training phase, MSMTDA and MDA modules of the proposed MultiDAN can be separately trained in sequence. Firstly, the MSMTDA module is trained for 100 epochs with loss function L mst in Equation ( 4). The initial values of adaptive weighting parameter {θ s i , θ m ip , θ t pq } are uniformly set to 1, 0.05 and 0.02 respectively. In this stage, the segmentation model F ST , discriminators {D s i t p ,D t p t q } and parameter θ are trained jointly via Adam optimizer [18] with β 1 = 0.9 and β 2 = 0.999. The batch size and learning rate are set to 12 and 10 -4 . Secondly, the trained segmentation model F ST is utilized to segment the multiple target domains. We calculate the entropy grade R(x t ) (Equation ( 6)) of all the target predicted maps, and sort the target predictions with correspond images according to R(x t ). Then the sorted target images are separated into multiple subdomains evenly by setting K = 4. Thirdly, the MDA module are leaned with Algorithm 1. Specifically, the initial values of probability threshold in Equation ( 8) and entropy threshold in Equation ( 9) are determined by pixel ratio ρ u = 75% and ρ v = 65%. The initial values of adaptive weighting parameter {σ s i , σ m ip , σ t pq } are uniformly set to 1, 0.05 and 0.02 respectively. In every self-training stage, the segmentation model F T , discriminators {D c j n p ,D n p n q } and adaptive weighting strategy σ are jointly trained for 100 epochs through Adam optimizer [18] with β 1 = 0.9 and β 2 = 0.999. The batch size and learning rate are set to 12 and 10 -4 . All experiments are conducted on two NVIDIA Tesla P40 GPUs.We compare our MultiDAN with recently published state-of-theart (SOTA) UDA models involving SDA methods [3, 20] , MSDA methods [21, 46] , MTDA methods [31, 51] and MSMTDA methods [34, 37] . When training the SDA, MSDA and MTDA methods on the multisource and multitarget domains, we follow the related works [12, 13, 30-32, 46, 51] and integrate all source domains or target domains into one source domain or one target domain respectively. Tables 1 and 2 show the segmentation results of the proposed MultiDAN and the SOTA UDA methods. The baseline Deeplabv3+ performs the worst on multisource and multitarget AIS datasets. All the UDA models lead to apparent performance improvements. We can find that MSDA methods and MTDA methods are generally better than the SDA models. At the same time, MSMTDA approaches surpass the MSDA methods and MTDA methods. These results show the effectiveness and superiority of MSMTDA methods, which address the severe domain shift between multiple source and target domains while the MSDA and MTDA methods ignore the useful knowledge across multitarget data or multisource data. Among all the MSMTDA models, the proposed MultiDAN yields the highest mIoU and mF1. This highlights our MultiDAN has more advantages in MSMTDA tasks compared with the SOTA UDA approaches. Figures 4 and 5 draw the visual segmentation results of the UDA models of Tables 1 and 2 respectively. 3 , every component and combination of components can bring performance improvements over the baseline method, which verifies the effectiveness of each component of our MultiDAN in MSMTDA tasks. We compare the proposed PLUS and the popular pseudo label strategy adopted in recent SOTA UDA frameworks. Table 5 gives the quantitative performance of directly utilizing target predictions without pseudo label strategy (No), softmax probability threshold (ST) [22] , entropy threshold (ET) [33] , combination of softmax probability threshold and entropy threshold (EST) [3] and our PLUS (Ours). As shown in Table 5 , all the pseudo label approaches surpass directly utilizing target predictions without pseudo label strategy (No), proving the necessity of pseudo label strategies. Among all the pseudo label strategies, our PLUS achieves the best segmentation performance, which highlights the competitiveness of our PLUS. In supplementary materials, we show the visual feature distributions of our MultiDAN and comparing methods, and visual segmentation predictions and entropy the proposed MultiDAN with various stage numbers (subdomain number) K. Then, we discuss the methods of determining the initial values of probability threshold µ in Equation ( 8) and entropy threshold υ in Equation ( 9), and probe the effect of initial values of µ and υ. Besides, we validate the effectiveness of adaptive weighting strategy (AWS) and study the effect of different source and target domains. 
conclusion: 
 This paper proposes a multistage, multisource and multitarget UDA network called MultiDAN to further solve serious multiple domain shift problem in practical applications of remote sensing images, consisting of simultaneous inter-domain shift between various target domains and intra-domain shift within each target domain.Extensive experiments on the open-source benchmark remote sensing data sets demonstrates the competitiveness and superiority of the proposed MultiDAN over the existing SOTA UDA models.  
: 
 remote sensing image interpretation applications, for example, climate change assessment, urban management, land cover monitoring, etc. Though these semantic segmentation models can achieve satisfactory performances [9] in a fully-supervised manner, they suffer from ubiquitous domain shift problems [28, 39, 41] in practical uses. That is to say, when the classifiers trained on training data (source domain) are directly employed to segment remote sensing images (target domain) drawn from different data distribution, their model performance will apparently drop.To address the aforesaid issues, unsupervised domain adaptation (UDA) has been widely adopt to replace manual annotating for unlabeled images, and shown remarkable advances for cross-domain semantic segmentation in remote sensing community [14, 28, 41] . In summary, the majority of UDA methods focus on aligning source domain and target domain in the feature space [27, 45] , output space [35, 36] and image space [2, 4] . On this basis, some self-supervised [22, 33] and stage-wise [3, 26, 43] adaptation methods are proposed to to further eliminate the serious UDA problems. Although the existing UDA approaches have shown significant progress for semantic segmentation of remote sensing images, most methods focus on single source single target domain adaptation (SDA) settings [3, 28, 41] . Such SDA settings limit the performance of existing UDA methods in real-world applications. Since in remote sensing community, each remote sensing image can be viewed as a single domain [34] due to various imaging modes. Thus the training data and test data usually involves multiple source domains and multiple target domains.To fully exploit the multiple domain knowledge, some multisource single target domain adaptation (MSDA) methods [19, 29, 44, 49] explore the abundant information from multiple source domains to adapt the classifier on a single target domain. Moreover, a few single source multitarget domain adaptation (MTDA) methods [24, 32, 50, 51] transfer the classifier trained on a single source domain to multiple target domains and exploit complementary knowledge among various target domains. Furthermore, some multisource and multitarget domain adaptation (MSMTDA) methods [34, 37] learn sufficient and complementary information from multisource and multitarget domains simultaneously, which can better eliminate the serious domain shift between multiple source and target domains against the MSDA and MTDA methods. However, in the MSMTDA scenarios, besides the domain shift across multisource and multitarget domains, there still exists severe multiple domain shift problem. As depicted in Figure 1 , multiple domain shift problem involves inter-domain shift across various target domains along with simultaneous intra-domain shift within each target domain, which hasn't been well addressed by the existing MSMTDA methods. Specifically, the inter-domain shift between different target domains is caused by differences in imaging process. The intra-domain shift within each single domain usually results from intra-class variation, illumination, diverse scene distributions and other factors. From Figure 1 , we can see that the adapted target classifier can output many accurate and low-entropy predictions on both target domains after MSMTDA. However, there still exist some high-entropy and terrible predicted results for both target domains. This indicates the simultaneous inter-domain shift and intra-domain shift will significantly cripple the segmentation performance of MSMTDA models.In this paper, we propose a novel unsupervised, multistage, multisource and multitarget domain adaptation network named Mul-tiDAN, which includes multisource and multitarget domain adaptation (MSMTDA), entropy-based clustering (EC) and multistage domain adaptation (MDA). Concretely, the MSMTDA module firstly learns feature-level multiple adversarial strategies to mitigate the source-target domain shift between multiple target and source domains as well as target-target domain shift across various target domains, with adaptive weighting strategy (AWS) to reduce the manual efforts of hyperparameter tuning. Second, built upon the confidence level of target predictions generated by MSMTDA module, EC clusters the various target domains into more fine-sorted subdomains for MDA module to solve the multiple domain shift problem. Then, to generate more confident pseudo labels for MDA module, we propose a novel pseudo label update strategy (PLUS), which dynamically update pseudo labels with high-confident and low-entropy predictions during every self-training stage. At last, to reduce the inter-domain shift between subdomains, MDA module adopts output-level multiple adversarial strategies and AWS to align the clean subdomains, including pseudo labels produced by PLUS, with other noisy subdomains by the proposed multistage adaptation algorithm (MAA).In conclusion, our main contributions are as follows:  
 
 Appendix:  


Decision: The paper proposes a novel approach for the task of semantic segmentation for remote sensing data. Reviewers noted the paper's clear design and well-illustrated goals, with each module addressing specific problems effectively. While some concerns about computational complexity, and the identification of clean and noisy subdomains were raised, the authors' responses were mostly satisfactory. It is suggested that the authors carefully revise the paper in the camera-ready version according to reviewers' comments.

Reviews: review: paper_summary: paper_summary: This paper proposes a unsupervised, multistage, multisource and multitarget domain adaptation network (MultiDAN), which involves multisource and multitarget domain adaptation (MSMTDA), entropy-based clustering (EC) and multistage domain adaptation (MDA). Besides, a new pseudo label update strategy (PLUS) is proposed. The function of each module is clear and the whole target of the paper is well illustrated.
strength_weakness: strength_weakness: 1. The paper is well designed and the paper writing is clear.
2. The experiments are adequate.
3. Each module is designed to address a problem, and it is reasonable to do so.
limitations: limitations: 1. What about the selection of K affecting the computation complexity?
2. Can the PLUS be used on other methods?
3. How are the clean and noisy subdomains identified?
paper_summary: paper_summary: This paper proposes a multi-stage framework for the task of Semantic Segmentation of Remote Sensing Images. The proposed method involves mainly three stages including a network for target prediction, a clustering module, and multistage domain adaptation. The authors aim to solve the issues of both inter- and intra-domain shift. Experimental results show that the proposed method performs favorably against the SOTA methods on the evaluated datasets.
strength_weakness: strength_weakness: 1. The motivation is well explained and the research context is clearly presented.
2. The experimental results of the method are good and the implementation details are described in detail.
limitations: limitations: 1. While the authors emphasize the importance of addressing intra- and inter-domain shifts simultaneously, the paper fails to demonstrate what bad effects can these two issues cause, respectively. On the contrary, these two issues are dealt with as a single issue as “the problem of the mixture of multi-source images”. Moreover, the experiments did not verify how the inter-domain and intra-domain issues were solved.
2. Concerning the tuning of hyper-parameters, although the paper presents a method of learning the parameters in equation 4, more theoretical and experimental justifications are needed to show why optimizing equation 4 can lead to the optimal solution of the problem
involving the three loss functions.
3. The model complexity of the methods in table 1 and 2 should be compared, since we can infer from the paper that the proposed method may require a large number of model parameters and FLOPs.
4. It is unclear why the model performance continues to become worse as the model is trained with more stages when K exceeds 4. After all, the added stages are supposed to refine the results of previous stages.
paper_summary: paper_summary: This paper proposes to handle the Multisource and Multitarget Domain Adaptation for Semantic Segmentation of Remote Sensing Images by adopting the MultiDAN method. Specifically, the multistage adaptation algorithm (MAA) mainly focuses on the inter-domain shift and the intra-domain shift. The pseudo-label update strategy (PLUS) is utilized to dynamically update the high-confidence or low-entropy pseudo-labels for unlabeled target domains. Experimental evaluation shows the effectiveness of the proposed method.
strength_weakness: strength_weakness: *	The authors reveal a critical gap in existing MSMTDA methods, which often overlook the issue of multiple domain shifts, including both intra-domain and inter-domain shifts within multiple target domains.
*	Experimental analysis is thorough, where the previous methods not tailored for the new task are reproduced.
limitations: limitations: *	In Fig. 1, the authors illustrate the low-entropy and high-entropy predictions generated by the adopted target classifier. However, the comparison between low-entropy predictions and high-entropy predictions is not intuitive enough. Suitable statistical techniques can be conducted to enhance the clarity of Fig. 1.
*	In Sec. 3.2, the authors describe the entropy and the mean entropy grade in Eqs. 5 and 6, respectively. However, as the main topic of Sec. 3.2 is entropy-based clustering, the detailed steps of the clustering process can be further illustrated.
*	The authors claim that existing MSMTDA methods overlook the problems of inter- and intra-domain shifts. However, the inter- and intra-domain attention modules proposed in AMDA [1] already address these issues. The authors are suggested to further highlight the superiority of their approach in addressing inter- and intra-domain shifts compared to AMDA.
*	In Sec 4.2, the authors claim they compare their method with the MSMTDA method AMDA. However, the relevant experiment results are not illustrated on Tabs. 1 and 2.

[1] Yuxi Wang, Zhaoxiang Zhang, Wangli Hao, and Chunfeng Song. 2020. Attention guided multiple source and target domain adaptation. IEEE TIP, 30: 892-906.