Connectivity-Driven Pseudo-Labeling Makes Stronger Cross-Domain Segmenters

Authors: dong  zhao
qi  zang
shuang  wang
nicu  sebe
zhun  zhong
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 23:23:28

Abstract: Presently, pseudo-labeling stands as a prevailing approach in cross-domain semantic segmentation, enhancing model efficacy by training with pixels assigned with reliable pseudo-labels.  However, we identify two key limitations within this paradigm: (1) under relatively severe domain shifts, most selected reliable pixels appear speckled and remain noisy. (2) when dealing with wild data, some pixels belonging to the open-set class may exhibit high confidence and also appear speckled. These two points make it difficult for the pixel-level selection mechanism to identify and correct these speckled close- and open-set noises. As a result, error accumulation is continuously introduced into subsequent self-training, leading to inefficiencies in pseudo-labeling. To address these limitations, we propose a novel method called Semantic Connectivity-driven Pseudo-labeling (SeCo).  SeCo formulates pseudo-labels at the connectivity level, which makes it easier to locate and correct closed and open set noise. Specifically, SeCo comprises two key components: Pixel Semantic Aggregation (PSA) and Semantic Connectivity Correction (SCC). Initially, PSA categorizes semantics into ``stuff'' and ``things'' categories and aggregates speckled pseudo-labels into semantic connectivity through efficient interaction with the Segment Anything Model (SAM).  This enables us not only to obtain accurate boundaries but also simplifies noise localization. Subsequently, SCC introduces a simple connectivity classification task, which enables us to locate and correct connectivity noise with the guidance of loss distribution. Extensive experiments demonstrate that SeCo can be flexibly applied to various cross-domain semantic segmentation tasks, \textit{i.e.} domain generalization and domain adaptation,  even including source-free, and black-box domain adaptation, significantly improving the performance of existing state-of-the-art methods. The code is provided in the appendix and will be open-source.

Main paper: 
 introduction: 
 Propelled by deep neural networks, remarkable strides have been achieved in semantic segmentation technology [8, 10, 80, 31] . However, deep segmentation models encounter a significant decline in adaptability when confronted with open domains. This challenge is mainly attributed to the inherent domain shift between the training and testing data [22, 9, 13] . To address cross-domain challenges, Domain Adaptation (DA) [21] and Domain Generalization (DG) [11] techniques have been proposed to enhance the segmenter's adaptability to the target domain or unseen domains. Pseudo-labeling [67] is a widely used technique in cross-domain semantic segmentation tasks, enhancing model efficiency by training with pixels assigned reliable pseudo-labels. The core of Figure 1 : Comparison of (a) Pseudo-Labels (PL), (b) pixel-level PL [39] , (c) SAM-refined PL [4] , and (d) the proposed connectivity-level PL. The white area in the PL represents the filtered area. Our method effectively filters out and corrects closed-set noise (the orange box) induced by domain shifts, as well as open-set noise (the cyan box) in the wild data (e.g., synthesized from stable diffusion [64] ). pseudo-labeling is how to eliminate semantic noise. In the DA task, various works are dedicated to designing efficient selection or training methods, such as multi-classifier voting [97, 90] or augmentation consistency [2, 26] principles to stabilize noisy training. Furthermore, in the DG task, advanced work [4] has also shown that pseudo-labeling can be used to leverage in-the-wild data synthesized by stable diffusion, enhancing the segmenter's generalization to unseen domains.Despite the significant advancements made by these methods, we have identified limitations in the pseudo-labels as depicted in Fig. 1 . Firstly, under relatively severe domain shifts, most selected reliable pixels appear speckled and remain noisy. (the orange box). Secondly, when dealing with in-the-wild data, pixels belonging to the open-set class may also be selected as 'reliable' and still exhibit speckle (the cyan box). These make it difficult for the selection mechanism to identify and correct these speckled close-and open-set noises. As a result, speckle noise labels with open-set or closed-set noise are introduced into the subsequent self-training process, leading to severe error accumulation. These issues indicate that constructing pixel-level uncertainty measures to filter noisy pseudo-labels is very challenging, especially in open environments.Segment Anything Model (SAM) [37] is a foundation segmentation model that takes both images and geometric prompts (points, boxes, masks) as input and outputs class-agnostic masks. Motivated by this, some attempts [4, 5] have been made to adopt SAM to refine the pseudo-labels. For instance, using reliable pixel pseudo-labels to prompt SAM to generate class-agnostic masks, and then assigning pseudo-classes to these masks. However, due to inappropriate prompting and semantic noise, these attempts may not improve the quality of pseudo-labels and even exacerbate their semantic noise, as shown in Fig. 1(c) .In this paper, we introduce a novel method called Semantic Connectivity-driven Pseudo-labeling (SeCo) for cross-domain semantic segmentation. SeCo models the distribution of pseudo-label noise at the connectivity level, allowing for effective correction of closed-set noise and removal of open-set noise. SeCo comprises two key components: Pixel Semantic Aggregation (PSA) and Semantic Connectivity Correction (SCC). Initially, PSA splits the categories into the "stuff" and "things" forms. Then, PSA efficiently aggregates speckled pseudo-labels into semantic connectivity 2 by interacting with the Segment Anything Model (SAM). This strategy not only ensures precise boundaries but also streamlines noise localization, as distinguishing noise at the connectivity level is inherently more straightforward than at the pixel level. Subsequently, SCC introduces a sample connectivity classification task for learning connectivity with noisy labels. As connectivity classification focuses on local overall categories, we propose to leverage the technique of early learning in noisy label The pipeline of the proposed Semantic Connectivity-Driven Pseudo-labeling (SeCo). In (a), pixel-level pseudo-labels are interactively aggregated into connectivity by SAM using the "stuff and things" manner, grouping semantically similar pixels. Then, in (b), these connectivities are treated as classification objects and are identified for semantic noise by noisy learning. This process is handled offline, and the corrected high-quality pseudo-labels can be used for further self-training. learning [88, 45] to identify connectivity noise, guided by loss distribution. As illustrated in Fig. 1(d) , the incorporation of the proposed connectivity-driven pseudo-labels significantly enhances the quality of pseudo-labels, showcasing complete structures and reduced close-and open-set category noise.In summary, the contributions of this paper are threefold. First, we identify the drawbacks of the pseudo-labeling technique and highlight the significance of semantic connectivity in addressing these challenges. Second, we propose a Semantic Connectivity-driven pseudo-labeling (SeCo) algorithm, which can effectively generate high-quality pseudo-labels, thereby facilitating robust domain adaptation. Third, extensive experiments underscore the versatility of SeCo in effectively addressing various cross-domain semantic segmentation tasks, including domain generalization, traditional, even source-free [38, 28] , and black-box [89] domain adaptation. Notably, SeCo achieves marked enhancements in the more challenging source-free and black-box domain adaptation tasks. 
method: 
 Problem Definition. Cross-domain semantic segmentation aims to transfer a segmentor trained on the labeled source domain D s = {(x i s , y i s )} Is i=1 to the unlabeled target domain D t = {(x i t )} It i=1 , where I s and I t indicate the number of samples for each domain respectively. x and y represent an image and corresponding ground-truth label. Presently, mainstream cross-domain segmentation methods optimize the following objective to enhance model adaptability,EQUATIONwhere L s is the supervised cross-entropy loss, β is the trade-off weight, L t is the unsupervised pseudo-labeling loss, and ŷt is the pseudo-label. This formula underscores the critical importance of the quality of pseudo-labels in improving the model's cross-domain ability. To alleviate the noise in pseudo-labels, various estimation [97, 3] and calibration [73, 7, 42, 79] methods have been introduced for pseudo-label selection. However, as mentioned above, the filtered pseudo-labels still encounter issues of constrained semantics and challenging localization of category noise.The presented Semantic Connectivity-driven pseudo-labeling (SeCo) is illustrated in Fig. 2 . SeCo comprises two components, namely Pixel Semantic Aggregation (PSA) and Semantic Connectivity Correction (SCC), working collaboratively to refine the low-quality and noisy pseudo-labels into high-quality and clean pseudo-labels. Initially, PSA aggregates pixels from the filtered pseudo-labels into connections by interacting with the segment anything model (SAM) [36] through stuff and things interactions. Subsequently, PSA segments the image into multiple connectivities based on their semantics. Guided by the connectivity set, SCC establishes a connectivity classifier, conducts connectivity pooling on image features, and classifies each connectivity. Leveraging information about fitting difficulty and loss distribution, SCC identifies and corrects noise. Finally, the connectivitydriven pseudo-labels, characterized by comprehensive semantics and low noise are achieved.Motivation: Why SAM? Pixel semantic aggregation (PSA) proposes utilizing reliable pixels within pseudo-labels as category references and subsequently aggregating pixels that share similar semantics into connections. Intuitively, the above goals can be achieved through interactive segmentation [66, 44, 62] or pixel clustering [55, 1] , but traditional techniques often struggle to accurately identify semantic boundaries in complex scenes, resulting in ambiguous aggregation. The advent interactive segmentation model, segment anything model (SAM) [36] , provides powerful semantic capture capabilities. With reasonable prompts, SAM has the potential to give accurate semantic boundaries even in complex scenes [47, 33] . Building on SAM's remarkable capability, we are motivated to investigate how to leverage the reliable yet limited pseudo-labels to prompt SAM effectively and enhance the completion of pseudo-label semantics.Discussion on Utilization of SAM. We analyze two naive solutions as outlined below. The first involves sampling the center pixels of each connected region on the pseudo-label as prompt points, as depicted in Fig. 3 (a). We observe that when prompt points of the same category contain noise, this method compromises the aggregated segmentation structure. The disruption is attributed to noisy prompts interfering with the cross-attention mechanism in SAM [36] .The second method represents an improved way, called semantic alignment [5] , aligning pseudolabels with the connectivity established by SAM. This involves selecting the pseudo-label with the maximum proportion in each connectivity as the category for the entire region, as illustrated in Fig. 3 (b). We note that while this approach can refine pseudo-labels, it is consistently influenced by SAM's uncertain semantic granularity, particularly in the context of neighboring instance objects. Fig. 3 (b) provides examples of failures in this method, where SAM aggregates two categories, "traffic sign" and "pole" into a semantic connected region, leading to misaligned pseudo-labels due to this uncertainty. Our analysis indicates that this issue arises because SAM constructs connectivity by uniformly sampling points in space as prompts and subsequently filtering out redundantly connected regions. This fails to ensure corresponding sampling points for neighboring instance objects, resulting in a semantic granularity deviation between SAM's connectivity and specific segmentation tasks.In summary, "prompts" interaction can aid in determining semantic granularity but is vulnerable to noise; Conversely, "alignment" interaction can alleviate noise interference but is susceptible to uncertain semantic granularity. Hence, implementing SAM in cross-domain segmentation is a considerable challenge without a thoughtful design. (For more discussion about SAM, see Sec.3.2)Our Strategy. Building upon the analysis above, we find that noise significantly affects "stuff" categories due to their larger size and higher pixel proportions, making them more prone to selecting noisy pixels. On the other hand, semantic granularity uncertainty is more prevalent in "things" categories, given their smaller size and dense adjacency. To this end, we propose to interact with SAM in the form of "stuff" and "thing". Specifically, for "stuff", we utilize semantic alignment to mitigate the impact of noisy prompts, while for "things", we employ box and point prompts to guide the semantic precision. The detailed algorithm is in Algorithm 1. An illustration of the proposed strategy is shown in Fig. 3(c ). PSA aggregates both precise and noisy pseudo-labels into connectivities, facilitating the locating and correction of noise. This simplicity arises from the fact that distinguishing noise at the connectivity level is much easier than at the pixel level, as it eliminates the necessity to scrutinize local semantics and instead focuses on the overall category (See experimental analysis in Appendix D).Inspired by this, we propose Semantic Connectivity Correction (SCC), introducing a simple connectivity classification task and detecting noise through loss distribution. Specifically, given the input image x i , we first obtain the connectivity mask list M = {m i,n } Ni n=1 and its corresponding connectivity-level pseudo-label ŷsc = {ŷ i,n sc } Ni n=1 from PSA, where N i represents the number of connectivities for the i-th sample x i . Then, we set up a connectivity classifier, comprising a feature extractor F scc and a linear layer MLP, and optimize it with the following objective,EQUATIONPool[•, mask] denotes the average pooling of features corresponding to the input mask, k ∈ [0, 1, ...K], and K is the category number. Optimizing L scc conducts a K-way classification for each connectivity with clean and noisy labels.Based on observations of early learning in noisy learning [88, 45, 82] : when training on noisy labels, deep neural networks, in the early stages of learning, initially match the training data with clean labels, and subsequently memorize examples with erroneous labels. We warm up the connectivity classifier for several epochs and then obtain the loss distribution by calculating Eq. (2) for each connectivity. As shown in Fig. 4 , the loss of connectivities presents bimodol distribution, and the clusters with larger losses correspond to higher noise, which better conforms to the observations. To this end, we employ a two-component Gaussian Mixture Model to effectively model the loss distribution using the Expectation-Maximization algorithm [82] . Subsequently, the probability of connectivity being noisy, denoted as η, can be reasonably approximated by the Gaussian distribution associated with bigger loss, i.e., η i,n = p(c|L scc (x, m i,n )). c is the parameters of the corresponding Gaussian distribution. We keep the clean connectivity by setting a noise threthod τ ns , i.e.,EQUATIONBesides, we find that many noisy connectivities can be corrected by setting another correction threshold τ cr on the output probability of connectivity classifier, i.e.,EQUATIONwhere p n,k scc represents the probability of class k for the n-th connectivity. We take the union of the two sets as the final connectivity-driven pseudo-label set D all = D clean ∪ D corr for self-training.We provide solutions on connectivity-driven pseudo-labels for different domain adaptation tasks.. Following [4] , we first use Stable Diffusion to synthesize simulated unseen domain data. Then, we use the DG model and our SeCo to pseudo-label these synthesized data and retrain the DG model on them.Unsupervised Domain Adaptation. The connectivity-driven pseudo-label set D all serves two primary functions. Firstly, they contribute to the pseudo-labeling L t loss in Eq. ( 1), providing accurate semantic guidance for the target domain. The second objective is to mitigate category bias in domain adaptation. We treat D all as a sample pool, where we resample minority classes in the target domain and duplicate them through copy-paste operation [16] onto both domains.Source-free & Black-box Domain Adaptation. In these scenarios, source access is restricted. This limitation prevents the deployment of the source loss L s in Eq. ( 1), making self-training more vulnerable to noise interference. Connectivity-driven pseudo-label set D all brings a novel idea to mitigate these challenges. With its contribution to accurate semantics and low noise, D all can be viewed as a well-organized labeled set, thereby transforming source-free and black-box domain adaptation tasks into semi-supervised segmentation tasks [84, 83] .Discussion about Fair Comparison. We acknowledge the potential concern regarding unfair comparisons between our SAM-incorporating method and existing approaches. First, it is important to emphasize the considerable challenges in applying SAM to CDSS tasks. Detailed experiments are in Table 5 of Section 3.2. We believe our work makes a significant contribution to exploring the potential of SAM-enhanced CDSS tasks. Second, our method is designed to be integrative, enhancing existing pseudo-labeling methods rather than competing with them (as demonstrated in Section 3.1 and Tables 1 and 2 ). Third, we conducted experiments without using SAM to validate that the proposed semantic connectivity denoising idea still has advantages, as shown in Fig. 5 of Section 3.2. We hope this work can inspire the community to further investigate the effective utilization of SAM in CDSS, embracing the popular trend of facilitating visual tasks with large-scale models, such as enhancing classification [52, 81, 61] with large-language models (e.g., GPT-3).Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Please refer the Sec. G.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experiments: 
 Datasets. We employ two real datasets (Cityscapes [12] and BDD-100k [86] ) alongside two synthetic datasets (GTA5 [63] and SYNTHIA [65] ). The details of these datasets are introduced in Section B.Implementation Details. Traditional UDA: We opted for two network architectures: DeepLabV2 [6] with ResNet-101 [19] and SegFormer [80] with MiT-B5. For DeepLabV2, we chose two classical methods, AdvEnt [70] and ProDA [90] , as the baselines. For SegFormer, we selected two highly successful UDA methods, DAFormer [24] and HRDA [25] , as the baselines. Source-free UDA: We maintain DeepLabV2 as the base network to align with existing works. We chose HCL [28] and the current SOTA method DTST [93] as baselines. Black-box UDA: We use two SOTA blackbox UDA methods, DINE [43] and BiMem [89] , as baselines. Across all tasks, for each baseline method, we select the pseudo-labels of its predicted top 50% confidence-ranked pixels for SeCo processing. Subsequently, we utilize the SAM [36] with Vision Transformer-H (ViT-H) [14] to generate connectivities. We refrain from using SAM to refine pseudo-labels for test data to avoid introducing extra inference overhead. The automatic mask generation process in SAM adheres to the official parameter settings. In Algorithm 1, the enlargement factor for the bounding box area is set to 1.5. The connectivity classifier is trained only for 5000 iterations in an early learning way for all tasks. The noise threshold (τ ns ) and correction threshold (τ cr ) are configured at 0.60 and 0.95, respectively.The performance of SeCo is shown in Tables 1, 2, 3 and 4 SeCo can be integrated with various SOTA domain adaptation and domain generalization methods, and significantly enhances their adaptability. Moreover, SeCo exhibits notable improvements for both source-free and black-box adaptation, overcoming limitations with the source domain data. Domain Generalization. In this experiment, we followed CLOUDS [4] , which uses synthetic data from Stable Diffusion (SD) to assist DG. Our aims are: 1) to show that our method can handle challenging synthesized data with open-set noise, and 2) to compare our SAM usage with the competitive scheme in CLOUDS. Compared to the DG baseline SHADE and HRDA, our method significantly improves their performance by 6.8% and 3.1%, respectively. Compared to CLOUDS, which also uses SAM, we achieve even greater improvements, further enhancing performance by 1.4% and 1.0%. Domain Adaptation. GTA5 → Cityscapes. Results are reported in Table 1 . In the UDA setting, the integration of SeCo with AdvEnt [70] leads to a notable performance improvement, achieving a 13.4% increase in mIoU score. Combining SeCo with ProDA [90] 4 . This task involves complex mixed-weather adaptation. SeCo consistently achieves stable performance improvements. Specifically, SeCo enhances the performance of two baseline methods, PyCDA [46] and ProDA [90] , by 11.4% and 7.8%, respectively, establishing itself as the new state of the art for this benchmark. In the source-free setting, SeCo achieves an improvement of 6.2% over the state-of-the-art method [95] , demonstrating sustained and stable performance gains.Can SAM benefit Cross-Domain Semantic Segmentation in another naive way? In ⃝ Use the backbone of SAM to empower CDSS. Table 5 shows that utilizing SAM as the backbone achieves a notable performance drop in UDA. The reduction mainly came from rare classes (e.g. "train" and "truck"). Both DAFomer and HRDA use Feature Distance to keep the semantic knowledge of ViT-B pre-trained on ImageNet, effectively improving the adaptation for rare classes. However, SAM's pre-training does not consider such semantic knowledge and thus obtains inferior results in rare classes. 2 ⃝ Is the UDA still necessary, if we use CLIP+SAM (CSAM) to get the initial pseudo label? We conduct the following experiment to explore the feasibility of CLIP+SAM: a) Use SAM to segment the input image. b) Extract the largest bounding rectangle from each segment. c) Create text descriptions for categories, e.g., "a photo of a road." d) Use CLIP to match image patches with text descriptions, assigning text labels as categories. CLIP+SAM+UDA combines pseudo-labels from CLIP+SAM and UDA, using voting fusion to select consistent predictions for training. Table 5 shows that CSAM cannot obtain competitive results on the target data, even when combined with UDA methods. The reasons are below. Given spatially uniform sampling as prompt points, 1) SAM is prone to over-segmentation results, which makes it difficult for CLIP to obtain sufficient context from small segments. 2) SAM's segments may conflict with the defined semantics of the target data, e.g., SAM always treats 'poles' and 'traffic sign' as one segment. 3 ⃝ Is using SAM on coarse pseudo-labels enough? Table 5 shows the comparison of using vanilla SAM and our method on the source-trained model's (Coarse) and UDA-adapted model's prediction. It shows that the gains of using SAM on coarse pseudo-labels are minimal and even negative on strong UDA baselines. This is because SAM risks introducing more semantic noise when extending semantic boundaries. Our method, even with coarse pseudo-labels, allows SAM to achieve greater benefits. The above discussions indicate the difficulty of applying SAM in cross-domain segmentation and the non-trivial design of our SAM-based method.Is Our Method Specific to SAM? The proposed Semantic Connectivity Correction (SCC) is general and not specific to SAM. SCC can work on any form of pseudo-labels, although their connectivity structure may not be as good as SAM's. In Fig. 5 , we report the results of directly using SCC on pseudo-labels generated by existing UDA models. Under this exact fair comparison, our method still achieves good improvement and is more competitive than widely used Knowledge Distillation (KD) [90] . Again, we want to emphasize that this work aims to bring a new perspective for enhancing CDSS to embrace the huge benefit of large-scale models, which is a non-trivial contribution. (b) refers to the interaction with SAM using semantic alignment [5] , as shown in Fig. 3 . SF-UDA: source-free UDA. BB-UDA: black-box UDA.Ablation Study. The results of ablation experiments are presented in Table 6 . We conduct analyses across different domain adaptation settings as follows.In UDA, PSA yields a performance improvement of 5.4% for ProDA [90] , surpassing the gains achieved by PSA (b) . Additionally, SCC builds upon PSA, contributing an additional 5.0% enhancement to ProDA. A similar trend is observed in the ablation study on DAFormer [24] . These findings suggest that, at the interaction level with SAM, PSA proves more effective than PSA (b) ; however, interacting solely with SAM is insufficient for achieving substantial self-training performance gains. SCC plays a crucial role in further filtering out noise propagated by SAM, leading to a significant enhancement in UDA performance.In source-free UDA, PSA results in a 3.0% performance improvement for DTST [93] , still outperforming PSA (b) . Due to the substantial initial pseudo-label noise in the source-free setting, PSA aggregates more noisy connections, resulting in a diminished performance gain compared to UDA. SCC, building upon PSA, brings an improvement of 5.4%, reinforcing the notion that SCC can effectively filter and correct propagated pseudo-labels.In balck-box UDA, PSA brings a marginal improvement, with only a gain of 1.2%. SCC on top of PSA achieves a substantial improvement of 7.3%, further confirming the aforementioned conclusions. These results underscore the importance of correcting noise within connections, especially under more significant domain shifts and weaker initial segmentation results. Detailed ablation on prompt way. We conduct ablation studies on "Prompting Only" (PO) and "Semantic Alignment"(SA) across multiple tasks in GTA → Cityscape in Table 7 . We provide two metrics for these detailed ablations: PL mIoU (pseudo-label quality on the training set) and Val. mIoU (model performance on the validation set after training with those pseudo-labels). As shown in the Table 7 , the "prompting only" method reduces the quality of pseudo-labels in the training set, leading to poor adaptation performance. This is because the unreliable interaction method introduces excessive noise into the pseudo-labels generated by SAM. "Semantic alignment" improves the quality of the training set pseudo-labels, but the improvement is limited, resulting in limited adaptation benefits. In contrast, our method enhances the quality of the training set pseudo-labels through better interaction, leading to superior performance gains.Ablation studies on our SCC and Dividemix. Our SCC is partly inspired by DivideMix [41] , however, we focus on mitigating the pixel-level noises in pseudo-labels raised by domain shifts and SAM refinement, while Dividemix focuses on mitigating the image-level label noises. Besides, we would like to emphasize that one of the main contributions of SCC is to provide the idea of denoising at the connectivity level, which makes it possible to apply other image-level denoising methods such as Dividemix to segmentation tasks. To better verify the effectiveness of our SCC, we make two experiments as show in Table 8 : a) directly applying DivideMix to pixel-level denoising. b) using DivideMix to denoise the pixels aggregated by our PSA (using SAM). The results show that pixel-level denoising methods based on DivideMix are inferior to SCC even with SAM, highlighting the advantage of denoising at the connectivity level.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
conclusion: 
 In this work, we propose Semantic Connectivity-driven Pseudo-labeling (SeCo), formulating pseudolabels at the connectivity level for structured and low-noise semantics. SeCo, comprising Pixel Semantic Aggregation (PSA) and Semantic Connectivity Correction (SCC), efficiently aggregates speckled pseudo-labels into semantic connectivity with SAM. SCC introduces a simple connectivity classification task for locating and correcting connected noise. Experiments demonstrate SeCo's flexibility and significant effectiveness in performance across various cross-domain semantic segmentation tasks. We hope that this work could inspire the community to apply SAM to more cross-domain, semi-supervised and few-shot segmentation settings.Algorithm 1 Aggregation of Pseudo-Labels with SAM 1: procedure AGGREGATEPSEUDOLABELS(Image x, Pseudo-Label ŷ, SAM Model) 2:Aggregate Pseudo-Labels for "things" category:3:Extract connectivities of ŷ 4:for each connectivity in ŷ of "things" category do 5:Compute enlarged maximum bounding box as box prompt 6:Compute geometric center as point prompt 7:Interact with SAM using box and point prompts 8:Obtain aggregated connectivity 9:Aggregate Pseudo-Labels for "stuff" category: for each "stuff" category in ŷ do 12:Align ŷ with the no-semantic connectivities by assigning the maximum proportion pseudo-label 13:Obtain aggregated connectivities 14:Merge stuff and thing: Output: Aggregated Pseudo-Label ŷpsa  
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We provide the procedure of aggregation of pseudo-labels in Algorithm 1.Hyper-Parameter Sensitivity. Fig. 6 illustrates the impact of τ ns and τ cr on the final model's adaptability (mIoU), connectivity accuracy, and the kept number of connectivity. We set the range of τ ns from 0.2 to 0.8 to balance excessive connectivity filtering for small values and noise persistence for large values. τ cr is maintained within a confidence threshold range of 0.85 to 0.99 to avoid error correction issues. It can be observed that within a specific range, the influence of τ ns and τ cr on the final model's adaptability is minimal. Regarding τ ns : a larger τ ns retains more connectivity but introduces more noise, leading to a decrease in adaptability. A smaller τ ns maintains higher connectivity accuracy, but a lower quantity reduces richness and results in a decrease in mIoU. Regarding τ cr : a larger τ cr corrects some confident connections, improving accuracy and adaptability. A smaller τ cr introduces more noise, compromising accuracy and adaptability. The final τ ns / τ cr is set to 0.6 / 0.95.Why is it easier to filter noise at the connectivity level than at the pixel level? We find that compared to pixel-level classification, connectivity-level classifiers can more easily construct a compact feature space, thus simplifying noise filtering. As shown in Fig. 7 , we measure the distance from features to class cluster centers (introduced as the "FID" metric in [72] ) for ADVENT, ProDA, DAFormer, and our method after adaptation. We use the FID value for each category in the ADVENT method as the baseline, and normalize the FID values of other methods by dividing them by this baseline. Therefore, the FID values for all categories in ADVENT are set to 1. A smaller FID value indicates a more compact cluster and better feature separability. The results indicate that our connectivity-level classifier significantly enhances feature separability.We explore SeCo's performance in more segmentation scenarios, including indoor scenes, crossdomain medical images, and cross-domain remote sensing images, as shown in Table 9 -Table 11 .Based on these positive experimental results, we believe SeCo has the potential to be integrated into more segmentation scenarios involving the use of unlabeled data.Indoor scenes: The commonly used segmentation dataset for indoor scenes is ADE20K, but no cross-domain segmentation benchmark exists. Thus, we conduct experiments on a semi-supervised segmentation task, which also involves utilizing pseudo-labels from unlabeled data (domain adaptation is seen as semi-supervised learning with domain shift). We perform PSA using the stuff and thing definitions in ADE20K and execute SCC with default parameters. We use the Unimatch model as the baseline and follow its settings. The results of incorporating SeCo are shown in the table below. In multiple labeled data splits (1/64 -1/8) in ADE20K, SeCo shows significant performance improvement compared to directly using SAM.We follow the medical image UDA setup from Sim-T[3], using the Endovis17 and Endovis18 abdominal surgery datasets collected from different devices containing 3 instrument type classes. We treat the segmentation objects as "things" and aggregate pixels using only boxes and points from the pseudo-label. The table below shows how SeCo greatly benefits SAM in this challenging task.Remote sensing: We follow the UDA setup in remote sensing from the CIA [4] , using the Potsdam and Vaihingen datasets collected from different satellites. These datasets contain five common semantic categories: car, tree, impervious surface, building, and low vegetation. We treat cars and buildings as "things," and the rest as "stuff." The table below shows that SeCo still achieves significant performance improvement compared to directly using SAM.Indoor Scenes: ADE20K Fig. 8 displays the pseudo-label outputs of PSA and SCC in the GTA5 → BDD-100K task. In this open compound adaptation task, the model's initial speckled pseudo-labels exhibit considerable noise. It is noticeable that PSA aggregates speckle noise into connected components, concurrently amplifying noisy pseudo-labels. Subsequently, SCC further suppresses and corrects the connected noise from PSA, leading to more structured and lower-noise pseudo-labels. This further validates the motivation behind the design of PSA and SCC.Fig. 9 demonstrates that SeCo has the potential to filter out noisy semantic connectivity when faced with domain shifts in various domain environments. Compared to directly using SAM to complete pseudo-labels, SeCo provides a safer and more efficient approach, to some extent avoiding the problem of error accumulation in self-training.Fig. 10 shows that for unseen domain environments simulated by Stable Diffusion, the pseudo-labels generated by basic domain generalization methods are very noisy and chaotic. In such cases, filtering at the pixel level is more challenging. As seen in (b), pixel-level filtering functions struggle to eliminate both closed-set and open-set noise. As seen in (c), directly using SAM to complete pseudolabels easily leads to the propagation of noise. (d) shows that our SeCo has the potential to filter out some of the noisy labels in such wild data.While our SeCo can integrate existing domain adaptation and domain generalization methods, it lacks an explicitly designed category-balanced connectivity noise filtering method. Additionally, the results demonstrate that SeCo can identify open-set noise in outdoor data. However, due to the current evaluation environment limitations, we cannot directly evaluate its effectiveness. In the future, SeCo is expected to incorporate more visual foundational models to enhance pseudo-labeling for outdoor data targeting unknown distributions.SeCo aims to enhance the adaptability of segmentation to unseen domains or target domains, and has the potential to be applied in open-world scenarios. It has the potential to be combined with Justification: We emphasized in the abstract and introduction that the main scope of our work is to use connected denoising technology to enhance the expressive ability of cross-domain segmentation.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: We make sure to preserve anonymity and with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).We refer to the concept of "connected components" in traditional image processing[30], and call the connected regions here as "semantic connectivity". 


Decision: The reviewers unanimously appreciated that the proposed method is well motivated and technically sound, and that the experiments are extensive. However, they at the same time raised concerns with potentially limited application scenarios (SXcd), unfair comparisons with previous work due to the use of SAM (SXcd), incremental novelty of the semantic connectivity correction module (pA2t), unclear part of the manuscript (pA2t), and lack of discussions on closely related work (pA2t); they also suggested additional in-depth analysis (hfgi). The authors' rebuttal and subsequent responses in the discussion period address these concerns and suggestions, and consequently, all the reviewers were supportive of the paper after the discussion period. 

The AC agrees with the reviewers and recommends acceptance. The authors are strongly encouraged to carefully revise the paper to reflect the valuable comments by the reviewers, to add new results and discussions brought up in the rebuttal and discussions, and to further improve the quality of writing.

Reviews: review: paper_summary: paper_summary: This paper tackles the cross-domain semantic segmentation problem. It incorporates two modules, including the Pixel Semantic Aggregation and the Semantic Connectivity Correctio modules. The former adopts the SAM model to separately refine pseudo-labels for thing and stuff classes. The latter adopts the refined masks to train a connectivity classifier to further rectify pseduo-labels. Detailed experiments are provided under both the domain adaptation and the domain generalization settings. Results demonstrate that the method can effectively enhance the pseudo-label's quality and is compatible with various baseline methods.
strength_weakness: strength_weakness: - The work may be limited to a relatively narrow field. Conducting experiments with the GTA5/Cityscape/SYNTHIA datasets has been the standard for cross-domain segmentation issues for a long while. Whether the proposed method can take effect in broader segmentation tasks concerning indoor scenes, medical images, and more categories is unsure.
- Though effective, the SAM model is pretrained with a huge amount of data associated with precise mask labels. Therefore, comparing previous methods without this well pretrained segmentation model risks fairness concerns.
questions: questions: Please see the weaknesses for details. Overall, this paper does not have obvious drawbacks to me, though its impact may be limited. Therefore, the reviewer recommends a borderline accept.
limitations: limitations: The authors adequately discussed limitations in the appendix.
paper_summary: paper_summary: This paper proposed an effective method to generate reliable high-quality pseudo-labels for cross-domain semantic segmentation. It introduces a novel method, Semantic Connectivity-driven Pseudo-labeling (SeCo), which addresses these issues by formulating pseudo-labels at the connectivity level, thereby improving noise localization and correction. The SeCo method consists of two components: Pixel Semantic Aggregation (PSA), which aggregates speckled pseudo-labels into semantic connectivity, and Semantic Connectivity Correction (SCC), which corrects connectivity noise through a classification task guided by loss distribution. Extensive experiments show that SeCo significantly enhances the performance of existing methods across various semantic segmentation tasks.
strength_weakness: strength_weakness: ### More detailed quantitative results

- It would be great if the authors could provide quantitative analysis when using prompting only or semantic alignment only without separating things and stuff.
- It would be helpful if the authors could provide the effect of the enlargement factor for the bounding box area, which is set to 1.5 by default. 
- It would be better if the authors could provide the effect of the number of training iterations for the connectivity classifier.
questions: questions: Overall, I like the concept of the proposed method and its effectiveness.
My initial recommendation is Weak Accept.
However, since I am unfamiliar with this research field, I will finalize the rating after discussion with other reviewers.
limitations: limitations: The authors have discussed the limitations.
paper_summary: paper_summary: This paper proposes a method to improve the quality of pseudo labels for unlabeled target data. Specifically, it designs different strategies for using SAM to handle 'stuff' and 'things,' which are classified based on the initial pseudo labels. Subsequently, it fits a Gaussian Mixture Model (GMM) on the per-connectivity/component loss distribution to differentiate between noisy and clean components. The proposed method is evaluated on several cross-domain semantic segmentation tasks and demonstrates significant improvements.
strength_weakness: strength_weakness: 1. The novelty of the semantic connectivity correction (SCC) module is limited, as it trivially adapts the technique in DivideMix (ICLR 2020) to handle components instead of images.

2.  It would be beneficial to discuss related work on noisy label learning since the second part (SCC) focuses on noisy label recognition and correction, which are common in noisy label learning.

3. The use of domain generalization (DG) with synthesized data is not a common setting and may not accurately represent DG. Additionally, Eq. 1 is not suitable if including DG settings as x_t is not accessible.

4. The implementation details of CLIP+SAM in Line 248 are unclear.

5.  In Table 6, it would be helpful to ablate the recognition and correction steps in SCC. What would the performance be without correcting the noise (Eq. 4)?
questions: questions: Please refer to the weaknesses section.
limitations: limitations: The authors have discussed the limitations and potential impact of the proposed method.

Comments: title: Reply to Reviewer pA2t - A more detailed comparison of SCC and Dividemix
Thank you for your response
comment: Thanks for the response and efforts to provide additional information. It addressed most of my concerns. I'll maintain my positive score.
Thank you for your feedback, it is very helpful for us to improve our paper.

First, we would like to emphasize that one of the main contributions of SCC is to provide the idea of ​​denoising at the connectivity level, which makes it possible to apply other image-level denoising methods such as Dividemix to segmentation tasks.
Building on this, we further provide the implementation of DivideMix at the connectivity level across multiple tasks, including GTA → Cityscapes (G→C UDA), SYNTHIA → Cityscapes (S→C UDA), GTA5 → BDD-100k (G→B OC-DA), Endovis17→Endovis18 (E17→E18) and Potsdom → Vaihingen (P→V).

**Implementation comparison**: In DivideMix, two models are first trained with different warm-up iterations. Then, each model uses different GMMs to select samples and then exchanges the selected samples (co-divide) between the two models for further co-guessing-based training with more iterations. The implementation requires careful selection of several hyperparameters, including the warm-up iterations for both models, the number of further training iterations, and the threshold for co-guessing. In contrast, SCC requires only a single model with fixed 5k early-stopping training for all tasks and then uses GMM for sample selection. 

**Performance comparison**:  The performance and training time of DivideMix at the connectivity level are shown in the table below. We observe that DivideMix performs well in tasks with lower noise rates, such as GTA → Cityscapes (UDA), where it achieves comparable performance to SCC.  However, in tasks with higher initial noise rates, such as GTA5 → BDD-100k (OC-DA), SCC outperforms DivideMix. This is because higher noise rates introduce more challenges in hyperparameter tuning, particularly in choosing the warm-up epochs for the two models, e.g., close warm-up iterations fail to create sufficiently different models, while significantly different iterations can lead to one model overfitting to noise. (The table below shows results where we chose models with 2.5K and 5K iterations for both models.) This complexity hinders DivideMix from maintaining stable denoising performance in more challenging scenarios. 

**Overall, compared to DivideMix, we think our SCC not only introduces a new idea of denoising at the connectivity level,  but offers a simpler implementation, requires less hyperparameter tuning, has shorter denoising training times, and delivers more stable denoising performance**.

| | Urban Scene            | | || |  | Medical              | | Remote Sensing      | |
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
|                            | G→ C |                    | S→ C|                    | G→ B |                    |  E17→E18 |                    | P→ V| |
| Methods                    | Val mIoU               | Training Time (/h) | Val mIoU                   | Training Time (/h) | Val mIoU                | Training Time (/h) | Val mIoU             | Training Time (/h) | Val mIoU            | Training Time (/h)  |
| PSA+**DivideMix-Pixel**        | 60.1                   | 10                 | 53.4                       | 10                 | 44.2                    | 12                 | 47.9                 | 8                  | 50.1                | 8                   |
| PSA+**DivideMix-Connectivity** | 63.9                   | 6                  | 57.3                       | 6                  | 47.7                    | 7                  | 58.7                 | 6                  | 65.3                | 6                   |
| PSA+**SCC**                       | **64.1 (+0.2)**            | 2.5                  | **58.6 (+1.3)**                | 2.5                  | **49.5 (+1.8)**             | 2.5                  | **60.4 (+1.7)**                 | 1.5                | **66.1 (+0.8)**               | 1.5                 |
Thank you to the authors for providing the rebuttal, including the additional ablation study and explanations. Most of my concerns have been addressed. Regarding the novelty of SCC, I agree with the authors that the tasks are different. However, the fundamental challenge of noisy label correction remains the same, and the technical novelty of SCC appears to be quite subtle. Additionally, when comparing with DivideMix, I believe it would be more convincing to apply DivideMix at the connectivity level. I would like to maintain my initial rating.