Transferring to Real-World Layouts: A Depth-aware Framework for Scene Adaptation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-08 16:17:39

Abstract: Scene segmentation via unsupervised domain adaptation (UDA) enables the transfer of knowledge acquired from source synthetic data to real-world target data, which largely reduces the need for manual pixel-level annotations in the target domain. To facilitate domain-invariant feature learning, existing methods typically mix data from both the source domain and target domain by simply copying and pasting the pixels. Such vanilla methods are usually sub-optimal since they do not take into account how well the mixed layouts correspond to real-world scenarios. Real-world scenarios are with an inherent layout. We observe that semantic categories, such as sidewalks, buildings, and sky, display relatively consistent depth distributions, and could be clearly distinguished in a depth map. Based on such observation, we propose a depth-aware framework to explicitly leverage depth estimation to mix the categories and facilitate the two complementary tasks, i.e., segmentation and depth learning in an end-to-end manner. In particular, the framework contains a Depth-guided Contextual Filter (DCF) forndata augmentation and a cross-task encoder for contextual learning. DCF simulates the real-world layouts, while the cross-task encoder further adaptively fuses the complementing features between two tasks. Besides, it is worth noting that several public datasets do not provide depth annotation. Therefore, we leverage the off-the-shelf depth estimation network to generate the pseudo depth. Extensive experiments show that our proposed methods, even with pseudo depth, achieve competitive performance on two widely-used bench-marks, i.e. 77.7 mIoU on GTA→Cityscapes and 69.3 mIoU on Synthia→Cityscapes.

Main paper: 
 introduction: 
 Semantic segmentation refers to the task of assigning pixel-level category labels in an image, which has achieved significant progress in the last few years [2, 6, 35, 63] . It is worth noting that prevailing Figure 1: (a) Considering the driving scenario, we observe that the object location is relatively stable according to the distance from the camera. Therefore, we propose a Depth-guided Contextual Filter (DCF) which is aware of the semantic categories distribution in terms of Near, Middle, and Far view to facilitate cross-domain mixing. (b) Since we explicitly take the semantic layout into consideration, our method achieves more realistic mixed samples compared to the competitive MIC (Vanilla Mixed Sample) [20] . As shown in the red dotted box, "new" buildings are pasted before the parked cars. models usually require large-scale training datasets with high-quality annotations, such as ADE20K [78] , to achieve good performance and but such pixel-level annotations in real-world are usually unaffordable and time-consuming [11] . One straightforward idea is to train networks with synthetic data so that the pixel-level annotations are easier to obtain [43, 44] . However, the network trained with synthetic data usually results in poor scalability when being deployed to a real-world environment due to multiple factors, such as weather, illumination, and road design. Therefore, researchers resort to unsupervised domain adaptation (UDA) to further tackle the variance between domains. One branch of UDA methods attempts to mitigate the domain shift by aligning the domain distributions [17, 37, 42, 51, 61] . Another potential paradigm to heal the domain shift is self-training [30, 67, 76, 81, 82] , which recursively refine the target pseudo-labels. Taking one step further, recent DACS [50] and follow-up works [8, 18-20, 23, 57, 62] combine self-training and ClassMix [40] to mix images from both source and target domain. In this way, these works could craft highly perturbed samples to assist training by facilitating learning shared knowledge between two domains. Specifically, cross-domain mixing aims to copy the corresponding regions of certain categories from a source domain image and paste them onto an unlabelled target domain image. We note that such a vanilla strategy leads to pasting a large amount of objects to the unrealistic depth position. It is because that every category has its own position distribution. For instance, the background classes such as "sky" and "vegetation" usually appear farther away, while the classes that occupy a small number of pixels such as "traffic signs" and "pole", usually appear closer as shown in Figure 1 (a) . Such crafted training data compromise contextual learning, leading to sub-optimal location prediction performance, especially for small objects.To address these limitations, we observe the real-world depth distribution and find that semantic categories are easily separated (disentangled) in the depth map since they follow a similar distribution under certain scenarios, e.g., urban. Therefore, we propose a new depth-aware framework, which contains Depth Contextual Filter (DCF) and a cross-task encoder. In particular, DCF removes unrealistic classes mixed with the real-world target training samples based on the depth information. On the other hand, multi-modal data could improve the performance of deep representations and the effective use of the deep multi-task features to facilitate the final predictions is crucial. The proposed cross-task encoder contains two specific heads to generate intermediate features for each task and an Adaptive Feature Optimization module (AFO). AFO encourages the network to optimize the fused multi-task features in an endto-end manner. Specifically, the proposed AFO adopts a series of transformer blocks to capture the information that is crucial to distinguish different categories and assigns high weights to discriminative features and vice versa.The main contributions are as follows:(1) We propose a simple Depth-Guided Contextual Filter (DCF) to explicitly leverage the key semantic categories distribution hidden in the depth map, enhancing the realism of cross-domain information mixing and refining the cross-domain layout mixing. (2) We propose an Adaptive Feature Optimization module (AFO) that enables the cross-task encoder to exploit the discriminative depth information and embed it with the visual feature which jointly facilitates semantic segmentation and pseudo depth estimation. (3) Albeit simple, the effectiveness of our proposed methods has been verified by extensive ablation studies. Despite the pseudo depth, our method still achieves competitive accuracy on two commonly used scene adaptation benchmarks, namely 77.7 mIoU on GTA→Cityscapes and 69.3 mIoU on Synthia→Cityscapes. 
related work 2.1 unsupervised domain adaptation: 
 Unsupervised domain adaptation (UDA) aims to train a model on a label-rich source domain and adapt the model to a label-scarce target domain. Some methods propose learning the domain-invariant knowledge by aligning the source and target distribution at different levels. For instance, AdaptSegNet [51] , ADVENT [54] , and CLAN [37] adversarially align the distributions in the feature space. CyCADA [17] diminishes the domain shift at both pixel-level and feature-level representation. DALN [4] proposes a discriminator-free adversarial learning network and leverages the predicted discriminative information for feature alignment. Both Wu et al. [61] and Yue et al. [68] learn domain-invariant features by transferring the input images into different styles, such as rainy and foggy, while Zhao et al. [74] and Zhang et al. [71] diversify the feature distribution via normalization and adding noise respectively. Another line of work refines pseudo-labels gradually under the iterative self-training framework, yielding competitive results. Following the motivation of generating highly reliable pseudo labels for further model optimization, CBST [81] adopts class-specific thresholds on top of self-training to improve the generated labels. Feng et al. [12] acquire pseudo labels with high precision by leveraging the group information. PyCDA [32] constructs pseudo-labels in various scales to further improve the training. Zheng et al. [75] introduce memory regularization to generate consistent pseudo labels. Other works propose either confidence regularization [76, 82] or category-aware rectification [69, 70] to improve the quality of pseudo labels. DACS [50] proposes a domain-mixed self-training pipeline to mix cross-domain images during training, avoiding training instabilities. Kim et al. [25] , Li et al. [31] and Wang et al. [56] combine adversarial and selftraining for further improvement. Chen et al. [5] establish a deliberated domain bridging (DDB) that aligns and interacts with the source and target domain in the intermediate space. SePiCo [62] and PiPa [8] adopt contrastive learning to align the domains. Liu et al. [34] addresses the label shift problem by adopting class-level feature alignment for conditional distribution alignment. Researchers also attempted to perform entropy minimization [7, 54] , and image translation [15, 65] . consistency regularization [1, 10, 39, 79] . Recent multi-target domain adaptation (MTDA) methods enable a single model to adapt a labeled source domain to multiple unlabeled target domains [13, 28, 47] . However, the above methods usually ignore the rich multi-modality information, which can be easily obtained from the depth sensor and other sensors.Semantic segmentation and geometric information are shown to be highly correlated [24, 49, 53, 58, 64, 72, 73] . Recently depth estimation has been increasingly used to improve the learning of semantics within the context of multi-task learning, but the depth information should be exploited more precisely to help the domain adaptation. SPIGAN [27] pioneered the use of geometric information as an additional supervision by regularizing the generator with an auxiliary depth regression task. DADA [55] introduces an adversarial training framework based on the fusion of semantic and depth predictions to facilitate the adaptation. GIO-Ada [9] leverages the geometric information on both the input level and output level to reduce domain shift. CTRL [45] encodes task dependencies between the semantic and depth predictions to capture the cross-task relationships. CorDA [57] bridges the domain gap by utilizing selfsupervised depth estimation on both domains. Wu et al. [60] propose to further support semantic segmentation by depth distribution density. Our work follows a similar spirit to leverage depth knowledge as auxiliary supervision. It is worth noting that our work is primarily different from existing works in the following two aspects: (1) from the data perspective, we explicitly delineate the depth distribution to refine data augmentation and construct realistic training samples to enhance contextual learning. (2) from the network perspective, our proposed multi-task learning network not only adopts auxiliary supervision for learning more robust deep representations but also facilitates the multi-task feature fusion by iterative deploying of transformer blocks to jointly learn the rich multi-task information for improving the final predictions. 
method 3.1 problem formulation: 
 In a typical Unsupervised Domain Adaptation (UDA) scenario, we have a source domain, denoted 𝑆, which consists of abundant labeled synthetic data. On the other hand, the target domain, represented by 𝑇 , contains unlabeled real-world data. For example, we have labeled training samples x 𝑆 , y 𝑆 , z 𝑆 ∼ X 𝑆 , Y 𝑆 , Z 𝑆 in the source domain, where x 𝑆 , y 𝑆 are the training image and the corresponding ground truth for semantic segmentation. z 𝑆 is the label for the depth estimation task. Similarly, we have unlabeled target images sampled from target domain data x 𝑇 , z 𝑇 ∼ X 𝑇 , Z 𝑇 , where x 𝑇 is the unlabeled sample in the target domain and z 𝑇 is the label for the depth estimation task. Since depth annotation is not supported by common public datasets, we adopt pseudo depth that can be easily generated by the off-the-shelf model [14] .In UDA, recent works Recent UDA works [8, 18-20, 40, 57] often employ pixel mixing to create cross-domain augmented samples. The basic idea is straightforward: take a portion of pixels from a source domain image and transplant them onto an equivalent area in a target domain image. However, this simple approach faces challenges due to the inherent differences in structure and layout between source and target domain data. Pre-calculate the density value p for each class 𝑖 at each depth interval from the target depth map z 𝑇 , 5:ŷ𝑀 ← M ⊙ y 𝑆 + (1 -M) ⊙ ŷ𝑇 , Randomly select 50% categories and copy the category ground truth label from the source image to target pseudo label Re-calculate the density value p after the mixing, 7:x 𝑀 ← M ⊙ x 𝑆 + (1 -M) ⊙ x 𝑇 ,Calculate the depth density distribution difference before and after mixing, 8:Filter the category once the difference exceeds the threshold, 9:Re-generate the depth-aware binary mask M 𝐷𝐶𝐹 , 10:ŷ𝐹 ← M D C F ⊙ y 𝑆 + 1 -M D C F ⊙ ŷ𝑇 , Generate the filtered training samples with new DCF mask x 𝐹 ← M D C F ⊙ x 𝑆 + 1 -M D C F ⊙ x 𝑇 , 11: Compute predictions ȳ𝑆 ← 𝑎𝑟𝑔𝑚𝑎𝑥 F 𝜃 x 𝑆 , ȳ𝐹 ← 𝑎𝑟𝑔𝑚𝑎𝑥 F 𝜃 x 𝐹 , 12:Compute loss for the batch: ℓ ← L ȳ𝑆 , y 𝑆 , ȳ𝐹 , ŷ𝐹 .Compute ∇ 𝜃 ℓ by backpropagation.Perform stochastic gradient descent. 15: end for 16: return F 𝜃 intervals (Δ𝑧 1 , ..., Δ𝑧 𝑛 ). For a given real-world target input image x 𝑇 combined with the pseudo label ŷ𝑇 and target depth map z 𝑇 , the density value at each depth interval (Δ𝑧 1 , ..., Δ𝑧 𝑛 ) for each class 𝑖 ∈ (1, . . . , 𝐶) can be counted and normalized as a probability. We denote the density value for class 𝑖 at the depth interval Δ𝑧 1 as 𝑝 𝑖 (Δ𝑧 1 ). All the density values make up the depth distribution in the target domain image. Then we randomly select half of the categories on the source images to paste on the target domain image. In practice, we apply a binary mask M to denote the corresponding pixels. Then naive cross-domain mixed image x 𝑀𝑖𝑥 and the mixed label ŷ𝑀𝑖𝑥 can be formulated as:EQUATIONEQUATIONwhere ⊙ denotes the element-wise multiplication of between the mask and the image. The naively mixed images are visualized in Figure 2 . It could be observed that due to the depth distribution difference between two domains, pixels of "Building" category are mixed from the source domain to the target domain, creating unrealistic images. Training with such training samples will compromise contextual learning. Therefore, we propose to filter the pixels that do not match the depth density distribution in the mixed image.After the naive mixing, we re-calculate the density value for each class at each depth interval. For example, the new density value for class 𝑖 at the depth interval Δ𝑧 1 is denoted as p𝑖 (Δ𝑧 1 ). Then we calculate the depth density distribution difference for each pasted category and denote the difference for class 𝑖 at the depth interval Δ𝑧 1 as Δ𝑝 𝑖 (Δ𝑧 1 ) = |𝑝 𝑖 (Δ𝑧 1 ) -p𝑖 (Δ𝑧 1 )|. Once Δ𝑝 𝑖 (Δ𝑧 1 ) exceeds the threshold of that category 𝑖, these pasted pixels are removed. After performing DCF, we confirm the final realistic pixels to be mixed and construct a depth-aware binary mask M D C F , which is changed dynamically based on the depth layout of the current target image.The filtered mixing samples are then generated. In practice, we directly apply the updated depth-aware mask to replace the original mask. Therefore, the new mixed sample and the label are as follows:EQUATIONEQUATIONBecause large objects such as "sky" and "terrain" usually aggregate and occupy a large amount of pixels and small objects only occupy a small amount of pixels in a certain depth range, we set different filtering thresholds for each category. DCF uses pseudo semantic labels for the target domain as there is no ground truth available. Since the label prediction is not stable in the early stage, we apply a warmup strategy to perform DCF after 10,000 iterations. Examples of the input images, naively mixed samples and filtered samples are presented in Figure 2 . The sample after the process of the DCF module has the pixels from the source domain that match the depth distribution of the target domain, helping the network to better deal with the domain gap.In order to exploit the relation between segmentation and depth learning, we introduce a multi-task scene adaptation framework including a high resolution semantic encoder, and a cross-task shared encoder with a feature optimization module, which is depicted in Figure 3 .The proposed framework incorporates and optimizes the fusion of depth information for improving the final semantic predictions.High Resolution Semantic Prediction. Most supervised methods use high resolution images for training, but common scene adaptation methods usually use random crops of the image that is half of the full resolution. To reduce the domain gap between scene adaptation and supervised learning while maintaining the GPU memory consumption, we adopt a high-resolution encoder to encode HR image crops into deep HR features. Then a semantic decoder is used to generate the HR semantic predictions ȳℎ𝑟 . We adopt the cross entropy loss for semantic segmentation:EQUATIONEQUATIONwhere ȳ𝑆 ℎ𝑟 and ȳ𝑇 ℎ𝑟 are high resolution semantic predictions. y 𝑆 is the one-hot semantic label for the source domain and ŷ𝐹 is the one-hot pseudo label for the depth-aware fused domain.Adaptive Feature Optimization. In addition to the high resolution encoder, We use another cross-task encoder to encode input images which are shared for both tasks. Depth maps are rich in spatial depth information, but a naive concatenation of depth information directly to visual information causes some interference, e.g. categories at similar depth positions are already well distinguished by visual information, and attention mechanisms can help the network to select the crucial part of the multitask information. In the proposed multi-task learning framework, the visual semantic feature and depth feature is generated by a visual head and a depth head, respectively. As shown in Figure 3 , after applying batch normalization, an Adaptive Feature Optimization module then concatenates the normalized input visual feature and the input depth feature to create a fused multi-task feature by concatenation as 𝑓 𝑖𝑛 𝑓 𝑢𝑠𝑒 = CONCAT 𝑓 𝑖𝑛 𝑣𝑖𝑠 , 𝑓 𝑖𝑛 𝑑𝑒𝑝𝑡ℎ . The fused feature is then fed into a series of transformer blocks to capture the key information between the two tasks. The attention mechanism adaptively adjusts the extent to which depth features are embedded in visual features:EQUATIONwhere W 𝑇 𝑟𝑎𝑛𝑠 is the transformer parameter. The learned output of the transformer blocks is a weight map 𝛾 which is multiplied back to the input visual feature and depth feature resulting in an optimized feature as: where W 𝐶𝑜𝑛𝑣 denotes the convolution parameter, ⊗ denotes the convolution operation and 𝝈 represents the sigmoid function. The weight matrix 𝜸 performs adaptive optimization of the muti-task features. Then, the fused feature 𝑓 𝑜𝑢𝑡 𝑓 𝑢𝑠𝑒 is fed into different decoders for predicting different final tasks, i.e., the visual and the depth task. The output features are essentially multimodal features containing crucial depth information:EQUATIONEQUATIONwhere ⊙ represents element-wise multiplication. The optimized visual and depth feature is then fed into the multimodal communication module for further processing. The multimodal communication module refines the learning of key information between two tasks by iterative use of transformer blocks. the inference is merely based on the visual input when the feature optimization is fished. The final semantic prediction ȳ𝑆 𝑣𝑖𝑠 and depth prediction z𝑆 can be generated from the final visual feature 𝑓 𝑓 𝑖𝑛𝑎𝑙 𝑣𝑖𝑠 and depth feature 𝑓 𝑓 𝑖𝑛𝑎𝑙 𝑑𝑒𝑝𝑡ℎ by the visual head and depth head . Similar to the high resolution predictions, we use the cross entropy loss for the semantic loss calculation:EQUATIONEQUATIONWe also employ the berHu loss for depth regression at source domain:EQUATIONwhere z and 𝑧 are predicted and ground truth semantic maps. Following [45, 55] , we deploy the reversed Huber criterion [26] , which is defined as :EQUATIONwhere 𝐻 is a positive threshold and we set it to 0.2 of the maximum depth residual. Finally, the overall loss function is:EQUATIONwhere hyperparameter 𝜆 𝑑𝑒𝑝𝑡ℎ is the loss weight. Considering that our main task is semantic segmentation and the depth estimation is the auxiliary task, we empirically 𝜆 𝑑𝑒𝑝𝑡ℎ = 0.1 × 10 -2 . We also designed the ablation studies to change the weight of depth task 𝜆 𝑑𝑒𝑝𝑡ℎ to the level of 10 -1 or 10 -3 . 
experiment 4.1 implementation details: 
 Datasets. We evaluate the proposed framework on two scene adaptation settings, i.e., GTA → Cityscapes and SYNTHIA → Cityscapes, following common protocols [1, 18-20, 50, 57] . Particularly, the GTA5 dataset [43] is the synthetic dataset collected from a video game, which contains 24,966 images annotated by 19 classes. Following [57] , we adopt depth information generated by Monodepth2 [14] model which is trained merely on GTA image sequences. SYN-THIA [44] is a synthetic urban scene dataset with 9,400 training images and 16 classes. Simulated depth information provided by SYNTHIA is adopted. GTA and SYNTHIA serve as source domain datasets. The target domain dataset is Cityscapes, which is collected from real-world street-view images. Cityscapes contains 2,975 unlabeled training images and 500 validation images. The resolution of Cityscapes is 2048 × 1024 and the common protocol downscales the size to 1024 × 512 to save memory. Following [57] , the stereo depth estimation from [46] is used. We leverage the Intersection Over Union (IoU) for per-class performance and the mean Intersection over Union (mIoU) over all classes to report the result. The code is based on Pytorch [41] . We will make our code open-source for reproducing all results. Experimental Setup. We adopt DAFormer [18] network with MiT-B5 backbone [63] for the high resolution encoder and DeepLabV2 network with ResNet-101 backbone for the cross-task encoder to reduce the memory consumption. All backbones are initialized with ImageNet pretraining. Our training procedure is based on selftraining methods with cross-domain mixing [18] [19] [20] 50] and enhanced by our proposed Depth-guided Contextual Filter. Following [19, 50] , the input image resolution is half of the full resolution for the cross-task encoder and full resolution for high resolution encoder. We utilize the same data augmentation, e.g., color jitter and Gaussian blur and empirically set pseudo labels threshold 0.968 following [50] .We train the network with batch size 2 for 40k iterations on a Tesla V100 GPU. Data Resolution. Our proposed depth-aware multi-task framework contains a high resolution encoder and a cross-task encoder with an adaptive feature optimization module (AFO). Previous works [31, 50, 52] downsample Cityscapes to 1024 × and GTA to 1280 × 720. Following [19] , for the high resolution encoder, we resize GTA to 2560 × 1440 and SYNTHIA to 2560 x 1520. Then the crop size is 1024 × 1024. In addition, SegFormer [63] MLP decoder with an embedding dimension of 256 is used for the high resolution branch.For the cross-task encoder branch, we follow common UDA methods [18, 50] to adopt 1024 × 512 pixels (half of the full resolution) for Cityscapes, 1280 × 760 for SYNTHIA and 1280 × 720 for GTA.In addition, a 512 × 512 random crop is extracted.Results on GTA→Cityscapes. We show our results on GTA → Cityscapes in Table 1 and highlight the best results in bold. It could be observed that our method yields significant performance improvement over the state-of-the-art method MIC [20] from 75.9 mIoU to 77.7 mIoU. Usually, classes that occupy a small number of pixels are difficult to adapt and have a comparably low IoU performance. However, our method demonstrates competitive IoU improvement on most categories especially on small objects such as +5.7 on "Rider", +5.4 on "Fence", +5.2 on "Wall", +4.4 on "Traffic Sign" and +3.4 on "Pole". The result shows the effectiveness of the proposed contextual filter and cross-task learning framework in the contextual learning. Our method also increases the mIoU performance of classes that aggregate and occupy a large amount of pixels in an image by a smaller margin such as +1.8 on "Pedestrain" and +1.1 on "Bike", probably because the rich texture and color information contained in the visual feature already has the ability to recognize these relatively easier classes. The above observations are also qualitatively reflected in Figure 4 , where we visualize the segmentation results of the proposed method and the comparison with previous strong transformer-based methods HRDA [19] , and MIC [20] . The qualitative results highlighted by white dash boxes show that the proposed method largely improved the prediction quality of challenging small object "Traffic Sign" and large category "Terrain". Results on Synthia→Cityscapes. We show our results on SYN-THIA → Cityscapes in Table 1 and the results show the consistent performance improvement of our method, increasing from 67.3 to 69.3 (+2.0 mIoU) compared to the state-of-the-art method MIC [20] . Especially, our method significantly increases the IoU performance of the challenging class "SideWalk" from 50.5 to 63.1 (+12.6 mIoU). It is also noticeable that our method remains competitive in segmenting most individual classes and yields a significant increase of +6.8 on "Road", +6.6 on "Bus", +3.9 on "Pole", +3.7 on "Road", +3.2 on "Wall" and +2.9 on "Truck".Ablation Study on Different Scene Adaptation Frameworks. We combine our method with different scene adaptation architectures on GTA→Cityscapes. Table 4 shows that our method achieves consistent and significant improvements across different methods with different network architectures. Firstly, our method improves the state-of-the-art performance by +1.8 mIoU. Then we evaluate the proposed method on two strong methods based on transformer backbone, yielding +3.2 mIoU and +2.3 mIoU performance increase on DAFormer [18] and HRDA [19] , respectively. Secondly, we evaluate our method on DeepLabV2 [6] architecture with ResNet-101 [16] backbone. We show that we improve the performance of the CNNbased cross-domain mixing method, i.e., DACS by +4.1 mIoU. The ablation study verifies the effectiveness of our method in leveraging depth information to enhance cross-domain mixing not only on Transformer-based networks but also on CNN-based architecture.Ablation Study on Different Components of the Proposed Method. In order to verify the effectiveness of our proposed components, we train four different models from M1 to M4 and show the result in Table 3 . "ST Base" means the self training baseline with semantic segmentation branch and depth regression branch. "Naive Mix" denotes the cross-domain mixing strategy. "DCF" represents the proposed depth-aware mixing (Depth-guided Contextual Filter). "AFO" denotes the proposed Adaptive Feature Optimization module and we used two different method to perform AFO. Firstly, we leverage channel attention (CA) that could select useful information along the channel dimension to perform the feature optimization. In this method, the fused feature is adaptively optimized by SENet [21] , the output is a weighted vector which is multiplied back to the visual and depth feature. We leavrage "AFO (CA)" to denote this method. Secondly, we leverage the iterative use of transformer block to adaptively optimize the multi-task feature. In this case, the output of the transformer block is a weighted map. The Multimodal Communication (MMC) module is then used to incorporate rich knowledge from the depth prediction. We denote this method as "AFO (Trans + MMC)". M1 is the self training baseline with depth regression based on DAFormer architecture. M2 adds the cross-domain mixing strategy for improvement and shows a competitive result of 76.0 mIoU. M3 is the model with the Depth-guided Contextual Filter, increasing the performance from 76.0 to 77.1 mIoU (+1.1 mIoU), which demonstrates the effectiveness of transferring the mixed training images to real-world layout with the help of the depth information. M4 adds the multi-task framework that leverages Channel Attention (CA) mechanism to fuse the discriminative depth feature into the visual feature. The segmentation result is increased by a small margin (+0.2 mIoU), which means CA could help the network to [20] 75.9 77.7 +1.8adaptively learn to focus or to ignore information from the auxiliary task to some extent. M5 is our proposed depth-aware multi-task model with both Depth-guided Contextual Filter and Adaptive Feature Optimization (AFO) module. Compared to M3, M5 has a mIoU increase of +0.6 from 77.1 to 77.7, which shows the effectiveness of multi-modal feature optimization using transformers to facilitate contextual learning.Ablation study on GTA+SYNTHIA → Cityscapes. We evaluate the proposed method on multi-source domains setting and report the quantitative result on GTA+SYNTHIA → Cityscapes. With multisource domain data, the model can be trained more robust to the unlabelled target environment. We adopt DACS [50] as our baseline with 52.1 mIoU (Only GTA) performance shown in Table 5 . With more source-domain data, the model yields a better result of 54.2 mIoU. Then, we can observe that our method yields a larger improvement from 54.2 to 56.7 mIoU, demonstrating that the proposed model could adapt multi-domain depth to the target domain and hence increase performance. 
conclusion: 
 In this work, we introduce a new depth-aware scene adaptation framework that effectively leverages the guidance of depth to enhance data augmentation and contextual learning. The proposed framework not only explicitly refines the cross-domain mixing by stimulating realworld layouts with the guidance of depth distributions of objects, but also introduced a cross-task encoder that adaptively optimizes the multi-task feature and focused on the discriminative depth feature to help contextual learning. By integrating our depth-aware framework into existing self-training methods based on either transformer or CNN, we achieve state-of-the-art performance on two widely used benchmarks and a significant improvement on small-scale categories.Extensive experimental results verify our motivation to transfer the training images to real-world layouts and demonstrate the effectiveness of our multi-task framework in improving scene adaptation performance. 
 
 Appendix:  


Decision: The paper studies the problem of incorporating depth for unsupervised semantic scene adaptation. The advantages include: 1. The paper presents a reasonable and practical depth-aware framework for domain adaptation in real-world scenarios. 2. The well-organized and comprehensive analysis is easy to follow and understand. 3. A variety of datasets and convincing results, along with ablation experiments, support the proposed method. 4. The innovative approach uses depth estimation to address layout mismatch issues in traditional unsupervised domain adaptation methods. 5. A cross-task encoder enables contextual learning and feature fusion, improving scene segmentation performance through collaborative learning across tasks. The disadvantages include: 1. The marginal improvement in the joint use of the DCF and AFO modules. 2. Results on real to real adaptation tasks. 

Overall, the paper has a high quality. All reviewers agree to accept this work. The ACs do not find significant reasons to overturn reviewers' recommendations and thus suggest accepting this paper.

Reviews: review: paper_summary: paper_summary: This paper focuses on unsupervised domain adaptation for scene segmentation. In particular, it highlights that layouts are important in real-world scenarios, and proposes a depth-aware framework to explicitly leverage depth estimation to mix the categories. The proposed DCF simulates real-world layouts and further adaptively fuses the complementing features via the cross-task encoder. Massive experimental results demonstrate its effectiveness.
strength_weakness: strength_weakness: 1. The motivation and proposed method is reasonable. Introducing depth into domain adaptation is practical in real-world scenarios.
2. The paper is well-organized and easy to follow.
limitations: limitations: 1. Since you use depth maps as supervision (which is easy to obtain on synthetic), I wonder whether the method can be generalized on real-to-real datasets. Can you try on Cityscape -> ACDC?
2. The proposed method introduces depth as supervision, can you also try to apply such designs on MIC?
3. The provided qualitative visualization shows some improvement in terrain, missing largely improved categories such as pole and wall show (refer to Table 1). Can you add more qualitative results for the significantly improved categories?
paper_summary: paper_summary: This paper proposes a novel framework to improve scene segmentation via unsupervised domain adaptation (UDA), which transfers knowledge from synthetic data to real-world data. The framework utilizes depth estimation to refine the mixing of data across domains, aiming to create more realistic training samples. A key component is the Depth-guided Contextual Filter (DCF), which adjusts the mixing based on depth, improving the relevance to real-world layouts. Even using pseudo depth data, The method achieves competitive results on standard benchmarks, demonstrating its effectiveness in reducing manual annotation needs and facilitating better domain adaptation.
strength_weakness: strength_weakness: The analysis is comprehensive and well-developed.
The paper is well-written and easy to understand.
The experimental part evaluates a rich variety of datasets, and the results are convincing. The ablation experiments also demonstrate the role of key modules.
limitations: limitations: 1. Limited improvement: The joint use of the DCF and AFO modules proposed in this paper only shows a 1.7% improvement over the Naive Mix, but significantly increases the computation resources and time required in the sample preprocessing stage.
2. From the ablation study in Table 3, the main improvement comes from the DCF module, while the elaborately designed multimodal communication AFO module has a very limited effect. The introduction of the transformer block in the AFO module to establish associations between two modalities significantly increases the number of parameters, complicating the training process and extending training time, yet results in only a very limited improvement (0.2% to 0.6%).
3. The paper focuses solely on cross-domain semantic segmentation tasks, but the AFO module needs to optimize losses for both tasks (semantic segmentation and depth estimation), requiring supervision with both semantic and depth ground truths. This significantly increases the difficulty of data acquisition and annotation in real-world applications.
4. The joint optimization of losses for depth estimation and semantic segmentation tasks increases the difficulty of optimization, making the model harder to learn. While it is reasonable to assume that depth features could aid the performance of semantic segmentation tasks, joint optimization is not necessarily the best approach. Generally, optimizing multiple tasks simultaneously can lead to decreased performance on individual tasks.
paper_summary: paper_summary: This paper proposes a depth-aware unsupervised domain adaptation framework for semantic segmentation tasks. The method utilizes a Depth-Guided Contextual Filter (DCF) and a cross-task encoder to enhance the mixing of source and target domain data using depth information, improving segmentation performance for small objects and complex scenes. Experimental results show that this method achieves significant performance improvements on the GTA→Cityscapes and Synthia→Cityscapes datasets. Even when using pseudo depth data, the method achieves competitive performance.
strength_weakness: strength_weakness: 1.The proposed depth-aware framework in the article represents an innovative approach that utilizes depth estimation to clearly distinguish and blend semantic categories, thereby simulating the layout of the real world. This method effectively addresses the issue of layout mismatch that may arise from simply blending source and target domain data in traditional unsupervised domain adaptation methods.
2.By introducing a cross-task encoder, the framework achieves contextual learning and feature fusion for the two complementary tasks of segmentation and depth learning. This collaborative learning approach across tasks contributes to extracting more comprehensive and accurate feature representations, thereby enhancing the performance of scene segmentation.
limitations: limitations: From what I know, the author has used all the datasets related to domain adaptation tasks from virtual scenes to real scenes in autonomous driving. The experiments compared many models related to previous research in this field, but is it necessary to include some models that are slightly old in time and have significantly different effects in the comparison list? At the same time, I think the method proposed by the author is universal and can also be tested in some tasks in  adverse scenes, such as Cityscapes to ACDC, for example.