Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel segmentation framework that could be aware of depth distribution density and thus improve the segmentation performance using unsupervised domain adaptation. The proposed network is based on the multi-task learning architecture that enables three different tasks: segmentation, depth regression, and depth density distribution estimation. At the end of the architecture, the adversarial network is appended, and thus the segmentation performance is significantly improved. The main contributions of this paper are (1) the suggestion of the probability density of depth distribution to bridge the gap in the unsupervised domain adaptation, (2) a novel idea of branch balance training that enables improving the segmentation performance, and (3) pseudo-labels refinement algorithm that utilizes the aggregation priors of pixels in the different categories on the source images. The evaluation was performed using synthetic images as the source domain and real-world images as the target domain. The experiments demonstrate the effectiveness of the proposed method in the unsupervised domain adaptive semantic segmentation task.

- The proposed architecture mainly aims at segmenting target objects in the images from the target domain, which has not been annotated from the source domain's knowledge (especially depth distribution density), which includes the annotated labels of segmentation map and depth distribution.

- The architecture is based on the simple multi-task learning architecture but utilizes the additional adversarial training architecture to improve the segmentation performance. 

- In multi-task learning, three tasks of segmentation, depth regression, and estimating the depth distribution density are included. 
  - The segmentation task is the same as the general semantic segmentation task (supervised learning with annotated labels) using cross-entropy loss. 
  - The depth regression is also supervised learning with the annotated labels, and the regression is performed using the berHu loss function.
  - From the segmentation map and the depth information, annotation for the density distribution can be generated. Then, the depth distribution density estimation is performed using the estimated density from the other two branches and the predicted density (branch balance training). 

- The proposed method was evaluated using two datasets of Cityscapes and Mapillary as the target domain. In addition, the SYNTHIA dataset is utilized as the source domain. The images in the target domain are real-world images, whereas the images in the source domains are generated synthetic images.  

- The experiments report the state-of-the-art performance of the proposed depth distribution-aware unsupervised domain adaptation network compared to other state-of-the-art deep learning models.

strengths_and_weaknesses: 1. Strengths
- The reviewer agrees that the significance and importance of the task addressed in this paper, especially the depth information in each category, can improve the segmentation performance. In addition, unsupervised domain adaptive semantic segmentation with depth distribution density has rarely been studied to our best knowledge, such that the idea of the proposed task and network in this paper would be a novel contribution. Furthermore, the experimental result reports the state-of-the-art performance of the proposed network, and thus the proposed network can be expected as a novel framework.

- Additionally, the problem definition is clear. The authors were motivated to utilize the depth information and the depth density distribution while segmenting the target object. The idea of using depth distribution density is simple yet effective. 

- The manuscript is well organized and well written.


2. Weakness
- More detailed descriptions are illustrated in the “Limitation” section. The main limitation suggested by the reviewer is the lack of detailed description. Please see below.
questions: * Questions & Discussions
0. The main questions and discussions are raised by the vague explanations or ignorance of the reviewer. Clear explanations and the discussions in the rebuttal/discussion phase would resolve the following questions. If the reviewer misunderstands the concepts or definitions, please fix the problems with clearer explanations.

1. The definition of the domain adaptation should be more discussed. In this paper, the authors addressed that the proposed framework could bridge the domain gap in UDA for semantic segmentation. The domain adaptation task mainly aims at bridging the domain gap such as style differences, geometric differences, or even statistical gaps between two domains. However, this paper has not discussed the main domain gap to be resolved. The authors repeatedly advert the utilization of depth information to bridge the domain gap. The clear explanations that the utilization of depth information can bridge the “what domain gap” in this paper should be discussed. The reviewer expects that the manuscript could be improved with clear illustrations of the problem definition.

2. The main technical contribution should be more discussed.

- In the first contribution (Line 61-62; Page 2), the authors address that the depth distribution is similar in the source and the target domain. The reviewer agrees that this conjecture is well working while regarding the large object and obvious objects like sky, land, road, etc. However, it is worried that this conjecture could be well appliable to the small objects like bikes and pedestrians, frequently appeared objects in one domain, and many objects from the same category (e.g., many cars are in one image). Depth distribution density is based on this hypothesis, the authors should illustrate the experimental or mathematical evidence for this hypothesis. The authors mentioned this in Line 104-106; Page 3, but the statistical analysis and evidence (table or graph) should be provided.

- The second contribution proposed by the authors is the branch balance training, which performs simple density regression using the existing loss function. It would be better to explain why the training method using a simple loss function can contribute to NeurIPS society. In addition, the novel idea beyond the simple equation (equation 6) should be discussed in the manuscript.

- The reviewer was impressed by the novel idea of the utilization of depth distribution density to improve the segmentation performance. Despite the simple architecture based on multi-task learning, the motivation and the proposed methodology could be a novelty. Therefore, the forest could be the contribution rather than the trees.

3. The training pipeline and the network architecture should be discussed. 

- The training procedure should be clearer. The proposed architecture can be divided into two pipelines: (1) multi-task learning architecture and (2) adversarial network architecture. In the multi-task learning architecture, the images in the source domain are used to train the network with their annotations (segmentation map, depth map, and the calculated density map). In contrast, the images in the target domain are used to train the network with a segmentation map (?), depth map (?), and the density map of the source domain. Then, it is concluded that the training process using the target domain only optimizes the parameters in the density-predictive pipeline, is it? Furthermore, the adversarial network architecture inputs the fused density map from the source and the target domain both. Therefore, it is concluded that the training process using the source and the target domains are concurrently performed on both (1) and (2) architectures. Is it truly the process to train the network? Other domain adaptation methods train the network using the following in general:

  - (a) Transfer the style of the source domain to that of the target domain and then train the network (M) with the property of the target domain. Then, M predicts the segmentation output using the images in the target domain. [1]

  - (b) Train the network (M) using the source domain. Then, transfer the style of the target domain to that of the source domain, and M predicts segmentation maps using the transferred images. [2, 3] 

  - (c) CNN architecture generates feature maps from the source and the target domains, and the CNN architecture aims at increasing the similarity between two feature maps [4].  

- The reviewer is interested in whether the proposed domain adaptation method is more similar to (a), (b), (c), or other existing methods (please list the reference). Or, if the proposed methods are significantly novel, then the training pipeline should be clearly illustrated. The reviewer recommends authors illustrate the training pipeline in a visual way (figures) or mathematical formulations (or algorithm table).

- Effectiveness of the adversarial training. Related to the question above, the reviewer is curious about the effectiveness of the adversarial training. As the reviewer understood, the adversarial training utilizes the combined feature maps of the density map and the predicted segmentation map. The authors should clearly describe the goal of the adversarial training. In the Generative Adversarial Network (GAN), the generator and the discriminator are simultaneously optimized by the min-max game. In the proposed network, what advantage could be realized by deceiving the discriminator here? Does the author aim at designing that the generators (here segmentation and density pipelines) aim at generating similar feature maps using the source and the target domains? In this case, the bias (biased to segmentation map or biased to density map) could be discussed. 

- Cost of the proposed framework. The reviewer is curious about the training time, prediction time, and memory resources. Inevitably, the multi-task learning-based network causes a heavy cost in prediction/training time and even in memory utilization. Quantitative analysis or discussion can significantly improve the quality of the manuscript. For the reviewer, since the proposed network includes multi-task learning- and adversarial learning-based architecture, the training and prediction cost can be extremely burdened compared to the conventional algorithms.

[1] Lee, Seunghun, et al. "ADAS: A Direct Adaptation Strategy for Multi-Target Domain Adaptive Semantic Segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Hou, Yunzhong, and Liang Zheng. "Source free domain adaptation with image translation." arXiv preprint arXiv:2008.07514 (2020).

[3] Lee, Kyungsu, Haeyun Lee, and Jae Youn Hwang. "Self-Mutating Network for Domain Adaptive Segmentation in Aerial Images." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[4] Ganin, Yaroslav, and Victor Lempitsky. "Unsupervised domain adaptation by backpropagation." International conference on machine learning. PMLR, 2015.

limitations: 1. Major issues
- Related to the question illustrated in "Question" the followings should be improved. (1) The definition of the domain adaptation and the domain gap should be discussed. (2) Limited contribution and novelty. (3) Vague explanations for the training pipelines. 

- Additional visual illustrations are required. Specifically, the predicted segmentation maps are well organized in the experiment parts. However, the outputs of the proposed network are segmentation, density, and depth maps. Therefore, the author should exhibit three maps to demonstrate that the proposed network is outstanding in the unsupervised domain adaptive semantic segmentation task. 

- Mathematical notations and abbreviations should be strict.

  - The mathematical modeling should be invited for newly proposed terms with precise definitions. For instance, the authors proposed depth distribution density and should provide a clear definition of this term. Likewise, "density map" and "depth map" should be invited with clear mathematical definitions, including rank, dimension, size, range, and type (matrix, vector, set, scalar, etc.).

  - It is better to use the familiar term: "annotation/ ground truth/ semantic label/ ground truth segmentation map/ ground truth semantic label" instead of "ground truth segmentation annotation". In this case, P_hat can be the "predicted segmentation map" or "segmentation prediction".

  - In Line 135; Page 4, X_i should be re-checked. As the authors notated, X_i depends on i. However, the authors described X_i as a tuple (x, y, z) which is regardless of i. Therefore, the reviewer think X_i should be the list of tuples (x, y, z) where the associated pixel is classified as i-th class. 

  - In Line 136; Page 4, j is firstly invited in Pi_{i,j} without any definition. Please clarify the definition of j.

  - In Line 140, Page 4, does N(~~~) indicate the flag (0 or 1) whether X_i is the gaussian distribution? In this case, if the input of the function N is only X_i, the author can omit the mean and the variance, even though N is adopted from the notation of the normal distribution "N~(0,1)".

  - GMM first appeared in Line 84; Page 2 without any definition. Please fix it with the gaussian mixture model in Line 46; Page 2. 

  - Algorithm I is significantly far from the mathematical modeling and algorithm. Please simplify the table.

2. Minor issues (recommendations)
- Justify the hyper-parameter selections (Line 165-166; Page 5/ Line 179-190; Page 5/ Line 192-193; Page 6/ Line 198-203; Page 6). In addition, since the architecture is based on the multi-task learning architecture and the memory of GPU is 11GB, the training should be conducted in small batch size. The batch size and the normalization method (batch/group/instance/etc normalization) should be discussed in this case.

- (if available) The experiments should be improved. The reviewer already understands that the evaluation metric of "mean Intersection over Union (mIoU)" can qualitatively measure the object details. However, recent studies [1-3] proposed new evaluation metrics to quantitatively measure the object detail (especially the boundaries of the target objects). Otherwise, the visualization of the feature map could illustrate the novel feature extraction when processing object details. Alternatively, the authors could refer to the activation maps [4]. More experimental or mathematical evidence should be justified to clearly report the outstanding segmentation performance in terms of the improved boundary-oriented segmentation.

[1] Fernandez-Moral, Eduardo, et al. "A new metric for evaluating semantic segmentation: leveraging global and contour accuracy." 2018 IEEE intelligent vehicles symposium (iv). IEEE, 2018.

[2] Lee, Kyungsu, et al. "Boundary-oriented binary building segmentation model with two scheme learning for aerial images." IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-17.

[3] Cheng, Bowen, et al. "Boundary IoU: Improving object-centric image segmentation evaluation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[4] Selvaraju, Ramprasaath R., et al. "Grad-cam: Visual explanations from deep networks via gradient-based localization." Proceedings of the IEEE international conference on computer vision. 2017.


3. Simple recommendations
- The reviewer recommends reviewing the grammar and typo errors to improve the quality of the manuscript.

- The reviewer recommends clarifying the meaning of the arrows in figure 1 (dashed line and other lines). Especially, L_bal uses the calculated density and the predicted density. What does "entropy" indicate? The inputs and outputs of each module are recommended to be clarified.

- Motivation for utilization of the berHu loss, ResNet101, ASPP, and AvgPooling.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work is about unsupervised domain adaptation for semantic segmentation and how to improve the domain transfer with depth information. In contrast to prior work, the authors investigate the distribution of depth for individual categories as a better bridge between domains than plain depth prediction. Specifically, a Gaussian Mixture Model (GMM) is fit to the source training data (x-y location and depth of each pixel, separately for each class), which is then used to compute a per-pixel density map, i.e., the probability of observing a depth value at a certain pixel location and with a certain semantic class. Then, a neural network is extended to not only predict semantics and depth, but also the depth distribution. For domain adaptation, an adversarial loss is computed on features that are a combination of the semantic and depth distribution branch. Finally, a spatial-aggregation prior to compute better pseudo labels for self-training is also proposed. Experiments on standard benchmarks (that include depth ground truth in the source domain) show state-of-the-art results and an ablation study investigates the impact of the individual contributions.
strengths_and_weaknesses: ### Strengths

- Well written, except a few grammatical issues and typos (please get those checked)
- The structure of the paper is well done, illustrations help understand the method
- Ablation studies are mostly good
- Main results show a good improvement over state-of-the-art
- Source code is made available

### Weaknesses

- The motivation about class-wise depth distribution in lines 41ff seems to be based heavily on typical driving scenes. It would be better to also think about other types of scenes (e.g., indoor) and see if the same argument holds.
- Throughout the paper, it was hard to imagine what "spatial aggregation priors" could mean until reading the actual section about it. Similarly in line 100, "the spatial aggregation scale of pixels" is unclear to me.
- The depth distribution is advertised as domain bridge, but doesn't this modality depend on the camera settings, which may be different in source and target domains? For instance, if the focal length is different (i.e., more or less zoom), then objects further away would be represented with more pixels and thus likely change the depth distribution.
- In nearly all equations, it would be better to end it with a comma (",") instead of having the comma in the beginning of the next line.
- For equation 5, I think it would be good to add an intuitive explanation that every pixel in D describes the probability of seeing the depth value with this semantic class at this pixel location under the learned GMM.
- Lines 155f: The argument about "balance" of branches is unclear to me, and also why the loss is named that way.
- Section 3.5 needs to be re-written. First, it was not clear to me that self-training would be part of the proposed solution. Second, in line 189, it's not clear initially what the thresholds are for and how the pseudo labels are "refined" with them. Finally, the spatial prior refinement algorithm needs a more intuitive explanation and potentially a figure for easier understanding.
- I do not understand the point the first paragraph in Sec. 4.5 (lines 267ff) tries to make.
questions: - I'm wondering if the same idea of adding depth distribution as additional task would help plain semantic segmentation on the source domain, i.e., in a non domain adaptation setting.
- Line 142: What's a "channel-wise GMM"?
- Is there any explanation why the performance drops so much for S3 in Table 4? I was actually wondering what the results look like if you did the same feature fusion as DADA.
limitations: One limitation has been discussed in the conclusion (sub-optimal architecture design for depth prediction branch). Potential societal impacts were not discussed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: For the UDA task, the authors propose to utilize the probability density of depth distribution to bridge the domain gap for semantic segmentation. 
The proposed method conduct experiments on the SYNTHIA-to-Cityscapes and SYNTHIA-to-Mapillary setting.
strengths_and_weaknesses: (+) The code is provided.
(+) Writing and presentation are good.
(+) The branch balance strategy and Pseudo-labels refinement are interesting.
(-) A lot of recent UDA approaches, e.g. [R1-R4] are missing in this paper. The proposed	method can not perform as well as many CVPR2021/CVPR2022/ICCV2021 works.
(-) The most common case under the UDA setting -- GTA5->Cityscapes is missing. I suspect the performance of the proposed method in this setting.

[R1] Zhang, Pan, et al. "Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[R2] Huo, Xinyue, et al. "Domain-Agnostic Prior for Transfer Semantic Segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[R3] Hoyer, Lukas, Dengxin Dai, and Luc Van Gool. "Daformer: Improving network architectures and training strategies for domain-adaptive semantic segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[R4] Wang, Qin, et al. "Domain adaptive semantic segmentation with self-supervised depth estimation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

questions: What is the training cost when involving the additional branches?

Is the model sensitive to the weight values in Line166? How do you pick those parameters?

After reading the whole paper, It is still very hard to understand the density of depth distribution. I highly recommend adding a visualization on the density map, and it might be helpful for readers to follow the overall idea.
limitations: Using additional branches for training introduced more time and resources to get a model.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper addresses the task of unsupervised domain adaptation in semantic segmentation. Following the same line as previous works [11,12,13], the paper leverages depth supervision in source domain as the privileged information to improve segmentation in target domain. The paper presents an intuitive idea based on the observation that: pixels of a particular semantic class often have similar positions in the image space and have similar depth ranges across source and target. Based on which, the paper proposes a task of estimating depth distribution density 2D maps, which are then used to guide the final output-level adversarial adaptation. Reported results are better than previous SOTAs.

** Technical details **
The network architecture is a standard MTL with three heads: one for the main task of segmentation, one for monocular depth and one for estimation the ``depth distribution density maps'' (see below for more details); the three branches are attached to the same Resnet101 backbone. The two branches segmentation and depth are supervised in source with cross-entropy and berHu losses as done in [12]. Depth distribution density supervision is gotten from segmentation prediction $\hat{P}$ and either ground-truth depth $Z_s$ in source or estimated depth $\hat{Z}_t$ in target.

The depth distribution of each semantic class is modeled as a Gaussian mixture model (GMM) in the image space; GMM parameters are estimated using source data. The number of components is fixed as 5. For each source image, using the depth ground-truth $Z_s$ and segmentation prediction $\hat{P}_s$, one can produce a depth distribution density map $D_s\in\mathbb{R}^{H\times W}$, i.e. at the position $(x,y)$, the value $D_s(x,y)$ is determined as the depth distribution density at $(x,y)$ of class $\hat{P}_s(x,y)$. For each target image, depth distribution density map $D_t$ is estimated using depth prediction $\hat{Z}_t$ and segmentation prediction  $\hat{P}_t$.

For adaptation, this work applies adversarial training using an additional discriminator [19,12]. Input to the discriminator is a fusion of segmentation prediction $\hat{P}$ and estimated depth density map $D$. To further boost performance with self-training, a heuristic pseudo-labelling procedure is used.
strengths_and_weaknesses: ** Strengths **
- The idea is intuitive and it seems easy to implement the density branch given the availability of existing codebase.
- Paper presentation is quite clear and easy to understand. However writing could be improved here and there, e.g. L144-146 I don't quite get the meaning.

** Weaknesses **
- This work lacks quantitative and qualitative analysis of the new branch for density distribution estimation. It's hard for me to imagine what are produced by the density branch. What advantages brought by the joint space of segmentation and density ( as compared to depth + seg joint space)? Some qualitative density results as well as t-SNE/UMAP visualization of joint-space features are needed.
- The self-training strategy is highly heuristic. Seemingly this strategy help recruit more rare-class pseudo-labels by relaxing the constraints for those (?) I don't understand why the values 50 and 4950 were chosen.

=========== Post-rebuttal:
I thank the authors for actively answering my questions during the rebuttal period. Most of my concerns were addressed. I'm happy to increase my score.
If this paper is accepted, I would like to see in the final version more discussions and qualitative results as promised by the authors, especially on the pseudo-labelling refinement.
questions: Questions:
- Why berhu loss is used for supervising the density branch? What is the reaso?

Some analyses that could improve this work:
  - Sensitivity analysis of hyper parameters, especially the distribution density one.
  
  - Although depth and density estimations are auxiliary tasks, it would be interesting to quantify (as done in [13]) or at least visualize 
  some depth and density results. Does density improve depth?

  - It's not yet been a standard for UDA in segmentation due to high computational cost but it would be highly 
  appreciated if the authors can provide mean and std performance with different runs and seeds.
limitations: The conclusion discusses a general limitation on depth quality, same as similar works.
No discussion on potential negative societal impacts was given.
review_title: null

Meta-review: metareview: The paper proposes a depth-aware segmentation framework that leverages unsupervised domain adaptation by segmenting, regressing the depth, and estimating the depth density distribution.  The reviewers' doubts about the paper were addressed during the rebuttal, and the reviewers agree that the authors answered most of their concerns, and that the remaining problems are not an impediment for publication.  While the paper doesn't achieve SotA, the new ideas and the shown experiments are enough to validate the proposal, and its ideas will be of interest to the community. 

I recommend the paper for publication given the contributions on the multi-task setup and the mixture of domains, and that the experiments validate the proposal.
confidence: Certain
Decision: Accept