Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper concerns the geometric distortion that causes domain gaps during unsupervised domain adaptation for semantic segmentation, which is motivated by the differences between rectilinear images and fisheye images. It is practically well-grounded.
The authors propose to train a deformation generator that consumes two images chosen from the source and target domains.
The output of the deformation generator is used to map the source image to the target style in terms of geometric distortion.
The training or adaptation is then happened by learning from the distorted source image and semantic segmentations.
The authors also propose a few benchmarks to evaluate the performance of the method and show good domain adaptation gain compared to baselines that concern global appearance change of the images.
An ablation study on each term in the proposed training loss is also performed.
strengths_and_weaknesses: Strength:
- the motivation is practically meaningful and also the introduced setting is interesting.
- the proposed method is simple yet shows good domain adaptation gain. Also, the proposed benchmarks could help develop more sophisticated methods that deal with both geometric distortion and appearance change.
- the constraints to impose semantic consistency of the generated geometric distortion are interesting.
- the paper is well-written and the idea is clearly conveyed.

Weakness:
- even though the motivation is well-grounded, the study of the validity of the problem, or how the motivating factor affects the adaptation performance is not crystal clear. For example, the authors should isolate geometric distortion from other factors like appearance and output space discrepancies. A simple experiment to do is to create a distorted target dataset from rectilinear images, and check how the other methods and the proposed method perform. The current message in this paper does not support a judgment on this aspect.
- In theory, an ideal method should close the gap given only geometric distortion. But there is concern on whether the proposed method con do it under only geometric distortion or not, since the proposed randomly picks two images even though they are not in correspondence. This may be okay, since we can think of the proposed as a kind of spatial augmentation method, but we need to check the results.
- The above concerns are critical for us to analyze the proposed method and the underlying problems that the proposed is trying to solve. For example, affine aug already works well on some of the directions in Table 1, does this mean that spatial augmentation/transformation is the key here? If so, would the proposed method a more sophisticated such augmentation method?
- Also, how does the proposed method compare to spatial transformation generated by optical networks? The proposed do not really care about finding the correct underlying dense matching, which makes OF look like a good candidate to try, say, use OF to output some flow and then warp the images to the target domain or vice versa and then adapt? What is the difference then?
- If camera distortion is the only factor that causes domain gap and is the focus of the current paper, then it seems that camera calibration would be a nice tool and can be performed quite well using existing tools. Then what does the paper tell us beyond that?
questions: The main concerns are listed in the above section, please provide more information since they are related to the significance of the proposed problem and method.

some comments and questions on writting
- the title "DaDA: Distortion-aware Domain Adaptation for Unsupervised Semantic Segmentation" may not be appropriate as semantic segmentation is not unsupervised.
- ln 58 "to enforce the semantic quality of relative deformation fields at the image- and the prediction level." how do you define semantic quality of deformation fields? also, what do you mean by "image- and prediction level"?
- ln 142, the equation is confusing, phi_s2t is mapping from the source domain to the target domain, which takes pixel coordinates from source and maps it to source.
limitations: Did not observe much negative societal impact. On the limitation side, the authors indicate that other methods that deal with texture differences between domains can be combined with the proposed one to further improve performance. However, this is also an indicator that the paper with a clean motivation is evaluating on datasets with multiple causal factors, which does not convey a clean message.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel segmentation framework that could be aware of distortion caused by optical and geometric reasons and thus improve the segmentation performance using unsupervised domain adaptation. The main contributions of this paper are (1) the framework that can be aware of distortion and utilize the information in the unsupervised semantic segmentation by using distortion-aware domain adaptation, (2) loss functions that effectively enforce the unsupervised domain adaptation and segmentation, and (3) new unsupervised domain adaptation benchmark including pre-existed datasets and newly proposed dataset, where the source and target images have additional domain gaps in the optical distortion.

The distortion-aware domain adaptation is realized by a novel Relative Distortion Learning (R.D.L.) using adversarial training (GAN) and Diffeomprohic transformation. The generator (G) generates deformation fields (visualization of distortion) between the source and target domain images. 

The proposed method was evaluated using four datasets, especially two for the source domain, and the other two are utilized as the target domain. The images in the source domain are real-world rectilinear images, and the images in the target domains are optically and geometrically distorted images captured by fisheye images. Here, the authors propose a new dataset called the FDD captured by fisheye cameras (200-degree F.O.V.). 

The experiments illustrate the outstanding performance of the proposed distortion-aware domain adaptation framework compared to other state-of-the-art deep learning models and other methods.

strengths_and_weaknesses: 1. Strengths
- The reviewer significantly understands the significance and importance of the task proposed in this paper. The distortion caused by the lens can significantly degrade the segmentation performance, especially the object on edge. In addition, domain adaptive semantic segmentation with lens distortion has rarely been studied to our best knowledge, such that the tasks proposed in this paper for the distortion-aware domain adaptation would be novel. Furthermore, since the dataset construction requires heavy cost, the construction of a large number of datasets for training machine learning models is noteworthy. 

- Additionally, the problem definition is clear. The authors defined the domain gap using not only distortion but also visual domain gap. At this point, in terms of the domain gap, the discrepancy between visual reasons and optical and geometric reasons should be discussed, and it should be decomposed. The authors proposed the “only” distortion-aware framework regardless of the visual domain gap.

- The manuscript is well organized and well written.

2. Weakness
- More detailed descriptions are illustrated in the “Limitation” section. Please see below.

questions:  * Questions & Discussions
1. The definition of the domain adaptation should be more discussed. In this paper, the authors conducted the experiments with the environment where the rectilinear images are the source domain, whereas the images with extreme distortion are the target domain. However, in terms of “general” domain adaptation, the source and the target could be changed. Additionally, both the source domain and the target domain could contain the distortion on the edge. The reviewer is curious that the proposed framework is effective only in a limited environment. Alternatively, the reviewer is curious that other environments should not be discussed in the real-world image. If this case, the authors should clearly discuss this limitation. It is also discussed below [Limitations]-[Major issues].

2. Effectiveness of the discriminator. The reviewer is curious about the effectiveness of the discriminator. Since the generator is well organized in the RDL and the loss function, the adversarial training can cause additional cost rather than a significant improvement in the accuracy. Otherwise, what if the discriminator categorizes the input images into domains (source and target). For instance, what if the discriminator aims to discriminate distortion style between I_T, I_(S->T), I_S, and I_(T->S). Since the generator generates the deformation field (S->T) and the inverse deformation field (T->S), training using four images can lead to the globally optimized discriminator, and it could be more efficient for the framework and the adversarial training. In addition, the ablation study for the discriminator could improve the quality of the manuscript.

3. Cost of the proposed framework. The reviewer is curious about the training time, prediction time, and memory resources. In the manuscript, the authors addressed that the proposed framework is effective compared to the algorithm that removes the distortion from the image. Quantitative analysis or discussion can significantly improve the quality of the manuscript. For the reviewer, since the proposed network includes GAN-based architecture, segmentation-based architecture, and three or more loss functions, the training and prediction cost can be extremely burdened compared to the conventional algorithms.

limitations: 1. Major issues
- The field of interest suggested in this paper is narrow. The reviewers agree that the distortion can significantly degrade the segmentation performance when the image includes extreme distortion like the image captured using a fisheye camera. Even though real-world images contain the distortion, the reviewers could not be convinced that the minor distortion significantly degrades the segmentation performance. It could be justified through more experiments. The authors should significantly address the necessity of distortion-aware domain adaptation in the machine learning society instead of computer vision society. Additionally, the following concepts and experiments should be discussed to justify that the proposed framework is effective in the "more general" domain adaptive segmentation. (1) Utilization of the dataset containing distortion as the source domain and rectilinear images as the target domain. (2) Utilization of the dataset containing distortion as the source domain and another dataset with distortion as the target domain. Note that, this paper discussed the environment where the rectilinear images are utilized as the source domain and the images with distortion as the target domain. Or the authors should justify the experimental environment in this paper.

- In addition, the above questions and the discussion are mainly concerned with a better quality of this paper for Neurips2022 (See Questions).

2. Minor issues
- Motivation for unsupervised learning needs to be more detailed. 
As the authors illustrated in Line 45-47 on page 2, the motivation for unsupervised learning is the lack of a real-world public dataset with segmented annotations. Clearly, the lack of annotations for the dataset, including many images, could lead to unsupervised learning. However, that sentence author mentioned in the manuscript implies that the lack of the images (or dataset) leads the unsupervised learning in this paper. As the authors already knew, unsupervised learning with a few images could induce problems with biased training (e.g., overfitting). The reviewer expects that the precise motivation for unsupervised learning should be addressed strongly and clearly and that the authors can improve the quality of the manuscript with few modifications.

- Small batch size should be discussed. As illustrated in Line 30-34, on page 1, in the supplementary file, the batch size for training deep learning models is four until convergence, and the batch size is significantly small. The importance of the normalization methods (e.g., batch normalization or group normalization) depends on the batch size. Therefore, discussing the batch size and the normalization methods would be better.

- The vague description of the dataset and preparations should be improved. (1) the authors mentioned, "Both source and target images are randomly cropped" in the supplementary file in Line 32, on page 1. As the reviewer understood, the images in the target domain include the distortion on edge. At this moment, the random cropping method for the images in the target domain is nonsense. (2) In the experiments, the authors split images into training and validation sets with the random selection method. Is it for the cross-validation or hold-out methods? More explanations and statistical analysis (if authors conducted experiments with many folds) should be discussed even in the supplementary files.

- Justification to search hyper-parameters should be discussed. Common hyper-parameters to train deep learning models are well illustrated in the manuscript and the supplementary files (especially the selection of beta 1-3). However, the reviewer is curious about the selection process of gamma. 

- (if possible) Experiments with other evaluation metrics could improve the quality of the manuscript. For instance, the boundary is a significantly important target in the segmentation tasks, and the distortion could degrade the boundary-oriented segmentation accuracy. Therefore, the discussion using evaluation metrics [1-3] for the boundary of the target objects can significantly improve the quality of the manuscript.

[1] Fernandez-Moral, Eduardo, et al. "A new metric for evaluating semantic segmentation: leveraging global and contour accuracy." 2018 IEEE intelligent vehicles symposium (iv). IEEE, 2018.

[2] Lee, Kyungsu, et al. "Boundary-oriented binary building segmentation model with two scheme learning for aerial images." IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-17.

[3] Cheng, Bowen, et al. "Boundary IoU: Improving object-centric image segmentation evaluation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

3. Simple recommendations
- Please re-check the typo and grammar errors to improve the quality of the manuscript.
- Complicated quantitative analysis (Tables 5 and 6 in the supplementary material) should be clearly illustrated. 
- The reason why the authors illustrated the visualization of the class-activation map in the manuscript should be described in the main manuscript.
- Public access to the code and the dataset should be preceded.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The author proposed a distortion-aware domain adaptation (DaDA) framework that is capable of modeling domain shifts in geometric deformation based on a relative distortion learning (RDL) method. 

The proposed method tackles the task of unsupervised domain adaptation for semantic image segmentation where unknown optical distortion exists between the source and target images.

Adequate experimental results also prove the validity of the proposed method。
strengths_and_weaknesses: 1.Strengths
This paper proposes a novel domain adaptation method for unsupervised semantic segmentation. And it combines geometric and optical distortion in domain shift. And the extensive experimental results highlight the effectiveness of our approach over state-of-the-art methods under unknown relative distortion across domains.
2.Weaknesses
See the “Questions” part.
questions: 1.The figures in the text are far apart from where the figures are mentioned in the body text. It is suggested that the author shorten the corresponding distance to facilitate the reader’s comparative reading.

2.Why are the edges of the reconstructed source domain image not neat? Such as the “d” part in figure2.

3.In the "Experimental Details" section, the performance of both the “AdaptSeg+RA+RDL” and “AdvEnt+RA+RDL” methods decreases compared to the RA removal method when cityscapes is the source domain dataset and FDD is the target domain dataset. The authors need an explanation for this experimental result.

4.Are the references in [17] and [18] duplicated?

5.The format of references is not uniform. For example, the format of the reference in [28] differs from the rest of the literature.

6.Most of the references are earlier than 2020. It is recommended to cite more papers in related fields after 2021.
limitations: The authors adequately address the limitations of their work and the potential negative social impact.
review_title: null

Meta-review: metareview: This paper proposes a new segmentation method with geometric insight to deal with the distortions.  As pointed out by our reviewers, this paper is featured with important practical value, clear problem definition, and interesting mathematical insight. During the rebuttal phase, most of the reviewers confirmed their support for an (weak) acceptance, and I believe this paper should be accepted as a poster paper. 
confidence: Certain
Decision: Accept